Linear Conic Programming Theory andLinear Conic Programming Theory and Applications1 Shu-Cherng Fang...

Linear Conic Programming Theory andApplications1

Shu-Cherng Fang

Department of Industrial and Systems EngineeringGraduate Program in Operations Research

North Carolina State University

1Fang, S. C., and W. Xing. “Linear Conic Programming” (2013).

Conic Programming 1 / 25

Content• Part I : Introduction• Part II: Preliminary and Convex Cone Structure• Part III: Duality Theory of Linear Conic Programming• Part IV: Interior Point Methods and Solution Software• Part V: Modelling and Applications• Part VI: Recent Research• Part VII: Practical LCoP


LCoP Part I – Introduction

Shu-Cherng Fang




Introduction

Content• Linear Conic Programs• Applications• Duality Theory and Algorithms• References


Linear Conic Program

Min C •Xs.t. A •X = B

X ∈ K(LCoP)

where K is a closed, convex cone; A, B and C are in the space ofinterests with • being an appropriate linear operator.


K = Rn+ (First Orthant)

When K = Rn+ = {x ∈ Rn|xi ≥ 0, i = 1, ..., n}, LCoP becomes LP.

Min cTxs.t. Ax = b

x ∈ Rn+

(LP)

where A ∈ Rm×n, b ∈ Rm and c ∈ Rn.

Equivalently,

Min cTxs.t. aTi x = bi, i = 1, . . . ,m

x ≥Rn+

0(LP)

where ai ∈ Rn, bi ∈ R and c ∈ Rn.


K = Rn+

Figure:R1+ Figure:R2

+ Figure:R3+


K = Ln (Lorentz Cone/Second OrderCone)

When K = Ln = {x ∈ Rn|√x21 + · · ·+ x2n−1 ≤ xn}, LCoP becomes

SOCP.Min cTxs.t. Ax = b

x ∈ Ln(SOCP)

where A ∈ Rm×n, b ∈ Rm and c ∈ Rn.

Equivalently,

Min cTxs.t. aTi x = bi, i = 1, . . . ,m

x ≥Ln 0(SOCP)

where ai ∈ Rn, bi ∈ R and c ∈ Rn.


K = Ln

Figure: L2 Figure: L3


K = Sn+ (Positive Semidefinite Cone)

When K = Sn+ = {X ∈ Rn×n|X = XT � 0}, LCoP becomes SDP.

Min C •Xs.t. Ai •X = bi, i = 1, ...,m

X ∈ Sn+(SDP)

where C,A1, ..., Am are given n× n symmetric matrices and b1, ..., bm aregiven scalars, and

M •X =∑i,j

MijXij = tr(MTX).

Equivalently,

Min C •Xs.t. Ai •X = bi, i = 1, ...,m

X � 0(SDP)


K = Sn+

S2+ =

{(x, y, z) ∈ R3|

[x yy z

]� 0.

}⇐⇒ x ≥ 0, z ≥ 0, xz ≥ y2.


Application of SOCP – I

Torricelli Point ProblemThe problem is proposed by Pierre de Fermat in 17th century. Giventhree points a, b and c on the R2 plane, find the point in the plane thatminimizes the total distance to the three given points. The solutionmethod was found by Torricelli, hence know as Torricelli point.

SOCP Formulation

Min t1 + t2 + t3

s.t.

[x− at1

]∈ L3,

[x− bt2

]∈ L3,

[x− ct3

]∈ L3

Question:C =? A =? X =? · · ·


Application of SOCP – II

Robust Portfolio DesignAssume returns r are known within an ellipsoid

E = {r = r + κΣ1/2u : ‖u‖2 ≤ 1}.

where r is the expected return, Σ is the empirical covariance matrix,0 < κ < 1 is a given constant.

robust counterpart: (optimize the worst case)

maxω

minr∈E{rTω : eTω = 1, ω ≥ 0}.


SOCP Formulation

Notice thatminr∈E

rTω

= min‖u‖2≤1

{rTω + κuT Σ1/2ω}

= rTω − κ‖Σ1/2ω‖2

Robust portfolio problem is an SOCP

Max rTω − κ‖Σ1/2ω‖2s.t. eTω = 1, ω ≥ 0

⇐⇒

Max ts.t. eTω = 1, ω ≥ 0[

κΣ1/2ωrTω − t

]∈ Ln+1

Question:C =? A =? X =? · · ·


Application of SDP – I

Correlation Matrix Verification

Consider three random variables A, B and C. By definition, theircorrelation coefficients ρAB , ρAC and ρBC are valid if and only if 1 ρAB ρAC

ρAB 1 ρBC

ρAC ρBC 1

� 0

Suppose we know from some prior knowledge (e.g. empirical results ofexperiments) that −0.2 ≤ ρAB ≤ −0.1 and 0.4 ≤ ρBC ≤ 0.5. What are thesmallest and largest values that ρAC can take?


SDP Formulation

The problem can be formulated as the following problem:

Min/Max ρAC

s.t. −0.2 ≤ ρAB ≤ −0.1

0.4 ≤ ρBC ≤ 0.5

ρAA = ρBB = ρCC = 1 ρAA ρAB ρAC

ρAB ρBB ρBC

ρAC ρBC ρCC

� 0


SDP Formulation

In order to formulate the problem as in standard form, we handle theinequality constraints by augmenting the variable matrix and introducingslack variables, for example

0 12 0 0 0 0 0

12 0 0 0 0 0 00 0 0 0 0 0 00 0 0 1 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 0

•

ρAA ρAB ρAC 0 0 0 0ρAB ρBB ρBC 0 0 0 0ρAC ρBC ρCC 0 0 0 0

0 0 0 s1 0 0 00 0 0 0 s2 0 00 0 0 0 0 s3 00 0 0 0 0 0 s4

= ρAB + s1 = −0.1


SDP Formulation

X =

1 ρAB ρAC 0 0 0 0ρAB 1 ρBC 0 0 0 0ρAC ρBC 1 0 0 0 00 0 0 s1 0 0 00 0 0 0 s2 0 00 0 0 0 0 s3 00 0 0 0 0 0 s4

, C =

0 0 12

0 0 0 00 0 0 0 0 0 012

0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 0

A1 =

0 12

0 0 0 0 012

0 0 0 0 0 00 0 0 0 0 0 00 0 0 1 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 0

A2 =?, A3 =? and A4 =?


Other Applications - QCQP =⇒ SOCP

The popularity of SOCP is also due to the fact that it is a generalized formof convex QCQP (Quadratically Constrained Quadratic Programming).Specifically, consider the following QCQP:

Min xTA0x+ 2bT0 x+ c0s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m

where A0 � 0, Ai � 0 for i = 1, . . . ,m.Note that

t ≥n∑

i=1

x2i ⇐⇒

∣∣∣∣∣∣∣∣∣

∣∣∣∣∣∣∣∣∣

x1...xn

(t− 1)/2

∣∣∣∣∣∣∣∣∣

∣∣∣∣∣∣∣∣∣2

≤ t+ 1

2⇐⇒

x1...xn

(t− 1)/2(t+ 1)/2

∈ Ln+2


Other Applications - QCQP =⇒ SOCP

Therefore, for each i = 1, . . . ,m

xTAix+ 2bTi x+ ci ≤ 0⇐⇒

A1/2i x

−1/2− bTi x− ci/21/2− bTi x− ci/2

∈ Ln+2

QCQP can be equivalently written as

Min u

s.t.

A1/20 x

−1/2− bT0 x+ u/2− c0/21/2− bT0 x+ u/2− c0/2

∈ Ln+2

A1/2i x

−1/2− bTi x− ci/21/2− bTi x− ci/2

∈ Ln+2, i = 1, . . . ,m.


Other Applications - SOCP =⇒ SDP

Ln+1 can be easily embedded into Sn+1+ by observing the fact that[

xt

]∈ Ln+1 ⇐⇒

[t xT

x tIn

]∈ Sn+1

+

Based on this, we will focus on the theorems and algorithms for SDP. Butthis does not mean that SOCP is useless or we should transform SOCPto SDP in any case.


Duality Theory of LP

Theorem (Weak Duality Theorem of LP)If x is primal feasible and y is dual feasible, then cTx ≥ bT y.

Theorem (Strong Duality Theorem of LP)

• If either LP or LD has a finite optimal solution, then so does the otherand they achieve the same optimal objective value.

• If either LP or LD has an unbounded objective value, then the otherhas no feasible solution.

How about the duality theorems of LCoP?


Algorithms for LP

Simplex Method for LP

• Starting from one vertex• Check whether current vertex is optimal or not. If yes, stop.

Otherwise, go to the next step.• Move to a neighboring vertex, go to the previous step.

The complexity of simplex method is not polynomial.

Polynomial-time Algorithms

• Ellipsoid Method• Karmarkar’s Projective Scaling Algorithm• Affine Scaling Algorithm: Primal, Dual and Primal-Dual

How about the algorithms for LCoP?


References

Books• Bertsekas D.P., Nedic A. and Ozdaglar A.E., Convex Analysis and

Optimization, Athena Scientific: Belmont, MA USA 2003• Boyd S. and Vandenberghe L., Convex Optimization, Cambridge University

Press: Cambridge, UK 2004• Fang S. and Puthenpura S., Linear Optimization and Extensions: Theory

and Algorithms, Prentice-Hall Inc.: Englewood Cliffs, NJ USA 1993• Nemirovski A., Lectures on Modern Convex Optimization: Analysis,

Algorithms, and Engineering Applications, Society for Industrial and AppliedMathematics: Philadelphia, PA USA 2001

• Renegar J., A Mathematical View of Interior-point Methods in ConvexOptimization, Society for Industrial and Applied Mathematics: Philadelphia,PA USA 2001

• Handbook of Semidefinite Programming: Theory, Algorithms, andApplications, edited by Wolkowicz H., Saigal R. and Vandenberghe L.,Kluwer Academic Publisher: Norwell, MA USA 2000


References

• Rockafellar R.T., Convex Analysis, Princeton University Press:Princeton, NJ USA 1970

• Wright. S., Primal-Dual Interior-Point Methods, Society for Industrialand Applied Mathematics: Philadelphia, PA USA 1997

Others

• Ye Y., Linear Conic Programming, lecture notes online: http://www.stanford.edu/class/msande314/sdpmain.pdf

• Todd M.J., Semidefinite Programming, lecture notes online:http://people.orie.cornell.edu/~miketodd/cornellonly/or637/or637.html

A very popular general purpose SDP solver, SeDuMi, of Jos F. Sturm canbe found in: http://sedumi.ie.lehigh.edu/


http://www.stanford.edu/class/msande314/sdpmain.pdf

http://www.stanford.edu/class/msande314/sdpmain.pdf

http://people.orie.cornell.edu/~miketodd/cornellonly/or637/or637.html

http://people.orie.cornell.edu/~miketodd/cornellonly/or637/or637.html

http://sedumi.ie.lehigh.edu/

LCoP Part II – Preliminaries and Convex ConeStructures

Shu-Cherng Fang




Preliminaries and Convex Cone Structures

Content• Vectors, Matrices, and Spaces• Inner Products and Norms• Open, Closed, Interior, and Boundary Sets• Functions• Linear Systems• Convex Sets and Functions


Vectors, Matrices and Spaces

• Real numbers: R, R+, R++

• Euclidean space: Rn

• First orthant: Rn+• n-dimensional (column) vector:

x = (x1, x2, . . . , xn)T

• Matrices space: Rm×n

• Matrix: M ∈ Rm×n, ith row Mi·, jth column M·j , ijth entry Mij(Mi,j)

• Symmetric square matrices space (n(n+ 1)/2-dimensional space):

Sn = {M ∈ Rn×n |M = MT }.



Given M ∈ Rm×n, N ∈ Rn×m, S ∈ Rn×n

• Determinant: det(S)

• Trace: tr(S)tr(MN) = tr(NM)

• Null space: N (M)= {x ∈ Rn|Mx = 0}.• Range space: R(M)= {y ∈ Rm|y = Mx for some x ∈ Rn}.• Positive semidefinite matrix:

S � 0 ⇐⇒ zTSz ≥ 0, ∀ z ∈ Rn

• Positive definite matrix:

S � 0 ⇐⇒ zTSz > 0, ∀ z ∈ Rn and z 6= 0



Theorem: (Schur complementary theorem)Given

X =

[A BBT C

]and S = C −BTA−1B,

if A � 0 thenX � (�)0⇔ S � (�)0


Inner Products and Norms

• Inner products:x • y = xT y =

∑i xiyi

X • Y = tr(XTY ) =∑i,j XijYij

• Norms:• Euclidean norm: ‖x‖2 =

√x • x

• p-norm: ‖x‖p = (∑n

i=1 |xi|p)1/p for p ≥ 1

• Infinity-norm: ‖x‖∞ = max{|x1|, . . . , |xn|}• Frobenius norm:

‖X‖F =√X •X =

√tr(XTX)

• Note that: xTAx = A • xxT


Open, Closed, Interior and Boundary Sets

• Neighborhood: N(x0; ε) = {x ∈ Rn| ‖x− x0‖ < ε}.

• Open: X ⊂ Rn is open if for any x ∈ X , there exists ε > 0 such thatN(x; ε) ⊂ X .

• Closed: X ⊂ Rn is closed, if Rn\X = {x ∈ Rn|x /∈ X} is open.

• Closure of a set X ⊂ Rn is the smallest closed set containing X andis denoted as cl(X ).


Open, Closed, Interior and Boundary Sets

• Interior: the interior of a given set X ⊂ Rn is

int(X ) = {x ∈ X |∃ εx > 0 such that N(x; εx) ⊂ X}

• Boundary of a set X ⊂ Rn:

bdry(X ) = cl(X )\int(X ) = {x ∈ cl(X )|x /∈ int(X )}

• Bounded: a set X ⊂ Rn is bounded if there exist an r > 0 such that

‖x‖ < r,∀x ∈ X


Functions

• Continuous: f : X ⊂ Rn is continuous at x0 if( i ) x0 ∈ X(i i) limx→x0 f(x) = f(x0)

• Continuous function: f ∈ C0(X ) means f is continuous at all pointsin X ⊂ Rn.

• Gradient: For f : X ⊂ Rn → R

∇f(x) = [∂f(x)

∂x1, · · · , ∂f(x)

∂xn]1×n

• Hessian: For f : X ⊂ Rn → R

F (x) = [∂2f(x)

∂xi∂xj]n×n

• Continuously differentiable function: f ∈ Cp(X ) (p = 1, 2, · · · ) meansf is p-th continuously differentiable over X ⊂ Rn.


Functions

Theorem (Taylor theorem)Let X be open, f ∈ Cp(X ), x1, x2 ∈ X , x1 6= x2 and

x(θ) = θx1 + (1− θ)x2 ∈ X , ∀ 0 ≤ θ ≤ 1.

Then ∃ x = θx1 + (1− θ)x2 ∈ X , 0 < θ < 1, s.t.

f(x2) = f(x1) +

p−1∑k=1

1

k!dkf(x1;x2 − x1) +

1

p!dpf(x;x2 − x1)

where dkf(x;h) is the k-th order differential of function f along h.


Functions: Big O and Small o

Let g(·) be a real-valued function on R.

• g(x) = O(m(x))

∃ c ≥ 0 such that ∣∣∣∣ g(x)

m(x)

∣∣∣∣ ≤ c as x→ 0 (or +∞)

• g(x) = o(m(x)) ∣∣∣∣ g(x)

m(x)

∣∣∣∣ = 0 as x→ 0 (or +∞)


Functions

Taylor theorem in small o formulation:• p = 1

f(x+ h) = f(x) +∇f(x)h+ o(‖h‖)• p = 2

f(x+ h) = f(x) +∇f(x)h+1

2hTF (x)h+ o(‖h‖2)


Linear Systems

Given x1, · · · , xm ∈ Rn• Linear combination:

m∑i=1

λixi,

where λi ∈ R, i = 1, . . . ,m.• Linearly independent

m∑i=1

λixi = 0⇒ λ1 = · · · = λm = 0

• Affine combination: a linear combination withm∑i=1

λi = 1

• Affinely independent: if x2 − x1, · · · , xm − x1 are linearlyindependent.


Linear Systems

• Convex combination: a linear combination withm∑i=1

λi = 1 and λi ≥ 0, i = 1, . . . ,m

• Hyperplane:

X = {x ∈ Rn|aTx =

n∑i=1

aixi = b}

• Affine space: affine combination of any two points in the space is stillin the space. (An intersection of finitely many hyperplanes.)

• Linear subspace: an affine space containing the origin.

We can always transform an affine space Y ⊂ Rn into a linearsubspace X ⊂ Rn by choosing x0 ∈ Y such that

X = {x− x0|x ∈ Y}


Linear Systems

• Half space:

X = {x ∈ Rn|aTx =

n∑i=1

aixi ≤ b}

• Polyhedron: an intersection of finitely many half spaces.

• Polytope: a bounded polyhedron

• Dimension of a linear subspace: the maximum number of linearlyindependent vectors in the subspace.

• Dimension of an affine space: the dimension of the transformedlinear subspace.

• Dimension of a polyhedron: the dimension of the smallest affinespace containing it.


Linear Systems

• Linear equations

a1 • x = b1a2 • x = b2· · · · · · · · ·

am • x = bm

⇒ Ax = b,

where a1, · · · , am and x are all in Rn.

A1 •X = b1A2 •X = b2· · · · · · · · ·

Am •X = bm

⇒ AX = b,

where A1, · · · , Am and X are all in Sn.• For convenience, A∗y =

∑mi=1 yiAi.


Convex Sets and Properties

• A set X ⊂ Rn is convex if for any x1 ∈ X and x2 ∈ X , we haveλx1 + (1− λ)x2 ∈ X , for all 0 ≤ λ ≤ 1.

• Convex hull: the smallest convex set containing a given set

conv(X ) = {x ∈ Rn|x =∑mi=1 λiy

i for some m ∈ N+,λi ≥ 0,

∑mi=1 λi = 1, and yi ∈ X , i = 1, . . . ,m}

• Dimension of a convex set: the dimension of the smallest affinespace containing it.

• Relative interior of a convex set X ⊂ Rn: suppose H is the smallestaffine space containing X ,

ri(X ) = {x ∈ Rn|∃ open set Y ⊆ Rn such that x ∈ Y ∩H ⊂ X}

• Supporting hyperplane H = {x ∈ Rn|aTx = b} of a convex set X :

aT y ≥ b,∀ y ∈ X and X ∩H 6= ∅.


Convex Functions and Properties

• Epigraph of a function f : X ⊂ Rn → R

epif = {(x, λ) ∈ Rn+1|λ ≥ f(x), x ∈ X}

• Closed function: if epif is a closed set.

• Convex function: if epif is a convex set.

• Concave function: if −f is a convex function.

• Convex hull function conv(f) of a function f : X ⊂ Rn → R is afunction on X such that epi(conv(f)) = conv(epi(f)).

Lemmaf : X ⊂ Rn → R is a convex function if and only if for any x1, x2 ∈ X and0 ≤ λ ≤ 1, we have

f(λx1 + (1− λ)x2) ≤ λf(x1) + (1− λ)f(x2).



• Subgradient d ∈ Rn of a convex function f : X ⊂ Rn at x ∈ X :

if for any y ∈ X ,f(y) ≥ f(x) + dT (y − x)

• The set {(y, λ) ∈ Rn+1|λ− dT y = f(x)− dTx} is a supportinghyperplane of epif at x.

• Subdifferential of a convex function f : X ⊂ Rn at x ∈ X :

∂f(x) = {d ∈ Rn|d is a subgradient of f at x}



Figure: (x, f(x))↔ (m, b) or (y, h(y))



• Conjugate (transform) of f : X ⊂ Rn → R:

h(y) = supx∈X{y • x− f(x)}

with h being defined on Y = {y ∈ Rn|h(y) < +∞}.

Lemmah : Y is a closed, convex function.

Lemma (Fenchel’s inequality)Given f : X and its conjugate h : Y, then

x • y ≤ f(x) + h(y), ∀ x ∈ X and y ∈ Y.

Moreover,x • y = f(x) + h(y) ⇐⇒ y ∈ ∂f(x)


Conjugate Functions and Properties

Let f : X ⊂ Rn → R be a function with its conjugate transform h : Y.• For α ∈ R, the conjugate of f + α is h− α.• For a ∈ Rn, the conjugate of f(x) = f(x) + x • a on X ish(y) = h(y − a), ∀ y ∈ Y.

• For a ∈ Rn, the conjugate of f(x) = f(x− a) on X ish(y) = h(y) + y • a, ∀ y ∈ Y.

• For λ > 0, the conjugate of f1(x) = λf(x) on X is h1(y) = λh( yλ ),∀ y ∈ λY.

• For λ > 0, the conjugate of f2(x) = f(xλ ) on λX is h2(y) = h(λy),∀ y ∈ Y/λ.

TheoremAssume that f1 : X and f2 : X have the same convex hull function. Thenthey have the same conjugate transform h : Y when it exists.


Conjugate Functions and Properties

We know the dual problem of LD is LP again. When will the conjugatetransform of h : Y become f : X?

Proper functionA convex function f is proper if its epigraph is non-empty and contains novertical lines, i.e. if f(x) < +∞ for at least one x and f(x) > −∞ forevery x.

TheoremLet f : X ⊂ Rn → R be a proper closed convex function with conjugatetransform h : Y. Then the conjugate transform of h : Y is f : X . Moreover,y ∈ ∂f(x) if and only if x ∈ ∂h(y). In this case,

x • y = f(x) + h(y) ⇐⇒ y ∈ ∂f(x) or x ∈ ∂h(y)


Convex Cone Structure

Content• Convex Cones and Properties• Partial Order and Ordered Vector Space• Some Examples


Convex Cones and Properties

• A set K ⊂ Rn is a cone if

∀x ∈ K and λ > 0⇒ λx ∈ K;

• A cone K ⊂ Rn is pointed if

K ∩ −K = {0};

• A cone K ⊂ Rn is solid ifintK 6= ∅;

• A cone K ⊂ Rn is proper if it is pointed, solid, closed and convex.



• Conic combination: a linear combination∑mi=1 λix

i with λi ≥ 0,xi ∈ Rn for all i = 1, . . . ,m.

• The conic hull of a set X ⊂ Rn is

cone(X ) = {x ∈ Rn|x =∑mi=1 λix

i, for some m ∈ N+

and xi ∈ X , λi ≥ 0, i = 1, . . . ,m.}

• The dual cone K∗ ⊂ Rn of a cone K ⊂ Rn is

K∗ = {y ∈ Rn|y • x ≥ 0,∀ x ∈ K}

K∗ is a closed, convex cone.• If K∗ = K, then K is a self-dual cone.



K, K1, K2 are convex cones in Rn.• (K∗)∗ = cl(K).• K1 ⊆ K2 ⇒ K∗2 ⊆ K∗1 .• K1 ∩K2, K1 ∪K2, K1 +K2, K1 ×K2 are all cones.• (K1 +K2)∗ = K∗1 ∩K∗2 .• K1, K2 closed⇒ K1 +K2 and K1 ×K2 are closed.• ri(K1 +K2) = ri(K1) + ri(K2).• ri(K1 ×K2) = ri(K1)× ri(K2).• The supporting hyperplane of K always contains the origin• If K is solid, then K∗ is pointed.• If K is pointed, then K∗ is solid.• If K is proper, then K∗ is proper.



• Rn+ ⊆ Rn, Ln ⊆ Rn, Sn+ ⊆ Rn×n are pointed, closed, convex andsolid cones, i.e., proper cones.

• (Rn+)∗ = Rn+, (Ln)∗ = Ln, (Sn+)∗ = Sn+.

• Rn1+ ×R

n2+ × · · · ×R

nk+ is a proper cone in R

k∑i=1

ni

.

• Ln1 × Ln2 × · · · × Lnk is a proper cone in Rk∑

i=1ni

.

• Sn1+ × S

n2+ × · · · × S


k∑i=1

ni×ni

.• Rn1

+ × Ln2 × Sn3+ is a proper cone in Rn1+n2+n3×n3 .


Partial Order and Ordered Vector Space

• A relation “≥” is a partial order on a set X if it has:

1. reflexivity: a ≥ a for all a ∈ X ;

2. antisymmetry: a ≥ b and b ≥ a imply a = b;

3. transitivity: a ≥ b and b ≥ c imply a ≥ c.

• An ordered vector space X is equipped with a partial order “≥” whichalso satisfies:

• homogeneity: a ≥ b and λ ∈ R+ imply λa ≥ λb;

• additivity: a ≥ b and c ≥ d imply a+ c ≥ b+ d.


Partial Order and Ordered Vector Space

• A proper cone K in a vector space can induce a partial order “≥K”

a ≥K b⇔ a− b ∈ K

which leads to an ordered vector space.• Similarly, we can define “≤K”

a ≤K b⇔ b ≥K a,

• Closeness of K allows passing limits in ≥K :

ai ≥K bi, ai → a, bi → b as i→∞ ⇒ a ≥K b.

• Solidness of K allows us to define a strict inequality:

a >K b⇔ a− b ∈ intK,

anda <K b⇔ b >K a.


Examples: Rn+

• Rn+ is a proper cone.

• Inner product: x • y = xT y

• (Rn+)∗ = Rn+ (self-dual)

• Partial order: “≥Rn+

”


Examples: Ln

• Ln / SOC(n− 1)Lorentz cone (secondorder cone)

Ln = {x ∈ Rn|xn ≥√x21 + · · ·+ x2n−1}

• Ln is a proper cone.

• Inner product: x • y = xT y

• (Ln)∗ = Ln (self-dual)

• Partial order: “≥Ln ”


Examples: Sn+

• Sn+ ⊂ Sn: the set of symmetric positive semidefinite matrices• Sn+ is a proper cone.• Inner product:

X • Y = tr(XTY )

• Another view:

vec(X) = [X11,√

2X12, X22,√

2X13,√

2X23, X33, · · · , Xnn]T ∈ Rn(n+1)

2

ThenX • Y = vec(X) • vec(Y ) =

∑i,j

XijYij

• Partial order: “≥Sn+

” or “�”


Examples: Sn+

Lemma(Sn+)∗ = Sn+ (self-dual)

Proof.“⊆”: If X ∈ (Sn+)∗, then zTXz = X • zzT ≥ 0, for all z ∈ Rn.Therefore, X ∈ Sn+.

“⊇”: For any Y ∈ Sn+,

Y =∑ni=1 λiz

i(zi)T ,

with λi ≥ 0.If X ∈ Sn+, then

X • Y =

n∑i=1

λiX • zi(zi)T =

n∑i=1

λi(zi)TXzi ≥ 0.

Therefore, X ∈ (Sn+)∗.Conic Programming 34 / 39

Examples: Cn and C∗n

• Copositive cone:

Cn = {X ∈ Sn|zTXz ≥ 0,∀ z ≥Rn+

0}

• Completely positive(nonnegative) cone:

C∗n =

{X ∈ Sn

∣∣∣∣∣ X =∑mi=1 z

i(zi)T , for some m ∈ N+

and zi ≥Rn+

0, i = 1, . . . ,m

}

• (Cn)∗ = C∗n and Cn = (C∗n)∗

• C∗n ⊂ Sn+ ⊂ Cn


Examples: Cones of NonnegativeQuadratic Functions — Homogeneous

• F ⊂ Rn• Nonnegative homogeneous quadratic functions over F

f(x) = xTAx ≥ 0,∀x ∈ F

f ⇔ A

• HDF = {A ∈ Sn|xTAx ≥ 0,∀x ∈ F} is a closed, convex cone.(i) Closeness:

xTAix ≥ 0 and Ai → A⇒ xTAx ≥ 0

(ii) Convexity:

xTAiX ≥ 0, i = 1, 2⇒ xT (λA1 + (1− λ)A2)x ≥ 0,∀ 0 ≤ λ ≤ 1


Examples: Cones of NonnegativeQuadratic Functions — Homogeneous

• HD∗F = cl(cone{xxT |x ∈ F})• (HDF )∗ = HD∗F and (HD∗F )∗ = HDF• Examples:

• F = Rn

HDF = HD∗F = Sn+

• F = Rn+

HDF = Cn and HD∗F = C∗n• F = {x ∈ Rn

+|eTx = 1}HDF = Cn and HD∗F = C∗n


Examples: Cones of NonnegativeQuadratic Functions — Nonhomogeneous

• Nonnegative quadratic functions over F ⊂ Rn

f(x) = xTAx+ 2bTx+ c ≥ 0,∀x ∈ F

f ⇔[c bT

b A

]• DF = {

[c bT

b A

]∈ Sn+1|

[1x

]T [c bT

b A

] [1x

]≥ 0,∀x ∈ F} is a closed,

convex cone.

• D∗F = cl(cone{[

1 xT

x xxT

]|x ∈ F})

• (D∗F )∗ = DF and (DF )∗ = D∗F


Examples: Cones of NonnegativeQuadratic Functions — Nonhomogeneous

• Examples:• F = Rn

DF = D∗F = Sn+1+

• F = Rn+

DF = Cn+1 and D∗F = C∗n+1


LCoP Part III – Duality Theory of Linear ConicProgramming

Shu-Cherng Fang




Duality Theory of Linear ConicProgramming

Content• Definition of LCoP and LCoD• Conjugate Duality Theory• Deriving LCoD from LCoP• Conic Duality Theorems for LCoP• Duality Theorems of LP, SOCP and SDP


Linear Conic Programs

Recall thatMin C •Xs.t. A •X = B

X ∈ K(LCoP)

where K is a closed, convex cone, A, C and B are in the space ofinterests with • being an appropriate linear operator.

Note that when K = Rn+ or Ln, X is a vector; when K = Sn+, X is an

n× n matrix.


Linear Conic Programs

Min c • xs.t. ai • x = bi, i = 1, . . . ,m

x ∈ K(LCoP)

where K is a closed, convex cone, such as Rn+, Ln, Sn+.

Max bT ys.t.

∑mi=1 yia

i + s = cs ∈ K∗, y ∈ Rm

(LCoD)

where K∗ is the dual cone of K.


Conjugate Duality Theory

Conjugate Program

inf f(x)s.t. x ∈ X ∩K

(CP)

where f : X ⊂ Rn → R and K is a cone in Rn.

Conjugate Dual

inf h(y)s.t. y ∈ Y ∩K∗

(CD)

where h : Y is the conjugate transform of f : X and K∗ is the dual coneof K.• feas(*) denotes the feasible domain of problem (*)• opt(*) denotes the optimal solution set of problem (*)• v(*) denotes the optimal value of problem (*)



Theorem (Conjugate duality theorem/KKT duality theorem)If x ∈ feas(CP) and y ∈ feas(CD), then

0 ≤ x • y ≤ f(x) + h(y)

with the equality holding if and only if

x • y = 0 and y ∈ ∂f(x),

in which casex ∈ opt(CP) and y ∈ opt(CD).

ProofThe inequality follows from Fenchel’s inequality and the definition of dualcone. The rest follows easily.



Theorem (Weak duality theorem)If both CP and CD are feasible, then

(i) v(CP) is finite and

v(CP) + h(y) ≥ 0,∀ y ∈ feas(CD);

(ii) v(CD) is finite and

v(CP) + v(CD) ≥ 0.

ProofThis theorem follows from the previous KKT duality theorem.



Theorem (Fenchel’s theorem/Strong duality theorem)Suppose that f : X and K are closed and convex. If v(CD) is finite andone of the following conditions holds:

(i) ri(K∗) ∩ ri(Y) 6= ∅,(ii) both K∗ and Y are polyhedrons,

thenv(CP) + v(CD) = 0 and opt(CP) 6= ∅.

Similarly, if v(CP) is finite and one of the following conditions holds:(i) ri(K) ∩ ri(X ) 6= ∅,(ii) both K and X are polyhedrons,

thenv(CP) + v(CD) = 0 and opt(CD) 6= ∅.

Proof: See Rockafellar’s book “Convex Analysis” Section 31.


Deriving LCoD from LCoP

LCoP

Min c • xs.t. ai • x = bi, i = 1, . . . ,m

x ∈ K(LCoP)

Deriving LCoD in the framework of conjugate program.



LCoP as CP

Variables: uT = (u0, u1, . . . , um) ∈ Rm+1;

f(u) = u0;

X = {u ∈ Rm+1|ui = bi, i = 1, . . . ,m};

K0 = {u ∈ Rm+1|u0 = c • x, ui = ai • x, x ∈ K, i = 1, . . . ,m}.

inf f(u)s.t. u ∈ X ∩K0



Corresponding CD

Variables: vT = (v0, v1, . . . , vm) ∈ Rm+1;

h(v) = supu∈X{u • v − f(u)} < +∞

= supu0∈R{(v0 − 1)u0 +

m∑i=1

bivi}

Hence

h(v) =∑m

i=1 bivi;

Y = {v ∈ Rm+1|v0 = 1};



Corresponding CD

Moreover,

K∗0 = {v ∈ Rm+1|v • u ≥ 0,∀u ∈ K0}

= {v ∈ Rm+1|(v0c +∑m

i=1 viai) • x ≥ 0,∀x ∈ K}

= {v ∈ Rm+1|v0c +∑m

i=1 viai ∈ K∗}.

Hence

Y ∩K∗0 = {v ∈ Rm+1|c +

m∑i=1

viai = s, s ∈ K∗}.

inf∑m

i=1 bivis.t. c +

∑mi=1 via

i = ss ∈ K∗



CD to LCoDDefine variables: y = −(v1, . . . , vm)T , we have

Max bT ys.t.

∑mi=1 yia

i + s = cs ∈ K∗, y ∈ Rm

(LCoD)

Therefore, the duality theorems of conjugate programs may apply toLCoP.


Conic Duality Theorems for LCoP

Theorem (Weak duality theorem)If both LCoP and LCoD are feasible, then

c • x ≥ bT y,∀x ∈ feas(LCoP) and (y, s) ∈ feas(LCoD).

Theorem (Strong duality theorem)

(i) If feas(LCoP) ∩ int(K) 6= ∅ and v(LCoP) is finite, then there exists(y∗, s∗) ∈ feas(LCoD) such that bT y∗ = v(LCoP).

(ii) If feas(LCoD) ∩ int(K∗) 6= ∅ and v(LCoD) is finite, then there existsx∗ ∈ feas(LCoP) such that c • x = v(LCoD).

Proof: See Aharon Ben-Tal and Arkadi Nemirovski’s book “Lectures onmodern convex optimization” Chapter 2.


Conic Duality Theorems for LCoP

Theorem (KKT duality theorem)If feas(LCoP) and feas(LCoD) are both nonempty andfeas(LCoP) ∩ int(K) 6= ∅, then x∗ is optimal for LCoP if and only if thefollowing conditions hold:

(i) x∗ ∈ feas(LCoP);

(ii) There exists (y∗, s∗) ∈ feas(LCoD);

(iii) c • x∗ = bT y∗ (or equivalently x∗ • s∗ = c • x∗ − bT y∗ = 0).Proof: =⇒ follows from strong duality theorem.

⇐= is obvious.


Linear Program (LP)

Min cTxs.t. Ax = b

x ≥Rn+

0(LP)

Max bT ys.t. AT y + s = c

s ≥Rn+

0(LD)


Linear Program (LP)

Theorem (LP duality theorem)

(i) If either LP or LD is unbounded, then the other one is infeasible.

(ii) If either v(LP) or v(LD) is finite, then there exist x∗ ∈ feas(LP) and(y∗, s∗) ∈ feas(LD) such that v(LP) = cTx∗ = bT y∗ = v(LD).

(iii) If LP is feasible and v(LP) is finite, then x∗ is optimal for LP if andonly if the following conditions hold:

(a) Ax∗ = b, x∗ ≥Rn+

0;

(b) there exists (y∗, s∗) satisfying AT y∗ + s∗ = c, s ≥Rn+

0;

(c) (x∗)T s∗ = cTx∗ − bT y∗ = 0.


Second Order Cone Program (SOCP)

Min cTxs.t. Ax = b

x ≥K 0(SOCP)

where K = Ln1 × · · · × Lnr = {x ∈ Rn|n1 + · · ·+ nr = n, (x1, ..., xn1)T ∈

Ln1 , ..., (xn−nr+1, ..., xn)T ∈ Lnr}.


s ≥K 0(SOCD)



Theorem (SOCP duality theorem)

(i) If either SOCP or SOCD is unbounded, then the other one isinfeasible.

(ii) If there exists a feasible solution x such that x ∈ int(K), andv(SOCP) is finite, then there exist (y∗, s∗) ∈ feas(SOCD) such thatv(SOCP) = bT y∗ = v(SOCD).

(iii) If there exists a feasible solution (y, s) such that s ∈ int(K), andv(SOCD) is finite, then there exist x∗ ∈ feas(SOCP) such thatv(SOCP) = cTx∗ = v(SOCD).



Theorem (SOCP duality theorem)

(iv) If both SOCP and SOCD are feasible, and there exists a feasiblesolution x such that x ∈ int(K), then x∗ is optimal for SOCP if andonly if the following conditions hold:

(a) Ax∗ = b, x∗ ≥K 0;

(b) there exists (y∗, s∗) satisfying AT y∗ + s∗ = c, s∗ ≥K 0;

(c) (x∗)T s∗ = cTx∗ − bT y∗ = 0.


Difference between LP and SOCP (interior feasible solution):

Min −x2

s.t. x1 − x3 = 0x ∈ L3

Max 0 · y

s.t.

0−10

− y

10−1

=

−y−1y

∈ L3

v(SOCP ) = 0 but SOCD is infeasible.

Figure: Feasible domain is a ray x1 = x3 in hyperplane x2 = 0. No feasibleinterior point.



Finite nonzero duality gap:

Min −x2

s.t. x1 + x3 − x4 + x5 = 0x2 + x4 = 1x ∈ L3 × L2

Max y2

s.t.

y1 +s1 = 0y2 +s2 = −1

y1 +s3 = 0−y1 +y2 +s4 = 0y1 +s5 = 0

s ∈ L3 × L2

x∗ =

−101

× [11

]y∗ =

[−1−1

]s∗ =

101

× [01

]

v(SOCP) = 0 6= −1 = v(SOCD)



Zero duality gap with non-attainable value:

Min x1

s.t. −x2 −x3 = 0x2 = −1

x ∈ L3

Max −y2s.t. s1 = 1

−y1 +y2 +s2 = 0−y1 +s3 = 0

s ∈ L3

x∗ =

0−11

v(SOCD) = 0 but not attainable.


Semidefinite Program (SDP)

Min C •Xs.t. AX = b

X � 0(SDP)

Max bT ys.t. A∗y + S = C

S � 0(SDD)

Note:

A∗y =

m∑i=1

yiAi



Theorem (SDP duality theorem)

(i) If either SDP or SDD is unbounded, then the other one is infeasible.

(ii) If there exists a feasible solution X such that X � 0, and v(SDP) isfinite, then there exist (y∗, S∗) ∈ feas(SDD) such thatv(SDP) = bT y∗ = v(SDD).

(iii) If there exists a feasible solution (y, S) such that S � 0, and v(SDD)is finite, then there exist X∗ ∈ feas(SDP) such thatv(SDP) = C •X∗ = v(SDD).



Theorem (SDP duality theorem)

(iv) If both SDP and SDD are feasible, and there exists a feasiblesolution X such that X � 0, then X∗ is optimal for SDP if and only ifthe following conditions hold:

(a) AX∗ = b, X∗ � 0;

(b) there exists (y∗, S∗) satisfying A∗y∗ + S∗ = C, S∗ � 0;

(c) X∗ • S∗ = C •X∗ − bT y∗ = 0.



Interior feasible solutionInfinite duality gap:

C =

[0 11 0

], A =

[0 00 1

], b = 0

X∗ =

[0 00 0

]and SDD is infeasible.

Zero duality gap with non-attainable value:

C =

[1 00 0

], A =

[0 11 0

], b = 1

v(SDP ) = 0 but is not attainable. y∗ = 0 and S∗ =

[1 00 0

].



Finite nonzero duality gap:

C =

0 0 00 0 10 1 0

, A1 =

0 0 00 1 00 0 0

, A2 =

1 0 00 0 −10 −1 0

, b =

[01

]

X∗ =

1 0 00 0 00 0 0

, y∗ =

[0−1

], S∗ =

1 0 00 0 00 0 0

v(SDP ) = 0 6= −1 = v(SDD)


LCoP Part IV – Interior Point Methods andSolution Software

Shu-Cherng Fang




Interior Point Methods and SolutionSoftware

Content• Interior Points and Primal-Dual Model• Barrier Functions and Optimal Systems• Central Path and Newton Methods• Path Following Method• CVX Solution Software


Interior Point Methods

• Interior point approach• Start from an interior point solution.• If the current solution is not good enough, then move to another interior

point solution.• Stop at an interior point solution whose objective value is close to the

optimum (within an ε gap).• Advantages:

• Polynomial time complexity (comparing to the simplex method for LP)• Excellent computational performance in practice (comparing to the

ellipsoid method)• Three types: primal; dual; primal-dual


Primal-dual Model

• Primal-dual type of LP

Min sTxs.t. Ax = b

AT y + s = cx ≥Rn

+0, s ≥Rn

+0

(LPD)

• Primal-dual type of SDP

Min S •Xs.t. AX = b

A∗y + S = CX � 0, S � 0

(SDPD)

• Note:AX = [A1 •X, · · · , Am •X]T

and A∗y =∑mi=1 yiAi


Interior Points

feas+(LP) = {x ∈ Rn|Ax = b, x >Rn+

0}feas+(LD) = {(y, s) ∈ Rm ×Rn|AT y + s = c, s >Rn

+0}

feas+(LPD) = feas+(LP)× feas+(LD)

feas+(SDP) = {X ∈ Sn|AX = b,X � 0}feas+(SDD) = {(y, S) ∈ Rm × Sn|A∗y + S = C, S � 0}feas+(SDPD) = feas+(SDP)× feas+(SDD)

• Assumptions:• feas+(LP) and feas+(LD) are not empty and the rows of A are linearly

independent.

• feas+(SDP) and feas+(SDD) are not empty and the vectors formed byAi in A are linearly independent.


Barrier function

• Properties required:• Strictly convex (concave).• Goes to +∞ (−∞) when the point is close to the boundary.• Sufficient continuous differentiability.

• Barrier functions:

LP : −∑ni=1 log xi SDP : − log det(X)

LD :∑ni=1 log si SDD : log det(S)

LPD : −∑ni=1 log(xisi) SDPD : − log det(XS)


LP with Barrier

Min cTx− µ∑ni=1 log xi

s.t. Ax = bx >Rn

+0

(LPB)

Max bT y + µ∑ni=1 log si

s.t. AT y + s = cs >Rn

+0

(LDB)

Min sTx− µ∑ni=1 log(xisi)

s.t. Ax = bAT y + s = cx >Rn

+0, s >Rn

+0

(LPDB)


Common Optimal System for LP withBarrier

Ax = bAT y + s = cΛxs = µex >Rn

+0, s >Rn

+0,

where e = (1, . . . , 1)T and Λx is a diagonal matrix with (Λx)ii = xi,i = 1, . . . , n.

Notice that

µ =xT s

n=cTx− bT y

n

When µ→ 0, sTx→ 0. Optimal!


SDP with Barrier

Min C •X − µ log det(X)s.t. AX = b

X � 0(SDPB)

Min bT y + µ log det(S)s.t. A∗y + S = C

S � 0(SDDB)

Min S •X − µ log det(XS)s.t. AX = b

A∗y + S = CX � 0, S � 0

(SDPDB)


Common Optimal System for SDP withBarrier

AX = bA∗y + S = CXS = µIX � 0, S � 0

Notice that

µ =S •Xn

=C •X − bT y

n

When µ→ 0, S •X → 0. Optimal!


Central Path for LP and SDP

CLP = {(x, y, s) ∈ feas+(LPD)|Λxs = µe, 0 < µ < +∞}

CSDP = {(X, y, S) ∈ feas+(SDPD)|XS = µI, 0 < µ < +∞}

Under proper assumptions:• For any 0 < µ < +∞, there exists a unique point on central path.

LP: (x(µ), y(µ), s(µ))

SDP: (X(µ), y(µ), S(µ))

• Given µ > 0, the set {(x, y, s) ∈ feas+(LPD)|Λxs = µe, 0 < µ < µ} isbounded.Given µ > 0, the set {(X, y, S) ∈ feas+(SDPD)|XS = µI, 0 < µ < µ}is bounded.


Example: Central Path (Primal)

Min x1 + x2

s.t. x1 + x2 ≤ 3x1 − x2 ≤ 1x2 ≤ 2x1 ≥ 0, x2 ≥ 0

Figure: Projection of central path on (x1, x2)


Example: Central Path (Dual)

Max 3y1 + y2 + 2y3

s.t. y1 + y2 ≤ 1y1 − y2 + y3 ≤ 1y1 ≤ 0y2 ≤ 0y3 ≤ 0

Figure: Projection of central path on (y1, y2)


Newton Method for LP

Given (x0, y0, s0) ∈ feas+(LPD) with µ0 = (s0)T x0

n and 0 ≤ γ ≤ 1, find(dx, dy, ds) satisfying

A(x0 + dx) = bAT (y0 + dy) + (s0 + ds) = cΛx0+dx(s0 + ds) = γµ0ex0 + dx >Rn

+0, s0 + ds >Rn

+0,

After linearization A 0 00 AT I

Λs0 0 Λx0

dxdyds

=

00

γµ0e− Λx0Λs0e

x0 + dx >Rn

+0, s0 + ds >Rn

+0,

Directly solve the equation is not easy.


Newton Method for LP

Linear scaling: Given a positive diagonal matrix D ∈ Rn×n,

A = AD, x0 = D−1x0, s0 = Ds0, c = Dc

A 0 00 AT I

Λs0 0 Λx0

dxdyds

=

00

γµ0e− Λx0Λs0e

x0 + dx >Rn

+0, s0 + ds >Rn

+0,

• D = Λx0 : x0 = e ⇒ x0 + dx >Rn+

0, ∀‖dx‖2 < 1 (Primal)

• D = Λ−1s0 : s0 = e ⇒ s0 + ds >Rn

+0, ∀‖ds‖2 < 1 (Dual)

• D = Λ1/2x0 Λ

−1/2s0 : v0 = x0 = s0 = Λ

1/2x0 Λ

1/2s0 e (Primal-dual)


Primal-Dual Interior-Point Method for LP

D = Λ1/2x0 Λ

−1/2s0 : A 0 0

0 AT II 0 I

dxdyds

=

00

γµ0Λ−1v0 e− v

0

x0 + dx >Rn

+0, s0 + ds >Rn

+0,

One can solveAAT dy = −A(γµ0Λ−1

v0 e− v0)

And then solve ds and dx:

ds = −AT dydx = −ds + γµ0Λ−1

v0 e− v0


Newton Method for SDP

Given (X0, y0, S0) ∈ feas+(SDPD) with µ0 = S0•X0

n and 0 ≤ γ ≤ 1, find(4X, dy,4S) satisfying

A(X0 +4X) = bA∗(y0 + dy) + (S0 +4S) = C(X0 +4X)(S0 +4S) = γµ0IX0 +4X � 0, S0 +4S � 0

After linearization

A4X = 0A∗dy + 4S = 0

4XS0 + X04S = γµ0I −X0S0

X0 +4X � 0, S0 +4S � 0.

Directly solve the equation is not easy.


Newton Method for SDP

Linear transformation: Given an invertible matrix L ∈ Rn×n, letA = (A1, . . . , Am), Ai = LTAiL for i = 1, . . . ,m.X0 = L−1X0L−T , S0 = LTS0L, C = LTCL.

A4X = 0

A∗dy + 4S = 0

4XS0 + X04S = γµ0I − X0S0

X0 +4X � 0, S0 +4S � 0

• L = (X0)1/2: X0 = I ⇒ X0 +4X � 0, ∀‖4X‖F < 1 (Primal)• L = (S0)−1/2: S0 = I ⇒ S0 +4S � 0, ∀‖4S‖F < 1 (Dual)• LLT = (S0)−1/2[(S0)1/2X0(S0)1/2]1/2(S0)−1/2:V 0 = X0 = S0 (Primal-dual)


Primal-Dual Interior-Point Method for SDP

LLT = (S0)−12 [(S0)

12X0(S0)

12 ]

12 (S0)−

12 :A 0 0

0 A∗ II 0 I

4Xdy4S

=

00

γµ0(V 0)−1 − V 0

X0 +4X � 0, S0 +4S � 0

One can solveAA∗dy = −A(γµ0(V 0)−1 − V 0)

And then solve 4S and 4X:

4S = −A∗dy4X = −4S + γµ0(V 0)−1 − V 0


Neighborhood of Central Path for LP

Notice that x0 = s0 = v0

• Distance to central path: u >Rn+0

δ(u) = ‖e− n

uTuΛuu‖2

• Neighborhood of the central path

N2(β) = {u|u >Rn+

0, δ(u) ≤ β}

N−∞(β) = {u|u >Rn+

0,Λuu ≥Rn+

(1− β)uTu

nI}


Examples: N2(12) and N−∞(12)

Figure: Neighborhood N2( 12) Figure: Neighborhood N−∞( 1

2)


Finding Step Length for LP

x0 + αdxs0 + αds

scaling back−−−−−−−−−−−→

[x1

s1

] new scaling−−−−−−−−−−→ v1 = x1 = s1

LemmaFor any 0 ≤ α ≤ 1,

µ1 =‖v1‖22n

=(x0 + αdx)T (s0 + αds)

n= (1− α+ γα)µ0



LemmaIf δ(v0) < 1 and α satisfies x0 + αdx >Rn

+0 and s0 + αds >Rn

+0, then

(1− α+ γα)δ(v1) ≤ (1− α)δ(v0) +α2

2

(γ2δ(v0)2

1− δ(v0)+ n(1− γ)2

)Proof:

µ1δ(v1) = µ1‖e− 1

µ1Λv1v

1‖2

= ‖(1− α+ γα)µ0e− Λ(v0+αdx)(v0 + αds)‖2

≤ ‖(1− α)µ0(e− 1

µ0Λv0v

0)‖2 + ‖α2Λdx ds‖2

≤ (1− α)µ0δ(v0) +α2

2‖dx + ds‖22

= (1− α)µ0δ(v0) +α2

2(γ2‖µ0Λ−1

v0 e− v0‖22 + (1− γ)2nµ0)

≤ (1− α)µ0δ(v0) +α2

2(µ0γ2δ(v0)2

1− δ(v0)+ (1− γ)2nµ0)



LemmaIf v0 ∈ N2(β) with β = 1

2 , γ = 11+1/

√2n

and α = 1, then

(i) v1 ∈ N2(β)

(ii) x1 • s1 = x1 • s1 = ‖v1‖22 = γµ0


Path Following Algorithm for LP

Step 1: (Initialization)ε > 0, (x0, y0, s0) with v0 ∈ N (β), where β = 1

2 .Set k = 0, γ = 1

1+1/√

2n, and α = 1.

Step 2: Solve the Newton system introduced above and get (dx, dy, ds).Set xk+1 = xk + αdx

yk+1 = yk + αdysk+1 = sk + αds

with vk+1 = Λ1/2

xk+1Λ1/2

sk+1e.Set k = k + 1.

Step 3: If xk • sk < ε, stop. Otherwise, go to Step 2.


Complexity for LP

TheoremGiven the above settings, we have

(i) vk ∈ N2(β), k = 0, 1, 2, . . ..(ii) The algorithms stops in

O(√n log

x0 • s0

ε)

steps and output a primal-dual solution satisfying

xk • sk < ε


Neighborhood of Central Path for SDP

Notice that X0 = S0 = V 0

• Distance to central path: U ∈ Sn+ and U � 0

δ(U) = ‖I − n

I • U2U2‖F , with U2 = UU

• Neighborhood of the central path

N2(β) = {U |U � 0, δ(U) ≤ β}

N−∞(β) = {U |U � 0, U2 � (1− β)I • U2

nI}


Finding Step Length for SDP

X0 + α4XS0 + α4S

scaling back−−−−−−−−−−−→

[X1

S1

] new scaling−−−−−−−−−−→ V 1 = X1 = S1

LemmaFor any 0 ≤ α ≤ 1,

µ1 =‖V 1‖2Fn

=tr[(X0 + α4X)(S0 + α4S)]

n= (1− α+ γα)µ0.



LemmaFor any square matrix U , we have

tr(U2) = ‖U + UT

2‖2F − ‖

U − UT

2‖2F ≤ ‖

U + UT

2‖2F

LemmaSuppose δ(V 0) < 1 and α ≥ 0 satisfies X0 + α4X � 0 andS0 + α4S � 0. Let

W =(X0 + α4X)(S0 + α4S) + ((X0 + α4X)(S0 + α4S))T

2

thenW = (1− α)(V 0)2 + αγµ0I + α24X4S +4S4X

2

andδ(V 1)2 ≤ ‖I − 1

µ1W‖2F



LemmaSuppose δ(V 0) < 1 and α ≥ 0 satisfies X0 + α4X � 0 andS0 + α4S � 0. Then

(1− α+ γα)δ(V 1) ≤ (1− α)δ(V 0) +α2

2

(γ2δ(V 0)2

1− δ(V 0)+ n(1− γ)2

)Proof

µ1δ(V 1) ≤ (1− α)µ0δ(V 0) + α2‖4X4S+4S4X2 ‖F

≤ (1− α)µ0δ(V 0) + α2

2 ‖4X +4S‖2F= (1− α)µ0δ(V 0) + α2

2 (γ2‖µ0(V 0)−1 − V 0‖2F + (1− γ)2nµ0)

≤ (1− α)µ0δ(V 0) + α2µ0

2

(γ2δ(V 0)2

1−δ(V 0) + n(1− γ)2)



LemmaIf V 0 ∈ N2(β) with β = 1

2 , γ = 11+1/

√2n

and α = 1, then

(i) V 1 ∈ N2(β)

(ii) X1 • S1 = X1 • S1 = ‖V 1‖2F = γµ0


Path Following Algorithm for SDP

Step 1: (Initialization)ε > 0, (X0, y0, S0) with V 0 ∈ N (β), where β = 1

2 .Set k = 0, γ = 1

1+1/√

2n, and α = 1.

Step 2: Solve the equation system introduced above and get(4X, dy,4S).Set Xk+1 = Xk + α4X

yk+1 = yk + αdySk+1 = Xk + α4S

with V k+1 = Xk+1 = Sk+1.Set k = k + 1.

Step 3: If Xk • Sk < ε, stop. Otherwise, go to Step 2.


Complexity

TheoremGiven the above settings, we have

(i) V k ∈ N2(β), k = 0, 1, 2, . . ..(ii) The algorithms stops in

O(√n log

X0 • S0

ε)

steps and output a primal-dual solution satisfying

Xk • Sk < ε


Example: Path Following Algorithm

min x1 + x2

s.t. x1 + x2 ≤ 3x1 − x2 ≤ 1x2 ≤ 2x1 ≥ 0, x2 ≥ 0

Figure: Path following algorithm with β = 1/2


Initialization and Improve the Performance

Initialization

• Big-M Method• Two-Phase Method• Self-Dual Embedding Method

Different Path-Following Methods

• Short Step Algorithm• Long Step Algorithm• Predictor-Corrector Algorithm• Largest Step Algorithm

Reference: Handbook of Semidefinite Programming: Theory, Algorithms,and Applications, edited by Wolkowicz H., Saigal R. and VandenbergheL., Kluwer Academic Publisher: Norwell, MA USA 2000


CVX Solution Software - Example 1


LCoP Part V – Modeling and Applications

Shu-Cherng Fang




Modeling and Applications

Content• Weber Problem• Matrix Optimization• Approximating Solutions of Linear Equations• Minimum of a Univariate Polynomial of Degree 2n

• Stochastic Queue Location Problem• Conic Reformulations of QCQP and Extensions• Robust Optimization


Weber Problem

In 1909, the German economist Alfred Weber introduced the problem offinding a best location for the warehouse of a company, such that the totaltransportation cost to serve the customers is minimum. Suppose thatthere are m customers needing to be served. Let the location ofcustomer i be ai ∈ R2, i = 1, . . . ,m. Suppose that customer may havedifferent demands, to be translated as weight ωi for customer i,i = 1, . . . ,m. Denote the desired location of the warehouse to be x.Then, the optimization problem becomes

Minm∑i=1

ωiti

s.t.

[x− aiti

]∈ L3, i = 1, . . . ,m


Matrix Optimization

Given A0, A1, . . . , Am, determine if there is y ∈ Rm such that

A0 +∑mi=1 yiAi � 0

which is equivalent to minimizing

λmax(A0 +∑mi=1 yiAi)

Notice that

t ≥ λmax(A0 +∑mi=1 yiAi) ⇔ tI −A0 −

∑mi=1 yiAi � 0

Equivalent Problem (SDP)

Min ts.t. tI −A0 −

∑mi=1 yiAi � 0

t ∈ R, y ∈ Rm


Approximating Solutions of LinearEquations

Given A ∈ Rm×n, b ∈ Rm, solve

Ax = b

Approximation:• l1 norm:

minx‖Ax− b‖1 ⇔ min

∑mi=1 ti

s.t. −ti ≤ Ai·x− bi ≤ ti, i = 1, . . . ,m.

• l2 norm:

minx‖Ax− b‖2 ⇔

min t

s.t.

[Ax− b

t

]∈ Lm+1


Approximating Solutions of LinearEquations

• l∞ norm:

minx‖Ax− b‖∞ ⇔ min t

s.t. −t ≤ Ai·x− bi ≤ t, i = 1, . . . ,m.

• Logarithm approximation: (b >Rm+

0)

minx

max1≤i≤m

| log(Ai·x)−log bi| ⇔min ts.t. 1/t ≤ Ai·x/bi ≤ t, i = 1, . . . ,m,

t > 0

⇔

min t

s.t.

t−Ai·x/bi 0 00 Ai·x/bi 10 1 t

� 0


Minimum of a Univariate Polynomial

Consider the problem of finding the minimum of a univariate polynomialof degree 2n:

Min x2n + a1x2n−1 + · · ·+ a2n−1x+ a2n

s.t. x ∈ R

This problem is equivalent to

Max ts.t. x2n + a1x

2n−1 + · · ·+ a2n−1x+ a2n − t ≥ 0 for all x ∈ R


Minimum of a Univariate Polynomial

It is well known that a univariate polynomial is nonnegative over the realdomain if and only if it can be written as sum of squares (SOS), which isequivalent to saying that there must be a positive semidefinite matrixX ∈ Sn+1

+ such that

x2n+a1x2n−1+· · ·+a2n−1x+a2n−t = (1, x, x2, · · · , xn)X(1, x, x2, · · · , xn)T .

Hence, this problem can be cast as an SDP:

Max ts.t. a2n − t = X11

a2n−k =∑

i+j=k+2

Xij , k = 1, . . . , 2n− 1

X(n+1),(n+1) = 1X ∈ Sn+1

+


Stochastic Queue Location Problem

BackgroundSuppose there are m potential customers to serve in the region.Customers’ demands are random, and once a customer calls for service,then the server in the service center will need to go to the customer toprovide the required service. In case the server is occupied, then thecustomer will have to wait. The goal is to find a good location for theservice center in order to minimize the expected waiting time of service.



Assumptions and notationsSuppose that the service calls from the customer are identicallyindependent distributed, and the demand process follows the Poissondistribution with overall arrival rate λ, and the probability that any servicecall is from customer i is assumed to be pi for i = 1, . . . ,m. The queueingprinciple is First Come First Service, and there is only one server in theservice center. This model can be regarded as M/G/1 queue as inQueueing theory, and the expected service time, including waiting timeand traveling, can be explicitly computed. To this end, denote the velocityof the server to be v, and the location of customer i is ai, i = 1, . . . ,m,and the location of the service center to be x.



Problem formulationThe expected waiting time for customer i is given by

ωi(x) =

(2λ/v2)m∑j=1

pj‖x− aj‖22

1− (2λ/v)m∑j=1

pj‖x− aj‖22+

1

v‖x− ai‖2,

where the first term in the expected term is the expected waiting time forthe server to be free and the second the term is the waiting time for theserver to travel after his departure at the service center.



Observing the fact that

‖x‖22/s ≤ t, s > 0⇐⇒∣∣∣∣∣∣∣∣[ x

t−s2

]∣∣∣∣∣∣∣∣2

≤ t+ s

2

We can formulate this problem as an SOCP:

Min (2mλ/v2)m∑i=1

piti + (1/v)m∑i=1

t0i

s.t.

[x− ait0i

]∈ L3,

x− aiti−s2

ti+s2

∈ L4, i = 1, ...,m

s ≤ 1− (2λ/v)m∑i=1

pisi,

[x− aisi

]∈ L3, i = 1, ...,m.


Quadratically Constrained QuadraticProgramming (QCQP)

QCQP:

min xTA0x+ 2bT0 x+ c0

s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m,

Ai ∈ Sn, bi ∈ Rn, ci ∈ R, i = 0, 1, . . . ,m.

Homogeneous QCQP(HQCQP):

min xTA0x

s.t. xTAix+ ci ≤ 0, i = 1, . . . ,m


Convex QCQP

Convex QCQP: A0, . . . , Am � 0Assume rank(Ai) = di, then Ai = BTi Bi, Bi ∈ Rdi×n, i = 0, . . . ,m.

xTAix+ 2bTi x+ ci ≤ 0 ⇔

Bix−1/2− bTi x− ci/21/2− bTi x− ci/2

∈ Ldi+2

Reformulating Convex QCQP as SOCP:

min u

s.t.

B0x−1/2− bT0 x+ u/2− ci/21/2− bT0 x+ u/2− ci/2

∈ Ld0+2


∈ Ldi+2 i = 1, . . . ,m


SDP Relaxation for QCQP

Recall that

xTAix+ 2bTi x+ ci =

[1x

]T [ci bTibi Ai

] [1x

]=

[ci bTibi Ai

]•[

1 xT

x xxT

]SDP Relaxation

min

[c0 bT0b0 A0

]• Y

s.t.

[ci bTibi Ai

]• Y ≤ 0, i = 1, . . . ,m

Y11 = 1Y � 0


SDP Relaxation for HQCQP

Recall thatxTAix = Ai • xxT

SDP Relaxation

min A0 •Xs.t. Ai •X ≤ −ci, i = 1, . . . ,m

X � 0


Exact Solutions from SDP Relaxation

TheoremWhen m = 1, if the optimal solution of the SDP relaxation for QCQPexists, then there exists a rank one optimal solution Y ∗ of the SDP

relaxation for QCQP. Furthermore, let Y ∗ =

[1 (x∗)T

x∗ x∗(x∗)T

], then x∗ is

optimal for QCQP.



LemmaLet G be a symmetric matrix and X be a positive semidefinite matrix withrank d. Suppose that G •X ≤ (=)0. Then there exists a rank-onedecomposition X =

∑di=1 xix

Ti such that xTi Gxi ≤ (=)0, i = 1, . . . , d.

TheoremWhen m = 2, suppose there exists x such that xTAix+ 2bTi x+ ci < 0,i = 1, 2, and there exists y1 ≥ 0, y2 ≥ 0 such that[c0 bT0b0 A0

]+ y1

[c1 bT1b1 A1

]+ y2

[c2 bT2b2 A2

]� 0. If, for the optimal solution Y ∗,

either[c1 bT1b1 A1

]• Y ∗ < 0 or

[c2 bT2b2 A2

]• Y ∗ < 0, then there is no gap

between QCQP and its SDP relaxation and at least one optimal solutioncan be obtained from the rank-one decomposition of Y ∗.



Theorem(HQCQP)When m = 2, suppose there exists x, such that xTAix+ ci < 0, i = 1, 2,and there exists y1 ≥ 0 and y2 ≥ 0 such that A0 + y1A1 + y2A2 � 0. Thenthere is no gap between HQCQP and its SDP relaxation and at least oneoptimal solution can be obtained from the rank-one decomposition of X∗.


One Extension of QCQP

Ext-QCQP

min xTA0x+ 2bT0 x+ c0

s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m,(x21, . . . , x

2n)T ∈ X

(Ext-QCQP)

where X is a closed convex set.SDP Relaxation:

min

[c0 bT0b0 A0

]• Y

s.t.

[ci bTibi Ai

]• Y ≤ 0, i = 1, . . . ,m

(Y22, Y33, . . . , Yn+1n+1)T ∈ XY11 = 1Y � 0



TheoremSuppose there is σ ∈ {−1, 1}n+1, such that all the off-diagonal elements

of Λσ

[ci bTibi Ai

]Λσ, 0 ≤ i ≤ m, are nonpositive. If Y ∗ is an optimal solution

for the SDP relaxation for Ext-QCQP, then

x∗ = σ1(σ2√Y ∗22, σ3

√Y ∗33, . . . , σn+1

√Yn+1n+1)T

is optimal for Ext-QCQP.


More Results on Ext-QCQP

A Special Case of Ext-QCQP

min xTAx

s.t. (x21, . . . , x2n)T ∈ X


min A •Xs.t. (X11, X22, . . . , Xnn)T ∈ X

X � 0


Approximation by Randomized Algorithm

• Suppose Z � 0, Zii = 1, i = 1, . . . , n. If ξ ∼ N(0, Z) andσ ∈ {−1, 1}n with

σi =

{1 ξi ≥ 0−1 ξi < 0

then E(σσT ) = 2π arcsinZ.

• Suppose Z � 0, Zii = 1, i = 1, . . . , n. Then arcsinZ � Z � 0.• 2

π arcsin t ≤ 1− α+ αt, ∀t ∈ [−1, 1] with α = 0.87856 . . ..



Given X � 0, let Z = D+XD+ + D, where D, D+, D are diagonalmatrices defined by

Dii =√Xii, D+

ii =

{(√Xii)

−1 Xii 6= 00 Xii = 0

, Dii =

{0 Xii 6= 01 Xii = 0

• X = DZD

• E(A • (DσσTD)) = A • (DE(σσT )D) = A • ( 2πD arcsin(Z)D).



TheoremSuppose A � 0, and X∗ is the minimizer of SDP relaxation. Let Z, D,D+, and D be define from X∗, and ξ ∼ N(0, Z) with corresponding σ.Then

E(A • (DσσTD)) ≤ 2

πmin{xTAx|(x21, . . . , x2n)T ∈ X}

Proof:E(A • (DσσTD))

= A • ( 2πD arcsin(Z)D)

≤ A • ( 2πDZD)

= 2πA •X

∗

≤ 2π min{xTAx|(x21, . . . , x2n)T ∈ X}



TheoremAssume that A � 0, Aij ≥ 0 for all i 6= j, and X∗ is the minimizer of SDPrelaxation. Let Z, D, D+ and D be defined from X∗ and ξ ∼ N(0, Z) withcorresponding σ. Then

E(A • (DσσTD)) ≤ αmin{xTAx|(x21, . . . , x2n)T ∈ X}

Proof:E(A • (DσσTD))

= A • ( 2πD arcsin(Z)D)

=∑i 6=j AijDiiDjj(

2π arcsinZij) +

∑ni=1AiiD

2ii

≤∑i 6=j AijDiiDjj(1− α+ αZij) +

∑ni=1AiiD

2ii

= (1− α)∑i,j AijDiiDjj + αA •X∗

≤ αA •X∗ ≤ αmin{xTAx|(x21, . . . , x2n)T ∈ X}Conic Programming 26 / 34

Example: MAX CUT Problem

Given a graph G = (V,E,W ), find an optimal partition of the node set intotwo subsets V1 and V2 such that the weighted cut is maximized.

max∑ni=1

∑nj=1 wij

1−xixj

2

s.t. x2i = 1, i = 1, . . . , n(MAX − CUT )

Reformulation as Ext-QCQP:

Aij =

{wij i 6= j

−∑k 6=i wik i = j

, X = {e}

TheoremIf wij ≥ 0, ∀i 6= j. Then the expected value of randomized algorithm is atleast α ≈ 0.878 times the value of the maximum cut.


Robust Optimization

min f0(x)

s.t. fi(x, ui) ≤ 0

∀ui ∈ Ui, i = 1, . . . ,m.

Motivation

• The parameters are inexact.• The parameters cannot be foreseen.• The parameters may vary with time and other environment factors.


Example: Robust Linear Program

min cTx

s.t. aTi x ≤ bi

∀ai ∈ {a0i +∑pj=1 uja

ji | ‖u‖2 ≤ 1},

i = 1, . . . ,m

For any x

maxai aTi x = maxu(a0i )

Tx+∑pj=1 uj(a

ji )Tx

= (a0i )Tx+ ‖((a1i )Tx, . . . , (a

pi )Tx)T ‖2


Example: Robust Linear Program

Therefore

aTi x ≤ bi,∀ai ⇔ ‖((a1i )Tx, . . . , (api )Tx)T ‖2 ≤ bi − (a0i )

Tx

Equivalent SOCP

min cTx

s.t.

(a1i )

Tx...

(api )Tx

bi − (a0i )Tx

∈ Lp+1, i = 1, . . . ,m


Robust Convex QCQP

min fTx

s.t. xTATAx ≤ 2bTx+ c

∀(A, b, c) ∈ {(A0, b0, c0) +∑pj=1 uj(A

j , bj , cj) | ‖u‖2 ≤ 1},

Let U(x) = (A0x,A1x, . . . , Apx) and

V (x) =

c0 + 2(b0)Tx 1

2c1 + (b1)Tx · · · 1

2cp + (bp)Tx

12c

1 + (b1)Tx 0...

. . .12cp + (bp)Tx 0


Robust Convex QCQP

xTATAx ≤ 2bTx+ c

⇔[

1u

]TUT (x)U(x)

[1u

]−[

1u

]TV (x)

[1u

]≤ 0,∀‖u‖2 ≤ 1

Lemma (S-lemma)Let P and Q be symmetric matrices and yTPy > 0 for some y. Then theimplication

zTPz ≥ 0 ⇒ zTQz ≥ 0

is valid if and only if Q− λP � 0 for some λ ≥ 0.


Robust Convex QCQP

From S-lemma,[1u

]TUT (x)U(x)

[1u

]−[

1u

]TV (x)

[1u

]≤ 0,∀‖u‖2 ≤ 1

⇔ V (x)− UT (x)U(x)− λ[1−I

]� 0,∃λ ≥ 0

⇔

V (x)− λ[1−I

]UT (x)

U(x) I

� 0,∃λ ≥ 0

(by Schur complementary theorem)


Robust Convex QCQP

Equivalent SDP

min fTx

s.t.

c0 + 2(b0)Tx− λ 1

2c1 + (b1)Tx · · · 1

2cp + (bp)Tx (A0x)T

12c

1 + (b1)Tx λ (A1x)T

.... . .

...12cp + (bp)Tx λ (Apx)T

A0x A1x · · · Apx I

� 0


LCoP Part VI – Recent Research

Shu-Cherng Fang




Recent Research

Content• Quadratically Constrained Quadratic Programming

(QCQP)


Quadratically Constrained QuadraticProgramming (QCQP)

QCQP:

Min xTA0x+ 2bT0 x+ c0

s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m

Ai ∈ Sn, bi ∈ Rn, ci ∈ R, i = 0, 1, . . . ,m.

Homogeneous QCQP(HQCQP):

Min xTA0x

s.t. xTAix+ ci ≤ 0, i = 1, . . . ,m


Convex QCQP

Convex QCQP: A0, . . . , Am � 0Assume rank(Ai) = di, then Ai = BTi Bi, Bi ∈ Rdi×n, i = 0, . . . ,m.

xTAix+ 2bTi x+ ci ≤ 0 ⇔


∈ Ldi+2

Reformulating Convex QCQP as SOCP:

Min u

s.t.

B0x−1/2− bT0 x+ u/2− c0/21/2− bT0 x+ u/2− c0/2

∈ Ld0+2


∈ Ldi+2 i = 1, . . . ,m


SDP Relaxation for QCQP

Recall that

xTAix+ 2bTi x+ ci =

[1x

]T [ci bTibi Ai

] [1x

]=

[ci bTibi Ai

]•[

1 xT

x xxT

]

Let Y =

[1x

] [1x

]T=

[1 xT

x xxT

]∈ Sn+1

+ , we have SDP Relaxation

Min

[c0 bT0b0 A0

]• Y

s.t.

[ci bTibi Ai

]• Y ≤ 0, i = 1, . . . ,m

Y11 = 1

Y � 0, rank(Y ) = 1

Note: No convexity is assumed.


SDP Relaxation for HQCQP

Recall thatxTAix = Ai • xxT

Let X = xxT ∈ Sn+, we have SDP Relaxation

Min A0 •Xs.t. Ai •X ≤ −ci, i = 1, . . . ,m

X � 0, rank(X) = 1



TheoremWhen m = 1, if the optimal solution of the SDP relaxation for QCQPexists, then there exists a rank one optimal solution Y ∗ of the SDP

relaxation for QCQP, and, furthermore, let Y ∗ =

[1 (x∗)T

x∗ x∗(x∗)T

], then x∗

is optimal for QCQP.



LemmaLet G be a symmetric matrix and X be a positive semidefinite matrix withrank d. Suppose that G •X ≤ (=)0. Then there exists a rank-onedecomposition X =

∑di=1 xix

Ti such that xTi Gxi ≤ (=)0, i = 1, . . . , d.

TheoremWhen m = 2, assume there exists x such that xTAix+ 2bTi x+ ci < 0,i = 1, 2, and there exists y1 ≥ 0, y2 ≥ 0 such that[c0 bT0b0 A0

]+ y1

[c1 bT1b1 A1

]+ y2

[c2 bT2b2 A2

]� 0. If, for the optimal solution Y ∗,

either[c1 bT1b1 A1

]• Y ∗ < 0 or

[c2 bT2b2 A2

]• Y ∗ < 0, then there is no gap

between QCQP and its SDP relaxation, and at least one optimal solutioncan be obtained from the rank-one decomposition of Y ∗.



Theorem(HQCQP)When m = 2, suppose there exists x, such that xTAix+ ci < 0, i = 1, 2,and there exists y1 ≥ 0 and y2 ≥ 0 such that A0 + y1A1 + y2A2 � 0. Thenthere is no gap between HQCQP and its SDP relaxation, and at least oneoptimal solution can be obtained from the rank-one decomposition of X∗.


One Extension of QCQP

Ext-QCQPMin xTA0x+ 2bT0 x+ c0

s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m(x2

1, . . . , x2n)T ∈ X

(Ext-QCQP)


Min

[c0 bT0b0 A0

]• Y

s.t.

[ci bTibi Ai

]• Y ≤ 0, i = 1, . . . ,m

(Y22, Y33, . . . , Y(n+1),(n+1))T ∈ X

Y11 = 1

Y � 0, rank(Y ) = 1



TheoremSuppose there is σ ∈ {−1, 1}n+1, such that all the off-diagonal elements

of Λσ

[ci bTibi Ai

]Λσ, 0 ≤ i ≤ m, are nonpositive. Here

(Λσ)ij =

{σi i = j0 i 6= j

If Y ∗ is an optimal solution for the SDP relaxation for Ext-QCQP, then

x∗ = σ1(σ2

√Y ∗22, σ3

√Y ∗33, . . . , σn+1

√Y(n+1),(n+1))

T

is optimal for Ext-QCQP.


More Results on Ext-QCQP

A Special Case of Ext-QCQP

Min xTAx

s.t. (x21, . . . , x

2n)T ∈ X


Min A •Xs.t. (X11, X22, . . . , Xnn)T ∈ X

X � 0

The previous theorem is satisfied automatically. Hence the optimalsolution for this special case of Ext-QCQP can be obtained by its SDPrelaxation.


Recent Results on QCQP

Content• Introduction• KKT solution and Conic Relaxation for QCQP• Adaptive Approximation to QCQP


Introduction

QCQP:

Min f(x) = xTA0x+ 2bT0 x+ c0

s.t. gi(x) = xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m,

Ai ∈ Sn, bi ∈ Rn, ci ∈ R, i = 0, 1, . . . ,m.Feasible domain:

feas(QCQP) = {x ∈ Rn|gi(x) ≤ 0, i = 1, . . . ,m}


Introduction

Previously we have learnt:• Equivalent SOCP for convex QCQP• SDP relaxation for QCQP• m = 1: optimal rank one solution from SDP relaxation• m = 2: optimal solution from non-binding solution of SDP relaxation• m = 2: optimal solution from SDP relaxation for HQCQP• Optimal solution for Ext-QCQP with all nonpositive off-diagonal

coefficients


Recall

• Nonnegative quadratic functions over F

f(x) = xTAx+ 2bTx+ c ≥ 0,∀x ∈ F

f ⇔[c bT

b A

]• DF = {

[c bT

b A

]∈ Sn+1|

[1x

]T [c bT

b A

] [1x

]≥ 0,∀x ∈ F} is a closed,

convex cone, containing Sn+1+ .

• D∗F = cl cone{[

1 xT

x xxT

]|x ∈ F

}• (D∗F )∗ = DF and (DF )∗ = D∗F• F = Rn =⇒ DF = Sn+1

+ , D∗F = Sn+1+

• F = Rn+ =⇒ DF = Cn+1, D∗F = C∗n+1


Question

How to use the cone of nonnegative quadratic functions to solve QCQP?

• Relation to KKT solution• Adaptively approximation


KKT solution and Conic Relaxation forQCQP

Ideas:


Constraint Qualification: LICQ

Active setGiven a feasible solution x, the active set I(x) is defined as

I(x) = {1 ≤ i ≤ m | gi(x) = 0}.

Linear independence constraint qualification (LICQ)Given a feasible solution x, if the vectors ∇gi(x), i ∈ I(x), are linearlyindependent, then the LICQ condition holds at x.


Lagrangian Function

Define

A(λ) = A0 +m∑i=1

λiAi, b(λ) = b0 +m∑i=1

λibi, c(λ) = c0 +m∑i=1

λici.

Lagrangian function of QCQP

L(x, λ) = f(x) +m∑i=1

λigi(x)

= xTA(λ)x+ 2bT (λ)x+ c(λ)

where λ ∈ Rm+ .


Karush-Kuhn-Tucker Conditions

Given a feasible solution x∗, if it satisfies the LICQ and is a localminimum, then there exists a Lagrangian multiplier vector λ∗ such that(x∗, λ∗) is a KKT solution satisfying

∇xL(x∗, λ∗) = 2A(λ∗)x∗ + 2b(λ∗) = 0,

λ∗i gi(x∗) = λ∗i ((x

∗)TAix∗ + 2bTi x

∗ + ci) = 0,

gi(x∗) = (x∗)TAix

∗ + 2bTi x∗ + ci ≤ 0, λ∗i ≥ 0, i = 1, . . . ,m.


“Copositive Cone” w.r.t. the FeasibleDomain (Cone of Nonnegative QuadraticFunctions)

Define

Dn+1 =

{U ∈ Sn+1

∣∣∣∣ U • [ 1 xT

x xxT

]≥ 0,∀x ∈ feas(QCQP)

}D∗n+1 = cl cone

{X =

[1x

] [1x

]T ∣∣∣∣ x ∈ feas(QCQP)

}


Sufficient Conditions

Lemma (Positive semidefiniteness condition).Let (x∗, λ∗) be a KKT solution. If A(λ∗) ∈ Sn+, then x∗ is a globally optimalsolution of QCQP.

Theorem (Copositiveness condition)Let (x∗, λ∗) be a KKT solution. If M(x∗, λ∗) ∈ Dn+1, then x∗ is an optimalsolution of the QCQP problem, where

M(x∗, λ∗) =

[c(λ∗)− f(x∗) bT (λ∗)

b(λ∗) A(λ∗)

].


Recall


An Equivalent Conic ProgrammingProblem of QCQP

Min

[c0 bT0b0 A0

]• Y

s.t. Y11 = 1,Y ∈ D∗n+1.

(CoP)

TheoremCoP and QCQP have the same optimal objective value.

ProofSee Sturm and Zhang’s paper “On Cones of Nonnegative QuadraticFunctions”.


Adding Valid Inequality Constraints

CoP1:

Min

[c0 bT0b0 A0

]• Y

s.t.

[ci bTibi Ai

]• Y ≤ 0,

i = 1, . . . ,m,

Y11 = 1,Y ∈ D∗n+1.

CoD1:

Max σ

s.t.

[c(λ)− σ bT (λ)b(λ) A(λ)

]∈ Dn+1,

λ ≥Rm+

0.

TheoremIf QCQP is bounded below, then the optimal objective values of QCQP,CoP1 and CoD1 are equal.


Finding the Lagrangian Multiplier Vector λ∗

AssumptionThere exists a KKT solution (x∗, λ∗) satisfying the LICQ condition andcopositiveness condition.

LemmaUnder the copositiveness condition assumption, (σ∗, λ∗) = (f(x∗), λ∗) isan optimal solution of CoD1. Furthermore, if the LICQ assumption holds,then for any (σ, λ) being an optimal solution of the CoD1 problem, wehave λ ≤Rn

+λ∗.


Finding the Lagrangian Multiplier Vector λ∗

TheoremLet (x∗, λ∗) be a KKT solution. If it satisfies the LICQ condition and thecopositiveness condition, then λ∗ is the unique optimal solution of themaximization problem CoD2

Max∑mi=1 λi

s.t.

[c(λ)− σ∗ bT (λ)b(λ) A(λ)

]∈ Dn+1,

λ ≥Rn+

0,

(CoD2)

where σ∗ is the optimal objective value obtained by solving CoD1.Furthermore, if A(λ∗) is invertible, then

x∗ = −A−1(λ∗)b(λ∗).


Relaxing the Conic Constraint

Choose a computable cone Kn+1 such that Sn+1+ ⊂ Kn+1 ⊂ Dn+1.

CoD1’:Max σ

s.t.

[c(λ)− σ bT (λ)b(λ) A(λ)

]∈ Kn+1,

λ ≥Rn+

0.

CoD2’:Max

∑mi=1 λi

s.t.

[c(λ)− σ∗ bT (λ)b(λ) A(λ)

]∈ Kn+1,

λ ≥Rn+

0.


Algorithm (QCQP)

STEP 1: Given a QCQP problem, solve the corresponding CoD1’ problem andget the optimal value σ∗. If failed, then stop and the problem cannot besolved by the current scheme.

STEP 2: Solve CoD2’ to get the optimal λ∗.STEP 3: Compute x∗ = −A+(λ∗)b(λ∗).STEP 4: If (x∗, λ∗) is a KKT solution, then return x∗ as a global optimal solution

of QCQP with the objective value f(x∗) = σ∗. Otherwise, return σ∗ asa lower bound of QCQP.

Here A+ is the Moore-Penrose generalized inverse for a given squarematrix A ∈ Rn×n.


Example

Min f(x) = xTA0x+ 2bT0 x

s.t. g1(x) = xTA1x+ 2bT1 x+ c1 ≤ 0,

g2(x) = xTA2x+ 2bT2 x+ c2 ≤ 0,

where

A0 =

−2 10 210 4 12 1 −7

, b0 =

−12−656

,A1 =

2 0 00 −2 00 0 8

, b1 =

0−2−64

, c1 = 512,

A2 =

2 0 00 2 00 0 2

, b2 =

−20−16

, c2 = 128.


Example

• The feasible domain is a subset of R3+.

• If we choose K4 = S4+ (SDP relaxation), then the optimal value of the relaxed problem

is 222.88.• Instead, choose K4 = S4

+ +N4.

• x∗ = (0, 0, 8)T with f(x∗) = 224 and λ∗ = (1, 2)T .

M(x∗, λ∗) =

320 −16 −8 −40−16 4 10 2−8 10 6 1−40 2 1 5

=

320 −16 −8 −40−16 4 0 2−8 0 6 1−40 2 1 5

+

0 0 0 00 0 10 00 10 0 00 0 0 0

∈ S4+ +N4,

A0 + λ∗1A1 + λ

∗2A2 =

4 10 210 6 12 1 5

/∈ S3+.


Summary

• Equivalent conic programming problems.

• The copositiveness condition which ensures a KKT solution beingoptimal.

• The maximal property of the Lagrangian multiplier vector under theLICQ condition and the copositiveness condition.

• Conic programming relaxation.• A way to design polynomial time algorithm to solve a class of QCQP.


Adaptive Approximation to QCQP

Consider a box constrained quadratic program


s.t. x ∈ G = [0, 1]n(BQP)

This problem is in general NP-hard.

Is there any polynomial time approximation?

Find an ellipsoid cover F1 ∪ · · · ∪ Fk ⊃ G such that the problem


s.t. x ∈ F1 ∪ · · · ∪ Fk(BQPR)

satisfies opt(BQP) ≤ opt(BQPR) + ε.


Conic Reformulation

Let H =

[c0 bT0b0 A0

], BQP becomes

Min H • Y

s.t. Y11 = 1

Y ∈ D∗G

(BCoP)

and BQPR becomes

Min H • Y

s.t. Y11 = 1

Y ∈ D∗F1∪···∪Fk

(BCoPR)

Max σ

s.t. σ

[1 00 0

]+ S = H

S ∈ DF1∪···∪Fk


Conic Reformulation

Lemma

DF1∪···∪Fk= DF1 ∩ · · · ∩ DFk

and

D∗F1∪···∪Fk= D∗F1

+ · · ·+D∗Fk

Rewrite the relaxation BCoPR as

Min H • Y

s.t. Y11 = 1

Y = Y1 + · · · , YkYi ∈ D∗Fi

, i = 1, . . . , k

(BCRP)

What is DF1∪···∪Fkfor the dual problem of BCRP?


Conic Reformulation: Dual Problem

Lemma (Linear Matrix Inequalities (LMI))Suppose each ellipsoid Fi = {x ∈ Rn|xTAix+ 2bTi x+ ci 6 0} with someAi � 0 and Int(Fi) 6= ∅. Then, M ∈ DF1∪···∪Fk

if and only if the followingsystem is feasible:

M + λi

[ci bTibi Ai

]� 0, λi > 0 for i = 1, 2, ..., k.

Dual of BCoRP:Max σ

σ

[1 00 0

]+ S = H

S + λi

[ci bTibi Ai

]� 0, λi ≥ 0,

i = 1, . . . , k.

(BCRD)


Optimality Condition

(Y ∗, Y ∗1 , ..., Y∗k ) is feasible to BCRP,

(σ∗, λ∗, S∗) is feasible to BCRD,

S∗ · Y ∗i = 0 and λ∗i

[bi cTici Qi

]• Y ∗i = 0, i = 1, 2, ...k.


Some Properties

LemmaS∗ is on the boundary of DF1∪···∪Fk

.

Lemma (Rank-one decomposition)Y ∗ can be decomposed in the form of

Y ∗ =k∑i=1

ni∑j=1

µij

[1xij

] [1xij

]T

with xij ∈ Fi, µij > 0 and∑ki=1

∑ni

j=1 µij = Y ∗11 = 1.Moreover, [

1xij

]TS∗[

1xij

]= 0,∀xij


Improving The Relaxation

TheoremIf any xij decomposed from Y ∗ is feasible for BQP, then it is an optimalsolution of BQP.

However, in general, they are not feasible.

Let x = xij = arg mini,j

[1xij

]TH

[1xij

]with corresponding Fi.

Notice that for Fi, the feasible part is Fi ∩ G. If we could cover Fi ∩ G bysmaller sets, whose union does not contain x, then the solution of thenew cover may be improved.


Improving The Relaxation

For simplicity, the ellipsoids have to be constructed carefully.

DefinitionLet T = {x ∈ Rn|ui 6 xi 6 vi} be a rectangular set in Rn, define aquadratic function

gT (x) =

n∑i=1

(2xi − vi − ui)2

(vi − ui)2− n for x ∈ Rn,

and an ellipsoidal set FT = {x ∈ Rn|gT (x) 6 0}. Then we say FT is thecorresponding ellipsoid of T .


Algorithm

STEP 1: Set T = G and Nmax. Set k = 1 and let the cover be F1 = FT .STEP 2: Solve BCRP and BCRD defined by the cover F1 ∪ · · · ∪ Fk to obtain Y ∗

and (σ∗, λ∗, S∗). Set lk = σ∗.STEP 3: Decompose Y ∗ to obtain x and Fi.STEP 4: If x ∈ G, then x is optimal. Stop and return.

Otherwise, set T be the rectangle covered by Fi.Find i∗ = arg maxi{max(xi − 1,−xi)}.Partition T into T1 = {x ∈ T |xi∗ ≤ ui∗+vi∗

2} and

T2 = {x ∈ T |xi∗ ≥ ui∗+vi∗2}.

Drop Fk from the cover, and add FT1 and FT2 into the cover.Set k = k + 1.

STEP 5: If k < Nmax, then go to STEP 2. Otherwise return max{l1, ..., lk} as alower bound for BQP.


Convergence

TheoremLet l∗k = maxi=1,...,k{li} be the largest lower bound up to the kth iteration,then limk→+∞ l∗k = opt(BQP ).


Example

Min xTA0x+ 2bT0 x

s.t. x ∈ [0, 1]10

with

A0 =

54 24 −79 88 11 −109 −76 −14 250 8124 303 −24 −61 24 −6 −5 112 85 76−79 −24 72 28 40 −9 64 −25 25 −388 −61 28 −81 −132 16 71 7 148 4411 24 40 −132 −86 −2 69 −7 14 22−109 −6 −9 16 −2 153 −34 −52 24 65−76 −5 64 71 69 −34 −149 −108 −11 113−14 112 −25 7 −7 −52 −108 49 −21 −106250 85 25 148 14 24 −11 −21 22 −8081 76 −3 44 22 65 113 −106 −80 −179

and

b0 =1

2[84 − 89 10 − 54 30 − 60 49 74 171 − 19]T .


Example

The optimal value is -611.

0 10 20 30 40 50−670

−660

−650

−640

−630

−620

−610

Iterations

Low

er B

ound

s

Adaptive BoundSDP Bound


Summary

• Developed an adaptive relaxation for BQP.• It applies to convex constrained QCQP too.• It may be applied to general QCQP, but needs well-designed covers.• Other techniques such as RLT can be involved in the relaxation.


LCoP Part VII – Practical LCoP

Shu-Cherng Fang



December 2017


Practical LCoP

Content• Convex Cones and Properties• Different Forms of LCoP• SOCP Formulation• SDP Formulation



K, K1, K2 are convex cones in Rn.• (K∗)∗ = cl(K).• K1 ⊆ K2 ⇒ K∗2 ⊆ K∗1 .• K1 ∩K2, K1 ∪K2, K1 +K2, K1 ×K2 are all cones.• (K1 +K2)∗ = K∗1 ∩K∗2 .• K1, K2 closed⇒ K1 +K2 and K1 ×K2 are closed.• ri(K1 +K2) = ri(K1) + ri(K2).• ri(K1 ×K2) = ri(K1)× ri(K2).• The supporting hyperplane of K always contains the origin• If K is solid, then K∗ is pointed.• If K is pointed, then K∗ is solid.• If K is proper, then K∗ is proper.



K, K1, K2 are convex cones in Rn.• Rn+ ⊆ Rn, Ln ⊆ Rn, Sn+ ⊆ Rn×n are pointed, closed, convex and

solid cones, i.e., proper cones.• (Rn+)∗ = Rn+, (Ln)∗ = Ln, (Sn+)∗ = Sn+.

• Rn1+ ×R

n2+ × · · · ×R


k∑i=1

ni

.

• Ln1 × Ln2 × · · · × Lnk is a proper cone in Rk∑

i=1ni

.

• Sn1+ × S

n2+ × · · · × S


k∑i=1

ni×ni

.• Rn1

+ × Ln2 × Sn3+ is a proper cone in Rn1+n2+n3×n3 .


Different Forms of LCoP

• LP (Given A ∈ Rm×n,b ∈ Rm, ai ∈ Rn, bi ∈ R and c ∈ Rn)

Min cTxs.t. Ax = b

x ≥ 0(LP)

Max bT ys.t. AT y ≤ c

y ∈ Rm(LD)

Min c • xs.t. ai • x = bi, i = 1, . . . ,m

x ∈ Rn+(ai ∈ Rn, bi ∈ R)

(LP)Max bT ys.t. AT y + s = c

y ∈ Rm, s ∈ Rn+(LD)

Max b • ys.t. c−AT y ∈ Rn+

y ∈ Rm(LD)



• SOCP (Given A ∈ Rm×n, b ∈ Rm, c ∈ Rn, ai ∈ Rn, bi ∈ R)

Min cTxs.t. Ax = b

x ∈ Ln(SOCP)


y ∈ Rm, s ∈ Ln(SOCD)

Min c • xs.t. ai • x = bi, i = 1, . . . ,m

x ≥Ln 0(SOCP)

Max bT ys.t. c−AT y ∈ Ln

y ∈ Rm(SOCD)



• SDP (Given Ai ∈ Rn×n, bi ∈ R and C ∈ Rn×n)

Min C •Xs.t. Ai •X = bi, i = 1, · · · ,m

X � 0(SDP)

Max bT y

s.t.m∑i=1

yiAi + S = C

y ∈ Rm, S ≥Sn+

0

(SDD)

Min C •Xs.t. Ai •X = bi, i = 1, · · · ,m

X ∈ Sn+(SDP)

Max bT y

s.t. C −m∑i=1

yiAi � 0

y ∈ Rm(SDD)

Max bT y

s.t. C −m∑i=1

yiAi ∈ Sn+yi ∈ R, i = 1, . . . .m

(SDD)



• Combinations

Min pTx+ qT ys.t. Ax+By = b

x ≥ 0, y ∈ Lm

Max pTx+ qT y + rT z (linear objective)s.t. Ax+By ≤ b (linear inequalities)

C −k∑i=1

ziAi � 0 (linear matrix inequalities)

x ≥ 0, y ∈ Lm, z ∈ Rk


Second Order Cone Program (SOCP)Formulation

• Combinations

Min c • xs.t. ai • x = bi, i = 1, . . . ,m

x ∈ K

where K = Ln1 × · · · × Lnr = {x ∈ Rn |r∑i=1

ni = n, (x1, · · · , xn1) ∈

Ln1 , · · · , (xn−nr+1, · · · , xn) ∈ Lnr}, c, ai ∈ Rn, bi ∈ R.


Second Order Cone Program (SOCP)Formulation

• For easy thinking, take r = 1 and n = n1.

Min cTxs.t. Ax = b

x ∈ K[Conic variables]

(SOCP)

Max bT ys.t. AT y + S = C

y ∈ Rm, S ∈ K(i.e., C −AT y ∈ K)[Conic constraints]

(SOCD)


SOCP Formulation

• Generalization:Since linear constraints are allowed, instead of considering x ∈ Ln,we may consider

y = Mx+ q ∈ Lm

i.e., a linear transformation of x in an SOC.

• Denote M =

(Am−1×ncT1×n

), q =

(bm−1×1d1×1

), we have

y =

(AcT

)x+

(bd

)=

(Ax+ bcTx+ d

)∈ Lm

⇔ ‖Ax+ b‖ ≤ cTx+ d


SOCP Formulation

• (SOCP) becomes

Min cTxs.t. ‖Ax+ b‖ ≤ cTx+ d

x ∈ Rn

where A ∈ R(m−1)×n, b ∈ Rm−1, c ∈ Rn, d ∈ R.• More generally, (SOCP) may be represented as

Min cTxs.t. ‖Aix+ bi‖ ≤ cTi x+ di, i = 1, · · · , r

x ∈ Rn

where Ai ∈ R(ni−1)×n, b ∈ Rni−1, ci ∈ Rn, di ∈ R, and x ∈ Rn with

n =r∑i=1

ni.


Questions

• Are the following problems SOCP?

Minp∑i=1

1/(aTi x+ bi)

s.t. aTi x+ bi > 0, i = 1, · · · , pcTj x+ dj ≥ 0, j = 1, · · · , q

Min xTP0x+ 2qT0 x+ r0s.t. xTPix+ 2qTi x+ ri ≤ 0, i = 1, · · · , p

P0, P1, · · · , Pp ∈ Sn++


SOC-Representable Sets and Functions

• Definition (SOC-Representable Set)

A set S ⊆ Rn is SOC-representable if there exists a linear transformation

L(x) = Mx+ q for x ∈ Swith M = (·)m×n and q ∈ Rm, such that

L(S) =

K∏i=1

Lmi for some K > 0 andK∑i=1

mi = m.

• Note that S is SOC-representable if and only if S is the x-spaceprojection of the solution set of K(> 0) SOC inequalities:

Mi

(xµ

)+ qi ∈ Lmi , i = 1, ...,K,

i.e., S = {x ∈ Rn|Mi

(xµ

)+ qi ∈ Lmi , i = 1, ...,K}.



• Theorem: If S1,S2 ⊆ Rn are SOC-representable sets, then

αS1(α ≥ 0),S1 ∩ S2,S1 × S2 are SOC-representable.

• Definition (SOC-Representable Function)

A function g(·) : Rn → R is SOC-representable if its epigraph

Epi(g) = {(x, t) ∈ Rn+1 | g(x) ≤ t} is an SOC representable

set in Rn+1.



• Theorem

If a function g(·) : Rn → R is SOC-representable, then any of its

lower level sets Gα = {x ∈ Rn | g(x) ≤ α} is an SOC-representable set.


SOCP for NLP

min f(x)s.t. gj(x) ≤ 0 j = 1, ..., J (NLP )

x ∈ Rn

⇐⇒min ts.t. f(x) ≤ t

gj(x) ≤ 0 j = 1, ..., Jx ∈ Rn, t ∈ R


SOCP for NLP

• TheoremIf f(·) and gj(·), j = 1, 2, ..., J , are all SOC-representable functions, then(NLP) becomes an SOCP.


Identify SOC-Representable Functions

• TheoremIf f1 and f2 are SOC-representable functions, then αf1(α ≥ 0); f1 + f2and max {f1, f2} are SOC-representable.

• TheoremIf f1 and f2 are convex and SOC-representable with f1(x) beingmonotone nondecreasing, then g(x) = f1(f2(x)) is convex andSOC-representable.


Identify SOC-Representable Functions

• Hyperbolic Constriants

Theorem: For w, x, y ∈ R,

w2 ≤ xy, x ≥ 0, y ≥ 0⇔∥∥∥∥( 2wx− y

)∥∥∥∥ ≤ x+ y ⇔

2wx− yx+ y

∈ L3.

Theorem: For w ∈ Rn, x, y ∈ R,

wTw ≤ xy, x ≥ 0, y ≥ 0⇔∥∥∥∥( 2wx− y

)∥∥∥∥ ≤ x+ y ⇔

2wx− yx+ y

∈ Ln+2.

• Quadratic Constraints

Theorem: For Q ∈ Sn++, q ∈ Rn, r ∈ R,

xTQx+ 2qTx+ r ≤ 0⇔∥∥∥Q1/2x+Q−1/2q

∥∥∥ ≤ (qTQ−1q − r)1/2.


SOC-Representable Functions

• Constant function

g(x) ≡ c, x ∈ Rn, c ∈ R

Consider the epigraph{(

xt

)∈ Rn+1 | c ≤ t

}.

Set A = (0)m×n, then ||Ax|| ≤ t− c, i.e.,(

Axt− c

)∈ Lm+1.



• Linear function

g(x) = aTx+ b, a ∈ Rn, b ∈ R

Consider the epigraph{( x

t

)| aTx+ b ≤ t

}.

Set C = (0)p×n, then the set can be represented by||Cx|| ≤ t− aTx− b.



• Square Root of Homogeneous Convex Quadratic Function

g(x) =√xTAx,A ∈ Sn+

The epigraph is{( x

t

)|√xTAx ≤ t

}.

Since A ∈ Sn+, there exists matrix B satisfying A = BTB, we have√(Bx)T (Bx) ≤ t, i.e., ‖Bx‖ ≤ t.



• Convex Quadratic Function

g(x) = xTAx+ bTx+ c, A ∈ Sn+, b ∈ Rn, c ∈ R

g(x) ≤ t is equivalent to√(Bx)TBx+

(t− bTx− c− 1)2

4≤ t− bTx− c+ 1

2,

where B is the matrix satisfying A = BTB, i.e.,∥∥∥∥ Bxt−bT x−c−1

2

∥∥∥∥ ≤ t− bTx− c+ 1

2.



• Reciprocal Function

g(x) = 1x , x > 0

g(x) ≤ t is equivalent to xt ≥ 1, x > 0, hence we have√1 +

(x− t)24

≤ x+ t

2, x > 0, t ≥ 0.

i.e., ∥∥∥∥ 1x−t2

∥∥∥∥ ≤ x+ t

2.



• Maximum Function

g(x) = max{x1, x2, ..., xn}

The epigraph{( x

t

)| max{x1, x2, ..., xn} ≤ t

}={( x

t

)|

t− x1 ≥ 0, ..., t− xn ≥ 0}

.



• Sum of the norms

g(x) =∑pi=1 ||Fix+ gi||, Fi ∈ Rni×n, gi ∈ Rni(

xt

)satisfies g(x) ≤ t is equivalent to ∃ t1, t2, ..., tp such that

||Fjx+ gj || ≤ tj , j = 1, ..., p and t1 + ...+ tp ≤ t.

• Max of the norms

g(x) = maxi=1,...,p ||Fix+ gi||, Fi ∈ Rni×n, gi ∈ Rni



• Sum of the fractional functions

g(x) =∑pi=1 1/(aTi x+ bi), where aTi x+ bi > 0, i = 1, ..., p(

xt


ti(aTi x+ bi) ≥ 1, i = 1, ..., p and t1 + ...+ tp ≤ t, i.e., ∃ t1, ..., tp such

that∥∥∥∥( 2

aTi x+ bi − ti

)∥∥∥∥ ≤ aTi x+ bi + ti, i = 1, ..., p and

t1 + ...+ tp ≤ t.



• Sum of the fractional functions

g(x) =∑pi=1

||Fix+gi||2aTi x+bi

, where aTi x+ bi > 0, i=1,...,p(xt


(Fix+ gi)T (Fi + gi) ≤ ti(aTi x+ bi), i = 1, ..., p and t1 + ...+ tp ≤ t,

i.e., ∃ t1, ..., tp such that∥∥∥∥( Fix+ giaTi x+ bi − ti

)∥∥∥∥ ≤ aTi x+ bi + ti, i = 1, ..., p and t1 + ...+ tp ≤ t.



• Geometrical mean

g(x) = −(x1...xn)1/n, x ∈ Rn+, where n = 2q, q > 0

• Geometrical mean of nonnegative affine functions

g(x) = −

(p∏i=1

(aTi x+ bi)

)1/p

with aTi x+ bi ≥ 0


Semidefinite Programming (SDP)Formulation

• Let C,Ai ∈ Sn×n, b ∈ Rm, bi ∈ R.

Min C •Xs.t. Ai •X = bi, i = 1, · · · ,m

X ∈ Sn+(SDP)

Max bT y

s.t.m∑i=1

yiAi + S = C

y ∈ Rm, S ∈ Sn+

(SDD)

Min C •Xs.t. Ai •X = bi, i = 1, . . . ,m

X � 0(SDP)

Max bT y

s.t. C −m∑i=1

yiAi � 0

y ∈ Rm(SDD)


SDP Formulation

• SDD is often easier to “see” and “work” on, since it has only mvariables. We may have different interpretations of SDD.

- Linear Matrix Inequality Constraint

C −m∑i=1

yiAi � 0

y ∈ Rm

- PSD conic constraint

C −m∑i=1

yiAi ∈ Sn+

y ∈ Rm


SDP Formulation

• SDD is often easier to “see” and “work” on, since it has only mvariables. We may have different interpretations of SDD.

- Linear transformation of variables in the cones11 = c11 −A1(1, 1)y1 −A2(1, 1)y2 − · · · −Am(1, 1)ym

sij = cij −A1(i, j)y1 −A2(i, j)y2 − · · · −Am(i, j)ym

snn = cnn −A1(n, n)y1 −A2(n, n)y2 − · · · −Am(n, n)ym

S = (sij) ∈ Sn+


SDP Formulation

• Useful Facts

1.(xt

)∈ Ln+1 ⇔

[t xT

x tIn

]∈ Sn+1

+

Any SOCP is an SDP!

2. Let M =

(P vvT d

), P � 0, v ∈ Rn, d ∈ R.

M � (�)0 if and only if d− vTP−1v ≥ (>) 0.

3. Let M =

(P BBT C

), P � 0, B,C proper size.

M � (�)0 if and only if S = C −BTP−1B � (�) 0.


SDP Formulation

• Useful Facts

4. For x ∈ Rn, M = xxT ∈ Sn+.

5. For x ∈ Rn and W ∈ Sn, then(

1 xT

x W

)∈ Sn+1

+ if and only if

W � xxT .


SDP Identification

• Types of constraints that can be modeled as SDP

1. Convex quadratic constraints (Q ∈ Sn+, q ∈ Rn, c ∈ R)

xTQx+ qTx+ c ≤ 0⇔ xT (MTM)x+ qTx+ c ≤ 0

⇔(

I MxxTMT −qTx− v

)� 0

• Convex QCQPMin xTQ0x+ qT0 x+ c0s.t. xTQix+ qTi x+ ci ≤ 0 i = 1, . . . ,m

x ∈ Rn


SDP Identification

• Convex QCQP

Min θs.t. xTQ0x+ qT0 x+ c0 ≤ θ

xTQix+ qTi x+ ci ≤ 0 i = 1, . . . ,mx ∈ Rn

Min θ

s.t.

(I M0x

xTM0 −qTx− c0 + θ

)� 0(

I MixxTM0 −qTx− ci

)� 0, i = 1, . . . ,m

x ∈ Rn, θ ∈ R


SDP Identification

2. Binary constraints (x ∈ Rn)

xj ∈ {−1, 1}, j = 1, . . . , n

Y = xxT � 0 (rank-one matrix constraint)

Yjj = 1

3. Distance constraints

‖x− a‖2 ≤ 1

⇔ (x− a)T (x− a) ≤ 1

⇔(

I x− a(x− a)T 1

)� 0


SDP Identification

4. Elliposid constraints (Q � 0)

(x− a)TQ(x− a) ≤ 1

⇔ (x− a)T√Q√Q(x− a) ≤ 1

⇔(

I√Qx−

√Qa

(√Qx−

√Qa)T 1

)� 0

5. Eigenvalue constraints

λI � S � µI (µ > λ ≥ 0)

a psd with eigenvalues between λ and µ.


SDP Identification

6. Fractional constraints

(cTx)2

dTx≤ t

⇔(

t cTxcTx dTx

)� 0


Linear Conic Programming Theory andLinear Conic Programming Theory and Applications1 Shu-Cherng Fang...

Documents

Transcript of Linear Conic Programming Theory andLinear Conic Programming Theory and Applications1 Shu-Cherng Fang...