Linear Conic Programming Theory andApplications1
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
1Fang, S. C., and W. Xing. “Linear Conic Programming” (2013).
Conic Programming 1 / 25
Content• Part I : Introduction• Part II: Preliminary and Convex Cone Structure• Part III: Duality Theory of Linear Conic Programming• Part IV: Interior Point Methods and Solution Software• Part V: Modelling and Applications• Part VI: Recent Research• Part VII: Practical LCoP
Conic Programming 2 / 25
LCoP Part I – Introduction
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 3 / 25
Introduction
Content• Linear Conic Programs• Applications• Duality Theory and Algorithms• References
Conic Programming 4 / 25
Linear Conic Program
Min C •Xs.t. A •X = B
X ∈ K(LCoP)
where K is a closed, convex cone; A, B and C are in the space ofinterests with • being an appropriate linear operator.
Conic Programming 5 / 25
K = Rn+ (First Orthant)
When K = Rn+ = {x ∈ Rn|xi ≥ 0, i = 1, ..., n}, LCoP becomes LP.
Min cTxs.t. Ax = b
x ∈ Rn+
(LP)
where A ∈ Rm×n, b ∈ Rm and c ∈ Rn.
Equivalently,
Min cTxs.t. aTi x = bi, i = 1, . . . ,m
x ≥Rn+
0(LP)
where ai ∈ Rn, bi ∈ R and c ∈ Rn.
Conic Programming 6 / 25
K = Rn+
Figure:R1+ Figure:R2
+ Figure:R3+
Conic Programming 7 / 25
K = Ln (Lorentz Cone/Second OrderCone)
When K = Ln = {x ∈ Rn|√x21 + · · ·+ x2n−1 ≤ xn}, LCoP becomes
SOCP.Min cTxs.t. Ax = b
x ∈ Ln(SOCP)
where A ∈ Rm×n, b ∈ Rm and c ∈ Rn.
Equivalently,
Min cTxs.t. aTi x = bi, i = 1, . . . ,m
x ≥Ln 0(SOCP)
where ai ∈ Rn, bi ∈ R and c ∈ Rn.
Conic Programming 8 / 25
K = Ln
Figure: L2 Figure: L3
Conic Programming 9 / 25
K = Sn+ (Positive Semidefinite Cone)
When K = Sn+ = {X ∈ Rn×n|X = XT � 0}, LCoP becomes SDP.
Min C •Xs.t. Ai •X = bi, i = 1, ...,m
X ∈ Sn+(SDP)
where C,A1, ..., Am are given n× n symmetric matrices and b1, ..., bm aregiven scalars, and
M •X =∑i,j
MijXij = tr(MTX).
Equivalently,
Min C •Xs.t. Ai •X = bi, i = 1, ...,m
X � 0(SDP)
Conic Programming 10 / 25
K = Sn+
S2+ =
{(x, y, z) ∈ R3|
[x yy z
]� 0.
}⇐⇒ x ≥ 0, z ≥ 0, xz ≥ y2.
Conic Programming 11 / 25
Application of SOCP – I
Torricelli Point ProblemThe problem is proposed by Pierre de Fermat in 17th century. Giventhree points a, b and c on the R2 plane, find the point in the plane thatminimizes the total distance to the three given points. The solutionmethod was found by Torricelli, hence know as Torricelli point.
SOCP Formulation
Min t1 + t2 + t3
s.t.
[x− at1
]∈ L3,
[x− bt2
]∈ L3,
[x− ct3
]∈ L3
Question:C =? A =? X =? · · ·
Conic Programming 12 / 25
Application of SOCP – II
Robust Portfolio DesignAssume returns r are known within an ellipsoid
E = {r = r + κΣ1/2u : ‖u‖2 ≤ 1}.
where r is the expected return, Σ is the empirical covariance matrix,0 < κ < 1 is a given constant.
robust counterpart: (optimize the worst case)
maxω
minr∈E{rTω : eTω = 1, ω ≥ 0}.
Conic Programming 13 / 25
SOCP Formulation
Notice thatminr∈E
rTω
= min‖u‖2≤1
{rTω + κuT Σ1/2ω}
= rTω − κ‖Σ1/2ω‖2
Robust portfolio problem is an SOCP
Max rTω − κ‖Σ1/2ω‖2s.t. eTω = 1, ω ≥ 0
⇐⇒
Max ts.t. eTω = 1, ω ≥ 0[
κΣ1/2ωrTω − t
]∈ Ln+1
Question:C =? A =? X =? · · ·
Conic Programming 14 / 25
Application of SDP – I
Correlation Matrix Verification
Consider three random variables A, B and C. By definition, theircorrelation coefficients ρAB , ρAC and ρBC are valid if and only if 1 ρAB ρAC
ρAB 1 ρBC
ρAC ρBC 1
� 0
Suppose we know from some prior knowledge (e.g. empirical results ofexperiments) that −0.2 ≤ ρAB ≤ −0.1 and 0.4 ≤ ρBC ≤ 0.5. What are thesmallest and largest values that ρAC can take?
Conic Programming 15 / 25
SDP Formulation
The problem can be formulated as the following problem:
Min/Max ρAC
s.t. −0.2 ≤ ρAB ≤ −0.1
0.4 ≤ ρBC ≤ 0.5
ρAA = ρBB = ρCC = 1 ρAA ρAB ρAC
ρAB ρBB ρBC
ρAC ρBC ρCC
� 0
Conic Programming 16 / 25
SDP Formulation
In order to formulate the problem as in standard form, we handle theinequality constraints by augmenting the variable matrix and introducingslack variables, for example
0 12 0 0 0 0 0
12 0 0 0 0 0 00 0 0 0 0 0 00 0 0 1 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 0
•
ρAA ρAB ρAC 0 0 0 0ρAB ρBB ρBC 0 0 0 0ρAC ρBC ρCC 0 0 0 0
0 0 0 s1 0 0 00 0 0 0 s2 0 00 0 0 0 0 s3 00 0 0 0 0 0 s4
= ρAB + s1 = −0.1
Conic Programming 17 / 25
SDP Formulation
X =
1 ρAB ρAC 0 0 0 0ρAB 1 ρBC 0 0 0 0ρAC ρBC 1 0 0 0 00 0 0 s1 0 0 00 0 0 0 s2 0 00 0 0 0 0 s3 00 0 0 0 0 0 s4
, C =
0 0 12
0 0 0 00 0 0 0 0 0 012
0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 0
A1 =
0 12
0 0 0 0 012
0 0 0 0 0 00 0 0 0 0 0 00 0 0 1 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 0
A2 =?, A3 =? and A4 =?
Conic Programming 18 / 25
Other Applications - QCQP =⇒ SOCP
The popularity of SOCP is also due to the fact that it is a generalized formof convex QCQP (Quadratically Constrained Quadratic Programming).Specifically, consider the following QCQP:
Min xTA0x+ 2bT0 x+ c0s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m
where A0 � 0, Ai � 0 for i = 1, . . . ,m.Note that
t ≥n∑
i=1
x2i ⇐⇒
∣∣∣∣∣∣∣∣∣
∣∣∣∣∣∣∣∣∣
x1...xn
(t− 1)/2
∣∣∣∣∣∣∣∣∣
∣∣∣∣∣∣∣∣∣2
≤ t+ 1
2⇐⇒
x1...xn
(t− 1)/2(t+ 1)/2
∈ Ln+2
Conic Programming 19 / 25
Other Applications - QCQP =⇒ SOCP
Therefore, for each i = 1, . . . ,m
xTAix+ 2bTi x+ ci ≤ 0⇐⇒
A1/2i x
−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ln+2
QCQP can be equivalently written as
Min u
s.t.
A1/20 x
−1/2− bT0 x+ u/2− c0/21/2− bT0 x+ u/2− c0/2
∈ Ln+2
A1/2i x
−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ln+2, i = 1, . . . ,m.
Conic Programming 20 / 25
Other Applications - SOCP =⇒ SDP
Ln+1 can be easily embedded into Sn+1+ by observing the fact that[
xt
]∈ Ln+1 ⇐⇒
[t xT
x tIn
]∈ Sn+1
+
Based on this, we will focus on the theorems and algorithms for SDP. Butthis does not mean that SOCP is useless or we should transform SOCPto SDP in any case.
Conic Programming 21 / 25
Duality Theory of LP
Theorem (Weak Duality Theorem of LP)If x is primal feasible and y is dual feasible, then cTx ≥ bT y.
Theorem (Strong Duality Theorem of LP)
• If either LP or LD has a finite optimal solution, then so does the otherand they achieve the same optimal objective value.
• If either LP or LD has an unbounded objective value, then the otherhas no feasible solution.
How about the duality theorems of LCoP?
Conic Programming 22 / 25
Algorithms for LP
Simplex Method for LP
• Starting from one vertex• Check whether current vertex is optimal or not. If yes, stop.
Otherwise, go to the next step.• Move to a neighboring vertex, go to the previous step.
The complexity of simplex method is not polynomial.
Polynomial-time Algorithms
• Ellipsoid Method• Karmarkar’s Projective Scaling Algorithm• Affine Scaling Algorithm: Primal, Dual and Primal-Dual
How about the algorithms for LCoP?
Conic Programming 23 / 25
References
Books• Bertsekas D.P., Nedic A. and Ozdaglar A.E., Convex Analysis and
Optimization, Athena Scientific: Belmont, MA USA 2003• Boyd S. and Vandenberghe L., Convex Optimization, Cambridge University
Press: Cambridge, UK 2004• Fang S. and Puthenpura S., Linear Optimization and Extensions: Theory
and Algorithms, Prentice-Hall Inc.: Englewood Cliffs, NJ USA 1993• Nemirovski A., Lectures on Modern Convex Optimization: Analysis,
Algorithms, and Engineering Applications, Society for Industrial and AppliedMathematics: Philadelphia, PA USA 2001
• Renegar J., A Mathematical View of Interior-point Methods in ConvexOptimization, Society for Industrial and Applied Mathematics: Philadelphia,PA USA 2001
• Handbook of Semidefinite Programming: Theory, Algorithms, andApplications, edited by Wolkowicz H., Saigal R. and Vandenberghe L.,Kluwer Academic Publisher: Norwell, MA USA 2000
Conic Programming 24 / 25
References
• Rockafellar R.T., Convex Analysis, Princeton University Press:Princeton, NJ USA 1970
• Wright. S., Primal-Dual Interior-Point Methods, Society for Industrialand Applied Mathematics: Philadelphia, PA USA 1997
Others
• Ye Y., Linear Conic Programming, lecture notes online: http://www.stanford.edu/class/msande314/sdpmain.pdf
• Todd M.J., Semidefinite Programming, lecture notes online:http://people.orie.cornell.edu/~miketodd/cornellonly/or637/or637.html
A very popular general purpose SDP solver, SeDuMi, of Jos F. Sturm canbe found in: http://sedumi.ie.lehigh.edu/
Conic Programming 25 / 25
LCoP Part II – Preliminaries and Convex ConeStructures
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 39
Preliminaries and Convex Cone Structures
Content• Vectors, Matrices, and Spaces• Inner Products and Norms• Open, Closed, Interior, and Boundary Sets• Functions• Linear Systems• Convex Sets and Functions
Conic Programming 2 / 39
Vectors, Matrices and Spaces
• Real numbers: R, R+, R++
• Euclidean space: Rn
• First orthant: Rn+• n-dimensional (column) vector:
x = (x1, x2, . . . , xn)T
• Matrices space: Rm×n
• Matrix: M ∈ Rm×n, ith row Mi·, jth column M·j , ijth entry Mij(Mi,j)
• Symmetric square matrices space (n(n+ 1)/2-dimensional space):
Sn = {M ∈ Rn×n |M = MT }.
Conic Programming 3 / 39
Vectors, Matrices and Spaces
Given M ∈ Rm×n, N ∈ Rn×m, S ∈ Rn×n
• Determinant: det(S)
• Trace: tr(S)tr(MN) = tr(NM)
• Null space: N (M)= {x ∈ Rn|Mx = 0}.• Range space: R(M)= {y ∈ Rm|y = Mx for some x ∈ Rn}.• Positive semidefinite matrix:
S � 0 ⇐⇒ zTSz ≥ 0, ∀ z ∈ Rn
• Positive definite matrix:
S � 0 ⇐⇒ zTSz > 0, ∀ z ∈ Rn and z 6= 0
Conic Programming 4 / 39
Vectors, Matrices and Spaces
Theorem: (Schur complementary theorem)Given
X =
[A BBT C
]and S = C −BTA−1B,
if A � 0 thenX � (�)0⇔ S � (�)0
Conic Programming 5 / 39
Inner Products and Norms
• Inner products:x • y = xT y =
∑i xiyi
X • Y = tr(XTY ) =∑i,j XijYij
• Norms:• Euclidean norm: ‖x‖2 =
√x • x
• p-norm: ‖x‖p = (∑n
i=1 |xi|p)1/p for p ≥ 1
• Infinity-norm: ‖x‖∞ = max{|x1|, . . . , |xn|}• Frobenius norm:
‖X‖F =√X •X =
√tr(XTX)
• Note that: xTAx = A • xxT
Conic Programming 6 / 39
Open, Closed, Interior and Boundary Sets
• Neighborhood: N(x0; ε) = {x ∈ Rn| ‖x− x0‖ < ε}.
• Open: X ⊂ Rn is open if for any x ∈ X , there exists ε > 0 such thatN(x; ε) ⊂ X .
• Closed: X ⊂ Rn is closed, if Rn\X = {x ∈ Rn|x /∈ X} is open.
• Closure of a set X ⊂ Rn is the smallest closed set containing X andis denoted as cl(X ).
Conic Programming 7 / 39
Open, Closed, Interior and Boundary Sets
• Interior: the interior of a given set X ⊂ Rn is
int(X ) = {x ∈ X |∃ εx > 0 such that N(x; εx) ⊂ X}
• Boundary of a set X ⊂ Rn:
bdry(X ) = cl(X )\int(X ) = {x ∈ cl(X )|x /∈ int(X )}
• Bounded: a set X ⊂ Rn is bounded if there exist an r > 0 such that
‖x‖ < r,∀x ∈ X
Conic Programming 8 / 39
Functions
• Continuous: f : X ⊂ Rn is continuous at x0 if( i ) x0 ∈ X(i i) limx→x0 f(x) = f(x0)
• Continuous function: f ∈ C0(X ) means f is continuous at all pointsin X ⊂ Rn.
• Gradient: For f : X ⊂ Rn → R
∇f(x) = [∂f(x)
∂x1, · · · , ∂f(x)
∂xn]1×n
• Hessian: For f : X ⊂ Rn → R
F (x) = [∂2f(x)
∂xi∂xj]n×n
• Continuously differentiable function: f ∈ Cp(X ) (p = 1, 2, · · · ) meansf is p-th continuously differentiable over X ⊂ Rn.
Conic Programming 9 / 39
Functions
Theorem (Taylor theorem)Let X be open, f ∈ Cp(X ), x1, x2 ∈ X , x1 6= x2 and
x(θ) = θx1 + (1− θ)x2 ∈ X , ∀ 0 ≤ θ ≤ 1.
Then ∃ x = θx1 + (1− θ)x2 ∈ X , 0 < θ < 1, s.t.
f(x2) = f(x1) +
p−1∑k=1
1
k!dkf(x1;x2 − x1) +
1
p!dpf(x;x2 − x1)
where dkf(x;h) is the k-th order differential of function f along h.
Conic Programming 10 / 39
Functions: Big O and Small o
Let g(·) be a real-valued function on R.
• g(x) = O(m(x))
∃ c ≥ 0 such that ∣∣∣∣ g(x)
m(x)
∣∣∣∣ ≤ c as x→ 0 (or +∞)
• g(x) = o(m(x)) ∣∣∣∣ g(x)
m(x)
∣∣∣∣ = 0 as x→ 0 (or +∞)
Conic Programming 11 / 39
Functions
Taylor theorem in small o formulation:• p = 1
f(x+ h) = f(x) +∇f(x)h+ o(‖h‖)• p = 2
f(x+ h) = f(x) +∇f(x)h+1
2hTF (x)h+ o(‖h‖2)
Conic Programming 12 / 39
Linear Systems
Given x1, · · · , xm ∈ Rn• Linear combination:
m∑i=1
λixi,
where λi ∈ R, i = 1, . . . ,m.• Linearly independent
m∑i=1
λixi = 0⇒ λ1 = · · · = λm = 0
• Affine combination: a linear combination withm∑i=1
λi = 1
• Affinely independent: if x2 − x1, · · · , xm − x1 are linearlyindependent.
Conic Programming 13 / 39
Linear Systems
• Convex combination: a linear combination withm∑i=1
λi = 1 and λi ≥ 0, i = 1, . . . ,m
• Hyperplane:
X = {x ∈ Rn|aTx =
n∑i=1
aixi = b}
• Affine space: affine combination of any two points in the space is stillin the space. (An intersection of finitely many hyperplanes.)
• Linear subspace: an affine space containing the origin.
We can always transform an affine space Y ⊂ Rn into a linearsubspace X ⊂ Rn by choosing x0 ∈ Y such that
X = {x− x0|x ∈ Y}
Conic Programming 14 / 39
Linear Systems
• Half space:
X = {x ∈ Rn|aTx =
n∑i=1
aixi ≤ b}
• Polyhedron: an intersection of finitely many half spaces.
• Polytope: a bounded polyhedron
• Dimension of a linear subspace: the maximum number of linearlyindependent vectors in the subspace.
• Dimension of an affine space: the dimension of the transformedlinear subspace.
• Dimension of a polyhedron: the dimension of the smallest affinespace containing it.
Conic Programming 15 / 39
Linear Systems
• Linear equations
a1 • x = b1a2 • x = b2· · · · · · · · ·
am • x = bm
⇒ Ax = b,
where a1, · · · , am and x are all in Rn.
A1 •X = b1A2 •X = b2· · · · · · · · ·
Am •X = bm
⇒ AX = b,
where A1, · · · , Am and X are all in Sn.• For convenience, A∗y =
∑mi=1 yiAi.
Conic Programming 16 / 39
Convex Sets and Properties
• A set X ⊂ Rn is convex if for any x1 ∈ X and x2 ∈ X , we haveλx1 + (1− λ)x2 ∈ X , for all 0 ≤ λ ≤ 1.
• Convex hull: the smallest convex set containing a given set
conv(X ) = {x ∈ Rn|x =∑mi=1 λiy
i for some m ∈ N+,λi ≥ 0,
∑mi=1 λi = 1, and yi ∈ X , i = 1, . . . ,m}
• Dimension of a convex set: the dimension of the smallest affinespace containing it.
• Relative interior of a convex set X ⊂ Rn: suppose H is the smallestaffine space containing X ,
ri(X ) = {x ∈ Rn|∃ open set Y ⊆ Rn such that x ∈ Y ∩H ⊂ X}
• Supporting hyperplane H = {x ∈ Rn|aTx = b} of a convex set X :
aT y ≥ b,∀ y ∈ X and X ∩H 6= ∅.
Conic Programming 17 / 39
Convex Functions and Properties
• Epigraph of a function f : X ⊂ Rn → R
epif = {(x, λ) ∈ Rn+1|λ ≥ f(x), x ∈ X}
• Closed function: if epif is a closed set.
• Convex function: if epif is a convex set.
• Concave function: if −f is a convex function.
• Convex hull function conv(f) of a function f : X ⊂ Rn → R is afunction on X such that epi(conv(f)) = conv(epi(f)).
Lemmaf : X ⊂ Rn → R is a convex function if and only if for any x1, x2 ∈ X and0 ≤ λ ≤ 1, we have
f(λx1 + (1− λ)x2) ≤ λf(x1) + (1− λ)f(x2).
Conic Programming 18 / 39
Convex Functions and Properties
• Subgradient d ∈ Rn of a convex function f : X ⊂ Rn at x ∈ X :
if for any y ∈ X ,f(y) ≥ f(x) + dT (y − x)
• The set {(y, λ) ∈ Rn+1|λ− dT y = f(x)− dTx} is a supportinghyperplane of epif at x.
• Subdifferential of a convex function f : X ⊂ Rn at x ∈ X :
∂f(x) = {d ∈ Rn|d is a subgradient of f at x}
Conic Programming 19 / 39
Convex Functions and Properties
Figure: (x, f(x))↔ (m, b) or (y, h(y))
Conic Programming 20 / 39
Convex Functions and Properties
• Conjugate (transform) of f : X ⊂ Rn → R:
h(y) = supx∈X{y • x− f(x)}
with h being defined on Y = {y ∈ Rn|h(y) < +∞}.
Lemmah : Y is a closed, convex function.
Lemma (Fenchel’s inequality)Given f : X and its conjugate h : Y, then
x • y ≤ f(x) + h(y), ∀ x ∈ X and y ∈ Y.
Moreover,x • y = f(x) + h(y) ⇐⇒ y ∈ ∂f(x)
Conic Programming 21 / 39
Conjugate Functions and Properties
Let f : X ⊂ Rn → R be a function with its conjugate transform h : Y.• For α ∈ R, the conjugate of f + α is h− α.• For a ∈ Rn, the conjugate of f(x) = f(x) + x • a on X ish(y) = h(y − a), ∀ y ∈ Y.
• For a ∈ Rn, the conjugate of f(x) = f(x− a) on X ish(y) = h(y) + y • a, ∀ y ∈ Y.
• For λ > 0, the conjugate of f1(x) = λf(x) on X is h1(y) = λh( yλ ),∀ y ∈ λY.
• For λ > 0, the conjugate of f2(x) = f(xλ ) on λX is h2(y) = h(λy),∀ y ∈ Y/λ.
TheoremAssume that f1 : X and f2 : X have the same convex hull function. Thenthey have the same conjugate transform h : Y when it exists.
Conic Programming 22 / 39
Conjugate Functions and Properties
We know the dual problem of LD is LP again. When will the conjugatetransform of h : Y become f : X?
Proper functionA convex function f is proper if its epigraph is non-empty and contains novertical lines, i.e. if f(x) < +∞ for at least one x and f(x) > −∞ forevery x.
TheoremLet f : X ⊂ Rn → R be a proper closed convex function with conjugatetransform h : Y. Then the conjugate transform of h : Y is f : X . Moreover,y ∈ ∂f(x) if and only if x ∈ ∂h(y). In this case,
x • y = f(x) + h(y) ⇐⇒ y ∈ ∂f(x) or x ∈ ∂h(y)
Conic Programming 23 / 39
Convex Cone Structure
Content• Convex Cones and Properties• Partial Order and Ordered Vector Space• Some Examples
Conic Programming 24 / 39
Convex Cones and Properties
• A set K ⊂ Rn is a cone if
∀x ∈ K and λ > 0⇒ λx ∈ K;
• A cone K ⊂ Rn is pointed if
K ∩ −K = {0};
• A cone K ⊂ Rn is solid ifintK 6= ∅;
• A cone K ⊂ Rn is proper if it is pointed, solid, closed and convex.
Conic Programming 25 / 39
Convex Cones and Properties
• Conic combination: a linear combination∑mi=1 λix
i with λi ≥ 0,xi ∈ Rn for all i = 1, . . . ,m.
• The conic hull of a set X ⊂ Rn is
cone(X ) = {x ∈ Rn|x =∑mi=1 λix
i, for some m ∈ N+
and xi ∈ X , λi ≥ 0, i = 1, . . . ,m.}
• The dual cone K∗ ⊂ Rn of a cone K ⊂ Rn is
K∗ = {y ∈ Rn|y • x ≥ 0,∀ x ∈ K}
K∗ is a closed, convex cone.• If K∗ = K, then K is a self-dual cone.
Conic Programming 26 / 39
Convex Cones and Properties
K, K1, K2 are convex cones in Rn.• (K∗)∗ = cl(K).• K1 ⊆ K2 ⇒ K∗2 ⊆ K∗1 .• K1 ∩K2, K1 ∪K2, K1 +K2, K1 ×K2 are all cones.• (K1 +K2)∗ = K∗1 ∩K∗2 .• K1, K2 closed⇒ K1 +K2 and K1 ×K2 are closed.• ri(K1 +K2) = ri(K1) + ri(K2).• ri(K1 ×K2) = ri(K1)× ri(K2).• The supporting hyperplane of K always contains the origin• If K is solid, then K∗ is pointed.• If K is pointed, then K∗ is solid.• If K is proper, then K∗ is proper.
Conic Programming 27 / 39
Convex Cones and Properties
• Rn+ ⊆ Rn, Ln ⊆ Rn, Sn+ ⊆ Rn×n are pointed, closed, convex andsolid cones, i.e., proper cones.
• (Rn+)∗ = Rn+, (Ln)∗ = Ln, (Sn+)∗ = Sn+.
• Rn1+ ×R
n2+ × · · · ×R
nk+ is a proper cone in R
k∑i=1
ni
.
• Ln1 × Ln2 × · · · × Lnk is a proper cone in Rk∑
i=1ni
.
• Sn1+ × S
n2+ × · · · × S
nk+ is a proper cone in R
k∑i=1
ni×ni
.• Rn1
+ × Ln2 × Sn3+ is a proper cone in Rn1+n2+n3×n3 .
Conic Programming 28 / 39
Partial Order and Ordered Vector Space
• A relation “≥” is a partial order on a set X if it has:
1. reflexivity: a ≥ a for all a ∈ X ;
2. antisymmetry: a ≥ b and b ≥ a imply a = b;
3. transitivity: a ≥ b and b ≥ c imply a ≥ c.
• An ordered vector space X is equipped with a partial order “≥” whichalso satisfies:
• homogeneity: a ≥ b and λ ∈ R+ imply λa ≥ λb;
• additivity: a ≥ b and c ≥ d imply a+ c ≥ b+ d.
Conic Programming 29 / 39
Partial Order and Ordered Vector Space
• A proper cone K in a vector space can induce a partial order “≥K”
a ≥K b⇔ a− b ∈ K
which leads to an ordered vector space.• Similarly, we can define “≤K”
a ≤K b⇔ b ≥K a,
• Closeness of K allows passing limits in ≥K :
ai ≥K bi, ai → a, bi → b as i→∞ ⇒ a ≥K b.
• Solidness of K allows us to define a strict inequality:
a >K b⇔ a− b ∈ intK,
anda <K b⇔ b >K a.
Conic Programming 30 / 39
Examples: Rn+
• Rn+ is a proper cone.
• Inner product: x • y = xT y
• (Rn+)∗ = Rn+ (self-dual)
• Partial order: “≥Rn+
”
Conic Programming 31 / 39
Examples: Ln
• Ln / SOC(n− 1)Lorentz cone (secondorder cone)
Ln = {x ∈ Rn|xn ≥√x21 + · · ·+ x2n−1}
• Ln is a proper cone.
• Inner product: x • y = xT y
• (Ln)∗ = Ln (self-dual)
• Partial order: “≥Ln ”
Conic Programming 32 / 39
Examples: Sn+
• Sn+ ⊂ Sn: the set of symmetric positive semidefinite matrices• Sn+ is a proper cone.• Inner product:
X • Y = tr(XTY )
• Another view:
vec(X) = [X11,√
2X12, X22,√
2X13,√
2X23, X33, · · · , Xnn]T ∈ Rn(n+1)
2
ThenX • Y = vec(X) • vec(Y ) =
∑i,j
XijYij
• Partial order: “≥Sn+
” or “�”
Conic Programming 33 / 39
Examples: Sn+
Lemma(Sn+)∗ = Sn+ (self-dual)
Proof.“⊆”: If X ∈ (Sn+)∗, then zTXz = X • zzT ≥ 0, for all z ∈ Rn.Therefore, X ∈ Sn+.
“⊇”: For any Y ∈ Sn+,
Y =∑ni=1 λiz
i(zi)T ,
with λi ≥ 0.If X ∈ Sn+, then
X • Y =
n∑i=1
λiX • zi(zi)T =
n∑i=1
λi(zi)TXzi ≥ 0.
Therefore, X ∈ (Sn+)∗.Conic Programming 34 / 39
Examples: Cn and C∗n
• Copositive cone:
Cn = {X ∈ Sn|zTXz ≥ 0,∀ z ≥Rn+
0}
• Completely positive(nonnegative) cone:
C∗n =
{X ∈ Sn
∣∣∣∣∣ X =∑mi=1 z
i(zi)T , for some m ∈ N+
and zi ≥Rn+
0, i = 1, . . . ,m
}
• (Cn)∗ = C∗n and Cn = (C∗n)∗
• C∗n ⊂ Sn+ ⊂ Cn
Conic Programming 35 / 39
Examples: Cones of NonnegativeQuadratic Functions — Homogeneous
• F ⊂ Rn• Nonnegative homogeneous quadratic functions over F
f(x) = xTAx ≥ 0,∀x ∈ F
f ⇔ A
• HDF = {A ∈ Sn|xTAx ≥ 0,∀x ∈ F} is a closed, convex cone.(i) Closeness:
xTAix ≥ 0 and Ai → A⇒ xTAx ≥ 0
(ii) Convexity:
xTAiX ≥ 0, i = 1, 2⇒ xT (λA1 + (1− λ)A2)x ≥ 0,∀ 0 ≤ λ ≤ 1
Conic Programming 36 / 39
Examples: Cones of NonnegativeQuadratic Functions — Homogeneous
• HD∗F = cl(cone{xxT |x ∈ F})• (HDF )∗ = HD∗F and (HD∗F )∗ = HDF• Examples:
• F = Rn
HDF = HD∗F = Sn+
• F = Rn+
HDF = Cn and HD∗F = C∗n• F = {x ∈ Rn
+|eTx = 1}HDF = Cn and HD∗F = C∗n
Conic Programming 37 / 39
Examples: Cones of NonnegativeQuadratic Functions — Nonhomogeneous
• Nonnegative quadratic functions over F ⊂ Rn
f(x) = xTAx+ 2bTx+ c ≥ 0,∀x ∈ F
f ⇔[c bT
b A
]• DF = {
[c bT
b A
]∈ Sn+1|
[1x
]T [c bT
b A
] [1x
]≥ 0,∀x ∈ F} is a closed,
convex cone.
• D∗F = cl(cone{[
1 xT
x xxT
]|x ∈ F})
• (D∗F )∗ = DF and (DF )∗ = D∗F
Conic Programming 38 / 39
Examples: Cones of NonnegativeQuadratic Functions — Nonhomogeneous
• Examples:• F = Rn
DF = D∗F = Sn+1+
• F = Rn+
DF = Cn+1 and D∗F = C∗n+1
Conic Programming 39 / 39
LCoP Part III – Duality Theory of Linear ConicProgramming
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 28
Duality Theory of Linear ConicProgramming
Content• Definition of LCoP and LCoD• Conjugate Duality Theory• Deriving LCoD from LCoP• Conic Duality Theorems for LCoP• Duality Theorems of LP, SOCP and SDP
Conic Programming 2 / 28
Linear Conic Programs
Recall thatMin C •Xs.t. A •X = B
X ∈ K(LCoP)
where K is a closed, convex cone, A, C and B are in the space ofinterests with • being an appropriate linear operator.
Note that when K = Rn+ or Ln, X is a vector; when K = Sn+, X is an
n× n matrix.
Conic Programming 3 / 28
Linear Conic Programs
Min c • xs.t. ai • x = bi, i = 1, . . . ,m
x ∈ K(LCoP)
where K is a closed, convex cone, such as Rn+, Ln, Sn+.
Max bT ys.t.
∑mi=1 yia
i + s = cs ∈ K∗, y ∈ Rm
(LCoD)
where K∗ is the dual cone of K.
Conic Programming 4 / 28
Conjugate Duality Theory
Conjugate Program
inf f(x)s.t. x ∈ X ∩K
(CP)
where f : X ⊂ Rn → R and K is a cone in Rn.
Conjugate Dual
inf h(y)s.t. y ∈ Y ∩K∗
(CD)
where h : Y is the conjugate transform of f : X and K∗ is the dual coneof K.• feas(*) denotes the feasible domain of problem (*)• opt(*) denotes the optimal solution set of problem (*)• v(*) denotes the optimal value of problem (*)
Conic Programming 5 / 28
Conjugate Duality Theory
Theorem (Conjugate duality theorem/KKT duality theorem)If x ∈ feas(CP) and y ∈ feas(CD), then
0 ≤ x • y ≤ f(x) + h(y)
with the equality holding if and only if
x • y = 0 and y ∈ ∂f(x),
in which casex ∈ opt(CP) and y ∈ opt(CD).
ProofThe inequality follows from Fenchel’s inequality and the definition of dualcone. The rest follows easily.
Conic Programming 6 / 28
Conjugate Duality Theory
Theorem (Weak duality theorem)If both CP and CD are feasible, then
(i) v(CP) is finite and
v(CP) + h(y) ≥ 0,∀ y ∈ feas(CD);
(ii) v(CD) is finite and
v(CP) + v(CD) ≥ 0.
ProofThis theorem follows from the previous KKT duality theorem.
Conic Programming 7 / 28
Conjugate Duality Theory
Theorem (Fenchel’s theorem/Strong duality theorem)Suppose that f : X and K are closed and convex. If v(CD) is finite andone of the following conditions holds:
(i) ri(K∗) ∩ ri(Y) 6= ∅,(ii) both K∗ and Y are polyhedrons,
thenv(CP) + v(CD) = 0 and opt(CP) 6= ∅.
Similarly, if v(CP) is finite and one of the following conditions holds:(i) ri(K) ∩ ri(X ) 6= ∅,(ii) both K and X are polyhedrons,
thenv(CP) + v(CD) = 0 and opt(CD) 6= ∅.
Proof: See Rockafellar’s book “Convex Analysis” Section 31.
Conic Programming 8 / 28
Deriving LCoD from LCoP
LCoP
Min c • xs.t. ai • x = bi, i = 1, . . . ,m
x ∈ K(LCoP)
Deriving LCoD in the framework of conjugate program.
Conic Programming 9 / 28
Deriving LCoD from LCoP
LCoP as CP
Variables: uT = (u0, u1, . . . , um) ∈ Rm+1;
f(u) = u0;
X = {u ∈ Rm+1|ui = bi, i = 1, . . . ,m};
K0 = {u ∈ Rm+1|u0 = c • x, ui = ai • x, x ∈ K, i = 1, . . . ,m}.
inf f(u)s.t. u ∈ X ∩K0
Conic Programming 10 / 28
Deriving LCoD from LCoP
Corresponding CD
Variables: vT = (v0, v1, . . . , vm) ∈ Rm+1;
h(v) = supu∈X{u • v − f(u)} < +∞
= supu0∈R{(v0 − 1)u0 +
m∑i=1
bivi}
Hence
h(v) =∑m
i=1 bivi;
Y = {v ∈ Rm+1|v0 = 1};
Conic Programming 11 / 28
Deriving LCoD from LCoP
Corresponding CD
Moreover,
K∗0 = {v ∈ Rm+1|v • u ≥ 0,∀u ∈ K0}
= {v ∈ Rm+1|(v0c +∑m
i=1 viai) • x ≥ 0,∀x ∈ K}
= {v ∈ Rm+1|v0c +∑m
i=1 viai ∈ K∗}.
Hence
Y ∩K∗0 = {v ∈ Rm+1|c +
m∑i=1
viai = s, s ∈ K∗}.
inf∑m
i=1 bivis.t. c +
∑mi=1 via
i = ss ∈ K∗
Conic Programming 12 / 28
Deriving LCoD from LCoP
CD to LCoDDefine variables: y = −(v1, . . . , vm)T , we have
Max bT ys.t.
∑mi=1 yia
i + s = cs ∈ K∗, y ∈ Rm
(LCoD)
Therefore, the duality theorems of conjugate programs may apply toLCoP.
Conic Programming 13 / 28
Conic Duality Theorems for LCoP
Theorem (Weak duality theorem)If both LCoP and LCoD are feasible, then
c • x ≥ bT y,∀x ∈ feas(LCoP) and (y, s) ∈ feas(LCoD).
Theorem (Strong duality theorem)
(i) If feas(LCoP) ∩ int(K) 6= ∅ and v(LCoP) is finite, then there exists(y∗, s∗) ∈ feas(LCoD) such that bT y∗ = v(LCoP).
(ii) If feas(LCoD) ∩ int(K∗) 6= ∅ and v(LCoD) is finite, then there existsx∗ ∈ feas(LCoP) such that c • x = v(LCoD).
Proof: See Aharon Ben-Tal and Arkadi Nemirovski’s book “Lectures onmodern convex optimization” Chapter 2.
Conic Programming 14 / 28
Conic Duality Theorems for LCoP
Theorem (KKT duality theorem)If feas(LCoP) and feas(LCoD) are both nonempty andfeas(LCoP) ∩ int(K) 6= ∅, then x∗ is optimal for LCoP if and only if thefollowing conditions hold:
(i) x∗ ∈ feas(LCoP);
(ii) There exists (y∗, s∗) ∈ feas(LCoD);
(iii) c • x∗ = bT y∗ (or equivalently x∗ • s∗ = c • x∗ − bT y∗ = 0).Proof: =⇒ follows from strong duality theorem.
⇐= is obvious.
Conic Programming 15 / 28
Linear Program (LP)
Min cTxs.t. Ax = b
x ≥Rn+
0(LP)
Max bT ys.t. AT y + s = c
s ≥Rn+
0(LD)
Conic Programming 16 / 28
Linear Program (LP)
Theorem (LP duality theorem)
(i) If either LP or LD is unbounded, then the other one is infeasible.
(ii) If either v(LP) or v(LD) is finite, then there exist x∗ ∈ feas(LP) and(y∗, s∗) ∈ feas(LD) such that v(LP) = cTx∗ = bT y∗ = v(LD).
(iii) If LP is feasible and v(LP) is finite, then x∗ is optimal for LP if andonly if the following conditions hold:
(a) Ax∗ = b, x∗ ≥Rn+
0;
(b) there exists (y∗, s∗) satisfying AT y∗ + s∗ = c, s ≥Rn+
0;
(c) (x∗)T s∗ = cTx∗ − bT y∗ = 0.
Conic Programming 17 / 28
Second Order Cone Program (SOCP)
Min cTxs.t. Ax = b
x ≥K 0(SOCP)
where K = Ln1 × · · · × Lnr = {x ∈ Rn|n1 + · · ·+ nr = n, (x1, ..., xn1)T ∈
Ln1 , ..., (xn−nr+1, ..., xn)T ∈ Lnr}.
Max bT ys.t. AT y + s = c
s ≥K 0(SOCD)
Conic Programming 18 / 28
Second Order Cone Program (SOCP)
Theorem (SOCP duality theorem)
(i) If either SOCP or SOCD is unbounded, then the other one isinfeasible.
(ii) If there exists a feasible solution x such that x ∈ int(K), andv(SOCP) is finite, then there exist (y∗, s∗) ∈ feas(SOCD) such thatv(SOCP) = bT y∗ = v(SOCD).
(iii) If there exists a feasible solution (y, s) such that s ∈ int(K), andv(SOCD) is finite, then there exist x∗ ∈ feas(SOCP) such thatv(SOCP) = cTx∗ = v(SOCD).
Conic Programming 19 / 28
Second Order Cone Program (SOCP)
Theorem (SOCP duality theorem)
(iv) If both SOCP and SOCD are feasible, and there exists a feasiblesolution x such that x ∈ int(K), then x∗ is optimal for SOCP if andonly if the following conditions hold:
(a) Ax∗ = b, x∗ ≥K 0;
(b) there exists (y∗, s∗) satisfying AT y∗ + s∗ = c, s∗ ≥K 0;
(c) (x∗)T s∗ = cTx∗ − bT y∗ = 0.
Conic Programming 20 / 28
Difference between LP and SOCP (interior feasible solution):
Min −x2
s.t. x1 − x3 = 0x ∈ L3
Max 0 · y
s.t.
0−10
− y
10−1
=
−y−1y
∈ L3
v(SOCP ) = 0 but SOCD is infeasible.
Figure: Feasible domain is a ray x1 = x3 in hyperplane x2 = 0. No feasibleinterior point.
Conic Programming 21 / 28
Second Order Cone Program (SOCP)
Finite nonzero duality gap:
Min −x2
s.t. x1 + x3 − x4 + x5 = 0x2 + x4 = 1x ∈ L3 × L2
Max y2
s.t.
y1 +s1 = 0y2 +s2 = −1
y1 +s3 = 0−y1 +y2 +s4 = 0y1 +s5 = 0
s ∈ L3 × L2
x∗ =
−101
× [11
]y∗ =
[−1−1
]s∗ =
101
× [01
]
v(SOCP) = 0 6= −1 = v(SOCD)
Conic Programming 22 / 28
Second Order Cone Program (SOCP)
Zero duality gap with non-attainable value:
Min x1
s.t. −x2 −x3 = 0x2 = −1
x ∈ L3
Max −y2s.t. s1 = 1
−y1 +y2 +s2 = 0−y1 +s3 = 0
s ∈ L3
x∗ =
0−11
v(SOCD) = 0 but not attainable.
Conic Programming 23 / 28
Semidefinite Program (SDP)
Min C •Xs.t. AX = b
X � 0(SDP)
Max bT ys.t. A∗y + S = C
S � 0(SDD)
Note:
A∗y =
m∑i=1
yiAi
Conic Programming 24 / 28
Semidefinite Program (SDP)
Theorem (SDP duality theorem)
(i) If either SDP or SDD is unbounded, then the other one is infeasible.
(ii) If there exists a feasible solution X such that X � 0, and v(SDP) isfinite, then there exist (y∗, S∗) ∈ feas(SDD) such thatv(SDP) = bT y∗ = v(SDD).
(iii) If there exists a feasible solution (y, S) such that S � 0, and v(SDD)is finite, then there exist X∗ ∈ feas(SDP) such thatv(SDP) = C •X∗ = v(SDD).
Conic Programming 25 / 28
Semidefinite Program (SDP)
Theorem (SDP duality theorem)
(iv) If both SDP and SDD are feasible, and there exists a feasiblesolution X such that X � 0, then X∗ is optimal for SDP if and only ifthe following conditions hold:
(a) AX∗ = b, X∗ � 0;
(b) there exists (y∗, S∗) satisfying A∗y∗ + S∗ = C, S∗ � 0;
(c) X∗ • S∗ = C •X∗ − bT y∗ = 0.
Conic Programming 26 / 28
Semidefinite Program (SDP)
Interior feasible solutionInfinite duality gap:
C =
[0 11 0
], A =
[0 00 1
], b = 0
X∗ =
[0 00 0
]and SDD is infeasible.
Zero duality gap with non-attainable value:
C =
[1 00 0
], A =
[0 11 0
], b = 1
v(SDP ) = 0 but is not attainable. y∗ = 0 and S∗ =
[1 00 0
].
Conic Programming 27 / 28
Semidefinite Program (SDP)
Finite nonzero duality gap:
C =
0 0 00 0 10 1 0
, A1 =
0 0 00 1 00 0 0
, A2 =
1 0 00 0 −10 −1 0
, b =
[01
]
X∗ =
1 0 00 0 00 0 0
, y∗ =
[0−1
], S∗ =
1 0 00 0 00 0 0
v(SDP ) = 0 6= −1 = v(SDD)
Conic Programming 28 / 28
LCoP Part IV – Interior Point Methods andSolution Software
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 39
Interior Point Methods and SolutionSoftware
Content• Interior Points and Primal-Dual Model• Barrier Functions and Optimal Systems• Central Path and Newton Methods• Path Following Method• CVX Solution Software
Conic Programming 2 / 39
Interior Point Methods
• Interior point approach• Start from an interior point solution.• If the current solution is not good enough, then move to another interior
point solution.• Stop at an interior point solution whose objective value is close to the
optimum (within an ε gap).• Advantages:
• Polynomial time complexity (comparing to the simplex method for LP)• Excellent computational performance in practice (comparing to the
ellipsoid method)• Three types: primal; dual; primal-dual
Conic Programming 3 / 39
Primal-dual Model
• Primal-dual type of LP
Min sTxs.t. Ax = b
AT y + s = cx ≥Rn
+0, s ≥Rn
+0
(LPD)
• Primal-dual type of SDP
Min S •Xs.t. AX = b
A∗y + S = CX � 0, S � 0
(SDPD)
• Note:AX = [A1 •X, · · · , Am •X]T
and A∗y =∑mi=1 yiAi
Conic Programming 4 / 39
Interior Points
feas+(LP) = {x ∈ Rn|Ax = b, x >Rn+
0}feas+(LD) = {(y, s) ∈ Rm ×Rn|AT y + s = c, s >Rn
+0}
feas+(LPD) = feas+(LP)× feas+(LD)
feas+(SDP) = {X ∈ Sn|AX = b,X � 0}feas+(SDD) = {(y, S) ∈ Rm × Sn|A∗y + S = C, S � 0}feas+(SDPD) = feas+(SDP)× feas+(SDD)
• Assumptions:• feas+(LP) and feas+(LD) are not empty and the rows of A are linearly
independent.
• feas+(SDP) and feas+(SDD) are not empty and the vectors formed byAi in A are linearly independent.
Conic Programming 5 / 39
Barrier function
• Properties required:• Strictly convex (concave).• Goes to +∞ (−∞) when the point is close to the boundary.• Sufficient continuous differentiability.
• Barrier functions:
LP : −∑ni=1 log xi SDP : − log det(X)
LD :∑ni=1 log si SDD : log det(S)
LPD : −∑ni=1 log(xisi) SDPD : − log det(XS)
Conic Programming 6 / 39
LP with Barrier
Min cTx− µ∑ni=1 log xi
s.t. Ax = bx >Rn
+0
(LPB)
Max bT y + µ∑ni=1 log si
s.t. AT y + s = cs >Rn
+0
(LDB)
Min sTx− µ∑ni=1 log(xisi)
s.t. Ax = bAT y + s = cx >Rn
+0, s >Rn
+0
(LPDB)
Conic Programming 7 / 39
Common Optimal System for LP withBarrier
Ax = bAT y + s = cΛxs = µex >Rn
+0, s >Rn
+0,
where e = (1, . . . , 1)T and Λx is a diagonal matrix with (Λx)ii = xi,i = 1, . . . , n.
Notice that
µ =xT s
n=cTx− bT y
n
When µ→ 0, sTx→ 0. Optimal!
Conic Programming 8 / 39
SDP with Barrier
Min C •X − µ log det(X)s.t. AX = b
X � 0(SDPB)
Min bT y + µ log det(S)s.t. A∗y + S = C
S � 0(SDDB)
Min S •X − µ log det(XS)s.t. AX = b
A∗y + S = CX � 0, S � 0
(SDPDB)
Conic Programming 9 / 39
Common Optimal System for SDP withBarrier
AX = bA∗y + S = CXS = µIX � 0, S � 0
Notice that
µ =S •Xn
=C •X − bT y
n
When µ→ 0, S •X → 0. Optimal!
Conic Programming 10 / 39
Central Path for LP and SDP
CLP = {(x, y, s) ∈ feas+(LPD)|Λxs = µe, 0 < µ < +∞}
CSDP = {(X, y, S) ∈ feas+(SDPD)|XS = µI, 0 < µ < +∞}
Under proper assumptions:• For any 0 < µ < +∞, there exists a unique point on central path.
LP: (x(µ), y(µ), s(µ))
SDP: (X(µ), y(µ), S(µ))
• Given µ > 0, the set {(x, y, s) ∈ feas+(LPD)|Λxs = µe, 0 < µ < µ} isbounded.Given µ > 0, the set {(X, y, S) ∈ feas+(SDPD)|XS = µI, 0 < µ < µ}is bounded.
Conic Programming 11 / 39
Example: Central Path (Primal)
Min x1 + x2
s.t. x1 + x2 ≤ 3x1 − x2 ≤ 1x2 ≤ 2x1 ≥ 0, x2 ≥ 0
Figure: Projection of central path on (x1, x2)
Conic Programming 12 / 39
Example: Central Path (Dual)
Max 3y1 + y2 + 2y3
s.t. y1 + y2 ≤ 1y1 − y2 + y3 ≤ 1y1 ≤ 0y2 ≤ 0y3 ≤ 0
Figure: Projection of central path on (y1, y2)
Conic Programming 13 / 39
Newton Method for LP
Given (x0, y0, s0) ∈ feas+(LPD) with µ0 = (s0)T x0
n and 0 ≤ γ ≤ 1, find(dx, dy, ds) satisfying
A(x0 + dx) = bAT (y0 + dy) + (s0 + ds) = cΛx0+dx(s0 + ds) = γµ0ex0 + dx >Rn
+0, s0 + ds >Rn
+0,
After linearization A 0 00 AT I
Λs0 0 Λx0
dxdyds
=
00
γµ0e− Λx0Λs0e
x0 + dx >Rn
+0, s0 + ds >Rn
+0,
Directly solve the equation is not easy.
Conic Programming 14 / 39
Newton Method for LP
Linear scaling: Given a positive diagonal matrix D ∈ Rn×n,
A = AD, x0 = D−1x0, s0 = Ds0, c = Dc
A 0 00 AT I
Λs0 0 Λx0
dxdyds
=
00
γµ0e− Λx0Λs0e
x0 + dx >Rn
+0, s0 + ds >Rn
+0,
• D = Λx0 : x0 = e ⇒ x0 + dx >Rn+
0, ∀‖dx‖2 < 1 (Primal)
• D = Λ−1s0 : s0 = e ⇒ s0 + ds >Rn
+0, ∀‖ds‖2 < 1 (Dual)
• D = Λ1/2x0 Λ
−1/2s0 : v0 = x0 = s0 = Λ
1/2x0 Λ
1/2s0 e (Primal-dual)
Conic Programming 15 / 39
Primal-Dual Interior-Point Method for LP
D = Λ1/2x0 Λ
−1/2s0 : A 0 0
0 AT II 0 I
dxdyds
=
00
γµ0Λ−1v0 e− v
0
x0 + dx >Rn
+0, s0 + ds >Rn
+0,
One can solveAAT dy = −A(γµ0Λ−1
v0 e− v0)
And then solve ds and dx:
ds = −AT dydx = −ds + γµ0Λ−1
v0 e− v0
Conic Programming 16 / 39
Newton Method for SDP
Given (X0, y0, S0) ∈ feas+(SDPD) with µ0 = S0•X0
n and 0 ≤ γ ≤ 1, find(4X, dy,4S) satisfying
A(X0 +4X) = bA∗(y0 + dy) + (S0 +4S) = C(X0 +4X)(S0 +4S) = γµ0IX0 +4X � 0, S0 +4S � 0
After linearization
A4X = 0A∗dy + 4S = 0
4XS0 + X04S = γµ0I −X0S0
X0 +4X � 0, S0 +4S � 0.
Directly solve the equation is not easy.
Conic Programming 17 / 39
Newton Method for SDP
Linear transformation: Given an invertible matrix L ∈ Rn×n, letA = (A1, . . . , Am), Ai = LTAiL for i = 1, . . . ,m.X0 = L−1X0L−T , S0 = LTS0L, C = LTCL.
A4X = 0
A∗dy + 4S = 0
4XS0 + X04S = γµ0I − X0S0
X0 +4X � 0, S0 +4S � 0
• L = (X0)1/2: X0 = I ⇒ X0 +4X � 0, ∀‖4X‖F < 1 (Primal)• L = (S0)−1/2: S0 = I ⇒ S0 +4S � 0, ∀‖4S‖F < 1 (Dual)• LLT = (S0)−1/2[(S0)1/2X0(S0)1/2]1/2(S0)−1/2:V 0 = X0 = S0 (Primal-dual)
Conic Programming 18 / 39
Primal-Dual Interior-Point Method for SDP
LLT = (S0)−12 [(S0)
12X0(S0)
12 ]
12 (S0)−
12 :A 0 0
0 A∗ II 0 I
4Xdy4S
=
00
γµ0(V 0)−1 − V 0
X0 +4X � 0, S0 +4S � 0
One can solveAA∗dy = −A(γµ0(V 0)−1 − V 0)
And then solve 4S and 4X:
4S = −A∗dy4X = −4S + γµ0(V 0)−1 − V 0
Conic Programming 19 / 39
Neighborhood of Central Path for LP
Notice that x0 = s0 = v0
• Distance to central path: u >Rn+0
δ(u) = ‖e− n
uTuΛuu‖2
• Neighborhood of the central path
N2(β) = {u|u >Rn+
0, δ(u) ≤ β}
N−∞(β) = {u|u >Rn+
0,Λuu ≥Rn+
(1− β)uTu
nI}
Conic Programming 20 / 39
Examples: N2(12) and N−∞(12)
Figure: Neighborhood N2( 12) Figure: Neighborhood N−∞( 1
2)
Conic Programming 21 / 39
Finding Step Length for LP
x0 + αdxs0 + αds
scaling back−−−−−−−−−−−→
[x1
s1
] new scaling−−−−−−−−−−→ v1 = x1 = s1
LemmaFor any 0 ≤ α ≤ 1,
µ1 =‖v1‖22n
=(x0 + αdx)T (s0 + αds)
n= (1− α+ γα)µ0
Conic Programming 22 / 39
Finding Step Length for LP
LemmaIf δ(v0) < 1 and α satisfies x0 + αdx >Rn
+0 and s0 + αds >Rn
+0, then
(1− α+ γα)δ(v1) ≤ (1− α)δ(v0) +α2
2
(γ2δ(v0)2
1− δ(v0)+ n(1− γ)2
)Proof:
µ1δ(v1) = µ1‖e− 1
µ1Λv1v
1‖2
= ‖(1− α+ γα)µ0e− Λ(v0+αdx)(v0 + αds)‖2
≤ ‖(1− α)µ0(e− 1
µ0Λv0v
0)‖2 + ‖α2Λdx ds‖2
≤ (1− α)µ0δ(v0) +α2
2‖dx + ds‖22
= (1− α)µ0δ(v0) +α2
2(γ2‖µ0Λ−1
v0 e− v0‖22 + (1− γ)2nµ0)
≤ (1− α)µ0δ(v0) +α2
2(µ0γ2δ(v0)2
1− δ(v0)+ (1− γ)2nµ0)
Conic Programming 23 / 39
Finding Step Length for LP
LemmaIf v0 ∈ N2(β) with β = 1
2 , γ = 11+1/
√2n
and α = 1, then
(i) v1 ∈ N2(β)
(ii) x1 • s1 = x1 • s1 = ‖v1‖22 = γµ0
Conic Programming 24 / 39
Path Following Algorithm for LP
Step 1: (Initialization)ε > 0, (x0, y0, s0) with v0 ∈ N (β), where β = 1
2 .Set k = 0, γ = 1
1+1/√
2n, and α = 1.
Step 2: Solve the Newton system introduced above and get (dx, dy, ds).Set xk+1 = xk + αdx
yk+1 = yk + αdysk+1 = sk + αds
with vk+1 = Λ1/2
xk+1Λ1/2
sk+1e.Set k = k + 1.
Step 3: If xk • sk < ε, stop. Otherwise, go to Step 2.
Conic Programming 25 / 39
Complexity for LP
TheoremGiven the above settings, we have
(i) vk ∈ N2(β), k = 0, 1, 2, . . ..(ii) The algorithms stops in
O(√n log
x0 • s0
ε)
steps and output a primal-dual solution satisfying
xk • sk < ε
Conic Programming 26 / 39
Neighborhood of Central Path for SDP
Notice that X0 = S0 = V 0
• Distance to central path: U ∈ Sn+ and U � 0
δ(U) = ‖I − n
I • U2U2‖F , with U2 = UU
• Neighborhood of the central path
N2(β) = {U |U � 0, δ(U) ≤ β}
N−∞(β) = {U |U � 0, U2 � (1− β)I • U2
nI}
Conic Programming 27 / 39
Finding Step Length for SDP
X0 + α4XS0 + α4S
scaling back−−−−−−−−−−−→
[X1
S1
] new scaling−−−−−−−−−−→ V 1 = X1 = S1
LemmaFor any 0 ≤ α ≤ 1,
µ1 =‖V 1‖2Fn
=tr[(X0 + α4X)(S0 + α4S)]
n= (1− α+ γα)µ0.
Conic Programming 28 / 39
Finding Step Length for SDP
LemmaFor any square matrix U , we have
tr(U2) = ‖U + UT
2‖2F − ‖
U − UT
2‖2F ≤ ‖
U + UT
2‖2F
LemmaSuppose δ(V 0) < 1 and α ≥ 0 satisfies X0 + α4X � 0 andS0 + α4S � 0. Let
W =(X0 + α4X)(S0 + α4S) + ((X0 + α4X)(S0 + α4S))T
2
thenW = (1− α)(V 0)2 + αγµ0I + α24X4S +4S4X
2
andδ(V 1)2 ≤ ‖I − 1
µ1W‖2F
Conic Programming 29 / 39
Finding Step Length for SDP
LemmaSuppose δ(V 0) < 1 and α ≥ 0 satisfies X0 + α4X � 0 andS0 + α4S � 0. Then
(1− α+ γα)δ(V 1) ≤ (1− α)δ(V 0) +α2
2
(γ2δ(V 0)2
1− δ(V 0)+ n(1− γ)2
)Proof
µ1δ(V 1) ≤ (1− α)µ0δ(V 0) + α2‖4X4S+4S4X2 ‖F
≤ (1− α)µ0δ(V 0) + α2
2 ‖4X +4S‖2F= (1− α)µ0δ(V 0) + α2
2 (γ2‖µ0(V 0)−1 − V 0‖2F + (1− γ)2nµ0)
≤ (1− α)µ0δ(V 0) + α2µ0
2
(γ2δ(V 0)2
1−δ(V 0) + n(1− γ)2)
Conic Programming 30 / 39
Finding Step Length for SDP
LemmaIf V 0 ∈ N2(β) with β = 1
2 , γ = 11+1/
√2n
and α = 1, then
(i) V 1 ∈ N2(β)
(ii) X1 • S1 = X1 • S1 = ‖V 1‖2F = γµ0
Conic Programming 31 / 39
Path Following Algorithm for SDP
Step 1: (Initialization)ε > 0, (X0, y0, S0) with V 0 ∈ N (β), where β = 1
2 .Set k = 0, γ = 1
1+1/√
2n, and α = 1.
Step 2: Solve the equation system introduced above and get(4X, dy,4S).Set Xk+1 = Xk + α4X
yk+1 = yk + αdySk+1 = Xk + α4S
with V k+1 = Xk+1 = Sk+1.Set k = k + 1.
Step 3: If Xk • Sk < ε, stop. Otherwise, go to Step 2.
Conic Programming 32 / 39
Complexity
TheoremGiven the above settings, we have
(i) V k ∈ N2(β), k = 0, 1, 2, . . ..(ii) The algorithms stops in
O(√n log
X0 • S0
ε)
steps and output a primal-dual solution satisfying
Xk • Sk < ε
Conic Programming 33 / 39
Example: Path Following Algorithm
min x1 + x2
s.t. x1 + x2 ≤ 3x1 − x2 ≤ 1x2 ≤ 2x1 ≥ 0, x2 ≥ 0
Figure: Path following algorithm with β = 1/2
Conic Programming 34 / 39
Initialization and Improve the Performance
Initialization
• Big-M Method• Two-Phase Method• Self-Dual Embedding Method
Different Path-Following Methods
• Short Step Algorithm• Long Step Algorithm• Predictor-Corrector Algorithm• Largest Step Algorithm
Reference: Handbook of Semidefinite Programming: Theory, Algorithms,and Applications, edited by Wolkowicz H., Saigal R. and VandenbergheL., Kluwer Academic Publisher: Norwell, MA USA 2000
Conic Programming 35 / 39
CVX Solution Software - Example 1
Conic Programming 36 / 39
CVX Solution Software - Example 2
Conic Programming 37 / 39
CVX Solution Software - Example 3
Conic Programming 38 / 39
CVX Solution Software - Example 4
Conic Programming 39 / 39
LCoP Part V – Modeling and Applications
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 34
Modeling and Applications
Content• Weber Problem• Matrix Optimization• Approximating Solutions of Linear Equations• Minimum of a Univariate Polynomial of Degree 2n
• Stochastic Queue Location Problem• Conic Reformulations of QCQP and Extensions• Robust Optimization
Conic Programming 2 / 34
Weber Problem
In 1909, the German economist Alfred Weber introduced the problem offinding a best location for the warehouse of a company, such that the totaltransportation cost to serve the customers is minimum. Suppose thatthere are m customers needing to be served. Let the location ofcustomer i be ai ∈ R2, i = 1, . . . ,m. Suppose that customer may havedifferent demands, to be translated as weight ωi for customer i,i = 1, . . . ,m. Denote the desired location of the warehouse to be x.Then, the optimization problem becomes
Minm∑i=1
ωiti
s.t.
[x− aiti
]∈ L3, i = 1, . . . ,m
Conic Programming 3 / 34
Matrix Optimization
Given A0, A1, . . . , Am, determine if there is y ∈ Rm such that
A0 +∑mi=1 yiAi � 0
which is equivalent to minimizing
λmax(A0 +∑mi=1 yiAi)
Notice that
t ≥ λmax(A0 +∑mi=1 yiAi) ⇔ tI −A0 −
∑mi=1 yiAi � 0
Equivalent Problem (SDP)
Min ts.t. tI −A0 −
∑mi=1 yiAi � 0
t ∈ R, y ∈ Rm
Conic Programming 4 / 34
Approximating Solutions of LinearEquations
Given A ∈ Rm×n, b ∈ Rm, solve
Ax = b
Approximation:• l1 norm:
minx‖Ax− b‖1 ⇔ min
∑mi=1 ti
s.t. −ti ≤ Ai·x− bi ≤ ti, i = 1, . . . ,m.
• l2 norm:
minx‖Ax− b‖2 ⇔
min t
s.t.
[Ax− b
t
]∈ Lm+1
Conic Programming 5 / 34
Approximating Solutions of LinearEquations
• l∞ norm:
minx‖Ax− b‖∞ ⇔ min t
s.t. −t ≤ Ai·x− bi ≤ t, i = 1, . . . ,m.
• Logarithm approximation: (b >Rm+
0)
minx
max1≤i≤m
| log(Ai·x)−log bi| ⇔min ts.t. 1/t ≤ Ai·x/bi ≤ t, i = 1, . . . ,m,
t > 0
⇔
min t
s.t.
t−Ai·x/bi 0 00 Ai·x/bi 10 1 t
� 0
Conic Programming 6 / 34
Minimum of a Univariate Polynomial
Consider the problem of finding the minimum of a univariate polynomialof degree 2n:
Min x2n + a1x2n−1 + · · ·+ a2n−1x+ a2n
s.t. x ∈ R
This problem is equivalent to
Max ts.t. x2n + a1x
2n−1 + · · ·+ a2n−1x+ a2n − t ≥ 0 for all x ∈ R
Conic Programming 7 / 34
Minimum of a Univariate Polynomial
It is well known that a univariate polynomial is nonnegative over the realdomain if and only if it can be written as sum of squares (SOS), which isequivalent to saying that there must be a positive semidefinite matrixX ∈ Sn+1
+ such that
x2n+a1x2n−1+· · ·+a2n−1x+a2n−t = (1, x, x2, · · · , xn)X(1, x, x2, · · · , xn)T .
Hence, this problem can be cast as an SDP:
Max ts.t. a2n − t = X11
a2n−k =∑
i+j=k+2
Xij , k = 1, . . . , 2n− 1
X(n+1),(n+1) = 1X ∈ Sn+1
+
Conic Programming 8 / 34
Stochastic Queue Location Problem
BackgroundSuppose there are m potential customers to serve in the region.Customers’ demands are random, and once a customer calls for service,then the server in the service center will need to go to the customer toprovide the required service. In case the server is occupied, then thecustomer will have to wait. The goal is to find a good location for theservice center in order to minimize the expected waiting time of service.
Conic Programming 9 / 34
Stochastic Queue Location Problem
Assumptions and notationsSuppose that the service calls from the customer are identicallyindependent distributed, and the demand process follows the Poissondistribution with overall arrival rate λ, and the probability that any servicecall is from customer i is assumed to be pi for i = 1, . . . ,m. The queueingprinciple is First Come First Service, and there is only one server in theservice center. This model can be regarded as M/G/1 queue as inQueueing theory, and the expected service time, including waiting timeand traveling, can be explicitly computed. To this end, denote the velocityof the server to be v, and the location of customer i is ai, i = 1, . . . ,m,and the location of the service center to be x.
Conic Programming 10 / 34
Stochastic Queue Location Problem
Problem formulationThe expected waiting time for customer i is given by
ωi(x) =
(2λ/v2)m∑j=1
pj‖x− aj‖22
1− (2λ/v)m∑j=1
pj‖x− aj‖22+
1
v‖x− ai‖2,
where the first term in the expected term is the expected waiting time forthe server to be free and the second the term is the waiting time for theserver to travel after his departure at the service center.
Conic Programming 11 / 34
Stochastic Queue Location Problem
Observing the fact that
‖x‖22/s ≤ t, s > 0⇐⇒∣∣∣∣∣∣∣∣[ x
t−s2
]∣∣∣∣∣∣∣∣2
≤ t+ s
2
We can formulate this problem as an SOCP:
Min (2mλ/v2)m∑i=1
piti + (1/v)m∑i=1
t0i
s.t.
[x− ait0i
]∈ L3,
x− aiti−s2
ti+s2
∈ L4, i = 1, ...,m
s ≤ 1− (2λ/v)m∑i=1
pisi,
[x− aisi
]∈ L3, i = 1, ...,m.
Conic Programming 12 / 34
Quadratically Constrained QuadraticProgramming (QCQP)
QCQP:
min xTA0x+ 2bT0 x+ c0
s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m,
Ai ∈ Sn, bi ∈ Rn, ci ∈ R, i = 0, 1, . . . ,m.
Homogeneous QCQP(HQCQP):
min xTA0x
s.t. xTAix+ ci ≤ 0, i = 1, . . . ,m
Conic Programming 13 / 34
Convex QCQP
Convex QCQP: A0, . . . , Am � 0Assume rank(Ai) = di, then Ai = BTi Bi, Bi ∈ Rdi×n, i = 0, . . . ,m.
xTAix+ 2bTi x+ ci ≤ 0 ⇔
Bix−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ldi+2
Reformulating Convex QCQP as SOCP:
min u
s.t.
B0x−1/2− bT0 x+ u/2− ci/21/2− bT0 x+ u/2− ci/2
∈ Ld0+2
Bix−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ldi+2 i = 1, . . . ,m
Conic Programming 14 / 34
SDP Relaxation for QCQP
Recall that
xTAix+ 2bTi x+ ci =
[1x
]T [ci bTibi Ai
] [1x
]=
[ci bTibi Ai
]•[
1 xT
x xxT
]SDP Relaxation
min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0, i = 1, . . . ,m
Y11 = 1Y � 0
Conic Programming 15 / 34
SDP Relaxation for HQCQP
Recall thatxTAix = Ai • xxT
SDP Relaxation
min A0 •Xs.t. Ai •X ≤ −ci, i = 1, . . . ,m
X � 0
Conic Programming 16 / 34
Exact Solutions from SDP Relaxation
TheoremWhen m = 1, if the optimal solution of the SDP relaxation for QCQPexists, then there exists a rank one optimal solution Y ∗ of the SDP
relaxation for QCQP. Furthermore, let Y ∗ =
[1 (x∗)T
x∗ x∗(x∗)T
], then x∗ is
optimal for QCQP.
Conic Programming 17 / 34
Exact Solutions from SDP Relaxation
LemmaLet G be a symmetric matrix and X be a positive semidefinite matrix withrank d. Suppose that G •X ≤ (=)0. Then there exists a rank-onedecomposition X =
∑di=1 xix
Ti such that xTi Gxi ≤ (=)0, i = 1, . . . , d.
TheoremWhen m = 2, suppose there exists x such that xTAix+ 2bTi x+ ci < 0,i = 1, 2, and there exists y1 ≥ 0, y2 ≥ 0 such that[c0 bT0b0 A0
]+ y1
[c1 bT1b1 A1
]+ y2
[c2 bT2b2 A2
]� 0. If, for the optimal solution Y ∗,
either[c1 bT1b1 A1
]• Y ∗ < 0 or
[c2 bT2b2 A2
]• Y ∗ < 0, then there is no gap
between QCQP and its SDP relaxation and at least one optimal solutioncan be obtained from the rank-one decomposition of Y ∗.
Conic Programming 18 / 34
Exact Solutions from SDP Relaxation
Theorem(HQCQP)When m = 2, suppose there exists x, such that xTAix+ ci < 0, i = 1, 2,and there exists y1 ≥ 0 and y2 ≥ 0 such that A0 + y1A1 + y2A2 � 0. Thenthere is no gap between HQCQP and its SDP relaxation and at least oneoptimal solution can be obtained from the rank-one decomposition of X∗.
Conic Programming 19 / 34
One Extension of QCQP
Ext-QCQP
min xTA0x+ 2bT0 x+ c0
s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m,(x21, . . . , x
2n)T ∈ X
(Ext-QCQP)
where X is a closed convex set.SDP Relaxation:
min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0, i = 1, . . . ,m
(Y22, Y33, . . . , Yn+1n+1)T ∈ XY11 = 1Y � 0
Conic Programming 20 / 34
Exact Solutions from SDP Relaxation
TheoremSuppose there is σ ∈ {−1, 1}n+1, such that all the off-diagonal elements
of Λσ
[ci bTibi Ai
]Λσ, 0 ≤ i ≤ m, are nonpositive. If Y ∗ is an optimal solution
for the SDP relaxation for Ext-QCQP, then
x∗ = σ1(σ2√Y ∗22, σ3
√Y ∗33, . . . , σn+1
√Yn+1n+1)T
is optimal for Ext-QCQP.
Conic Programming 21 / 34
More Results on Ext-QCQP
A Special Case of Ext-QCQP
min xTAx
s.t. (x21, . . . , x2n)T ∈ X
where X is a closed convex set.SDP Relaxation:
min A •Xs.t. (X11, X22, . . . , Xnn)T ∈ X
X � 0
Conic Programming 22 / 34
Approximation by Randomized Algorithm
• Suppose Z � 0, Zii = 1, i = 1, . . . , n. If ξ ∼ N(0, Z) andσ ∈ {−1, 1}n with
σi =
{1 ξi ≥ 0−1 ξi < 0
then E(σσT ) = 2π arcsinZ.
• Suppose Z � 0, Zii = 1, i = 1, . . . , n. Then arcsinZ � Z � 0.• 2
π arcsin t ≤ 1− α+ αt, ∀t ∈ [−1, 1] with α = 0.87856 . . ..
Conic Programming 23 / 34
Approximation by Randomized Algorithm
Given X � 0, let Z = D+XD+ + D, where D, D+, D are diagonalmatrices defined by
Dii =√Xii, D+
ii =
{(√Xii)
−1 Xii 6= 00 Xii = 0
, Dii =
{0 Xii 6= 01 Xii = 0
• X = DZD
• E(A • (DσσTD)) = A • (DE(σσT )D) = A • ( 2πD arcsin(Z)D).
Conic Programming 24 / 34
Approximation by Randomized Algorithm
TheoremSuppose A � 0, and X∗ is the minimizer of SDP relaxation. Let Z, D,D+, and D be define from X∗, and ξ ∼ N(0, Z) with corresponding σ.Then
E(A • (DσσTD)) ≤ 2
πmin{xTAx|(x21, . . . , x2n)T ∈ X}
Proof:E(A • (DσσTD))
= A • ( 2πD arcsin(Z)D)
≤ A • ( 2πDZD)
= 2πA •X
∗
≤ 2π min{xTAx|(x21, . . . , x2n)T ∈ X}
Conic Programming 25 / 34
Approximation by Randomized Algorithm
TheoremAssume that A � 0, Aij ≥ 0 for all i 6= j, and X∗ is the minimizer of SDPrelaxation. Let Z, D, D+ and D be defined from X∗ and ξ ∼ N(0, Z) withcorresponding σ. Then
E(A • (DσσTD)) ≤ αmin{xTAx|(x21, . . . , x2n)T ∈ X}
Proof:E(A • (DσσTD))
= A • ( 2πD arcsin(Z)D)
=∑i 6=j AijDiiDjj(
2π arcsinZij) +
∑ni=1AiiD
2ii
≤∑i 6=j AijDiiDjj(1− α+ αZij) +
∑ni=1AiiD
2ii
= (1− α)∑i,j AijDiiDjj + αA •X∗
≤ αA •X∗ ≤ αmin{xTAx|(x21, . . . , x2n)T ∈ X}Conic Programming 26 / 34
Example: MAX CUT Problem
Given a graph G = (V,E,W ), find an optimal partition of the node set intotwo subsets V1 and V2 such that the weighted cut is maximized.
max∑ni=1
∑nj=1 wij
1−xixj
2
s.t. x2i = 1, i = 1, . . . , n(MAX − CUT )
Reformulation as Ext-QCQP:
Aij =
{wij i 6= j
−∑k 6=i wik i = j
, X = {e}
TheoremIf wij ≥ 0, ∀i 6= j. Then the expected value of randomized algorithm is atleast α ≈ 0.878 times the value of the maximum cut.
Conic Programming 27 / 34
Robust Optimization
min f0(x)
s.t. fi(x, ui) ≤ 0
∀ui ∈ Ui, i = 1, . . . ,m.
Motivation
• The parameters are inexact.• The parameters cannot be foreseen.• The parameters may vary with time and other environment factors.
Conic Programming 28 / 34
Example: Robust Linear Program
min cTx
s.t. aTi x ≤ bi
∀ai ∈ {a0i +∑pj=1 uja
ji | ‖u‖2 ≤ 1},
i = 1, . . . ,m
For any x
maxai aTi x = maxu(a0i )
Tx+∑pj=1 uj(a
ji )Tx
= (a0i )Tx+ ‖((a1i )Tx, . . . , (a
pi )Tx)T ‖2
Conic Programming 29 / 34
Example: Robust Linear Program
Therefore
aTi x ≤ bi,∀ai ⇔ ‖((a1i )Tx, . . . , (api )Tx)T ‖2 ≤ bi − (a0i )
Tx
Equivalent SOCP
min cTx
s.t.
(a1i )
Tx...
(api )Tx
bi − (a0i )Tx
∈ Lp+1, i = 1, . . . ,m
Conic Programming 30 / 34
Robust Convex QCQP
min fTx
s.t. xTATAx ≤ 2bTx+ c
∀(A, b, c) ∈ {(A0, b0, c0) +∑pj=1 uj(A
j , bj , cj) | ‖u‖2 ≤ 1},
Let U(x) = (A0x,A1x, . . . , Apx) and
V (x) =
c0 + 2(b0)Tx 1
2c1 + (b1)Tx · · · 1
2cp + (bp)Tx
12c
1 + (b1)Tx 0...
. . .12cp + (bp)Tx 0
Conic Programming 31 / 34
Robust Convex QCQP
xTATAx ≤ 2bTx+ c
⇔[
1u
]TUT (x)U(x)
[1u
]−[
1u
]TV (x)
[1u
]≤ 0,∀‖u‖2 ≤ 1
Lemma (S-lemma)Let P and Q be symmetric matrices and yTPy > 0 for some y. Then theimplication
zTPz ≥ 0 ⇒ zTQz ≥ 0
is valid if and only if Q− λP � 0 for some λ ≥ 0.
Conic Programming 32 / 34
Robust Convex QCQP
From S-lemma,[1u
]TUT (x)U(x)
[1u
]−[
1u
]TV (x)
[1u
]≤ 0,∀‖u‖2 ≤ 1
⇔ V (x)− UT (x)U(x)− λ[1−I
]� 0,∃λ ≥ 0
⇔
V (x)− λ[1−I
]UT (x)
U(x) I
� 0,∃λ ≥ 0
(by Schur complementary theorem)
Conic Programming 33 / 34
Robust Convex QCQP
Equivalent SDP
min fTx
s.t.
c0 + 2(b0)Tx− λ 1
2c1 + (b1)Tx · · · 1
2cp + (bp)Tx (A0x)T
12c
1 + (b1)Tx λ (A1x)T
.... . .
...12cp + (bp)Tx λ (Apx)T
A0x A1x · · · Apx I
� 0
Conic Programming 34 / 34
LCoP Part VI – Recent Research
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 46
Recent Research
Content• Quadratically Constrained Quadratic Programming
(QCQP)
Conic Programming 2 / 46
Quadratically Constrained QuadraticProgramming (QCQP)
QCQP:
Min xTA0x+ 2bT0 x+ c0
s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m
Ai ∈ Sn, bi ∈ Rn, ci ∈ R, i = 0, 1, . . . ,m.
Homogeneous QCQP(HQCQP):
Min xTA0x
s.t. xTAix+ ci ≤ 0, i = 1, . . . ,m
Conic Programming 3 / 46
Convex QCQP
Convex QCQP: A0, . . . , Am � 0Assume rank(Ai) = di, then Ai = BTi Bi, Bi ∈ Rdi×n, i = 0, . . . ,m.
xTAix+ 2bTi x+ ci ≤ 0 ⇔
Bix−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ldi+2
Reformulating Convex QCQP as SOCP:
Min u
s.t.
B0x−1/2− bT0 x+ u/2− c0/21/2− bT0 x+ u/2− c0/2
∈ Ld0+2
Bix−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ldi+2 i = 1, . . . ,m
Conic Programming 4 / 46
SDP Relaxation for QCQP
Recall that
xTAix+ 2bTi x+ ci =
[1x
]T [ci bTibi Ai
] [1x
]=
[ci bTibi Ai
]•[
1 xT
x xxT
]
Let Y =
[1x
] [1x
]T=
[1 xT
x xxT
]∈ Sn+1
+ , we have SDP Relaxation
Min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0, i = 1, . . . ,m
Y11 = 1
Y � 0, rank(Y ) = 1
Note: No convexity is assumed.
Conic Programming 5 / 46
SDP Relaxation for HQCQP
Recall thatxTAix = Ai • xxT
Let X = xxT ∈ Sn+, we have SDP Relaxation
Min A0 •Xs.t. Ai •X ≤ −ci, i = 1, . . . ,m
X � 0, rank(X) = 1
Conic Programming 6 / 46
Exact Solutions from SDP Relaxation
TheoremWhen m = 1, if the optimal solution of the SDP relaxation for QCQPexists, then there exists a rank one optimal solution Y ∗ of the SDP
relaxation for QCQP, and, furthermore, let Y ∗ =
[1 (x∗)T
x∗ x∗(x∗)T
], then x∗
is optimal for QCQP.
Conic Programming 7 / 46
Exact Solutions from SDP Relaxation
LemmaLet G be a symmetric matrix and X be a positive semidefinite matrix withrank d. Suppose that G •X ≤ (=)0. Then there exists a rank-onedecomposition X =
∑di=1 xix
Ti such that xTi Gxi ≤ (=)0, i = 1, . . . , d.
TheoremWhen m = 2, assume there exists x such that xTAix+ 2bTi x+ ci < 0,i = 1, 2, and there exists y1 ≥ 0, y2 ≥ 0 such that[c0 bT0b0 A0
]+ y1
[c1 bT1b1 A1
]+ y2
[c2 bT2b2 A2
]� 0. If, for the optimal solution Y ∗,
either[c1 bT1b1 A1
]• Y ∗ < 0 or
[c2 bT2b2 A2
]• Y ∗ < 0, then there is no gap
between QCQP and its SDP relaxation, and at least one optimal solutioncan be obtained from the rank-one decomposition of Y ∗.
Conic Programming 8 / 46
Exact Solutions from SDP Relaxation
Theorem(HQCQP)When m = 2, suppose there exists x, such that xTAix+ ci < 0, i = 1, 2,and there exists y1 ≥ 0 and y2 ≥ 0 such that A0 + y1A1 + y2A2 � 0. Thenthere is no gap between HQCQP and its SDP relaxation, and at least oneoptimal solution can be obtained from the rank-one decomposition of X∗.
Conic Programming 9 / 46
One Extension of QCQP
Ext-QCQPMin xTA0x+ 2bT0 x+ c0
s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m(x2
1, . . . , x2n)T ∈ X
(Ext-QCQP)
where X is a closed convex set.SDP Relaxation:
Min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0, i = 1, . . . ,m
(Y22, Y33, . . . , Y(n+1),(n+1))T ∈ X
Y11 = 1
Y � 0, rank(Y ) = 1
Conic Programming 10 / 46
Exact Solutions from SDP Relaxation
TheoremSuppose there is σ ∈ {−1, 1}n+1, such that all the off-diagonal elements
of Λσ
[ci bTibi Ai
]Λσ, 0 ≤ i ≤ m, are nonpositive. Here
(Λσ)ij =
{σi i = j0 i 6= j
If Y ∗ is an optimal solution for the SDP relaxation for Ext-QCQP, then
x∗ = σ1(σ2
√Y ∗22, σ3
√Y ∗33, . . . , σn+1
√Y(n+1),(n+1))
T
is optimal for Ext-QCQP.
Conic Programming 11 / 46
More Results on Ext-QCQP
A Special Case of Ext-QCQP
Min xTAx
s.t. (x21, . . . , x
2n)T ∈ X
where X is a closed convex set.SDP Relaxation:
Min A •Xs.t. (X11, X22, . . . , Xnn)T ∈ X
X � 0
The previous theorem is satisfied automatically. Hence the optimalsolution for this special case of Ext-QCQP can be obtained by its SDPrelaxation.
Conic Programming 12 / 46
Recent Results on QCQP
Content• Introduction• KKT solution and Conic Relaxation for QCQP• Adaptive Approximation to QCQP
Conic Programming 13 / 46
Introduction
QCQP:
Min f(x) = xTA0x+ 2bT0 x+ c0
s.t. gi(x) = xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m,
Ai ∈ Sn, bi ∈ Rn, ci ∈ R, i = 0, 1, . . . ,m.Feasible domain:
feas(QCQP) = {x ∈ Rn|gi(x) ≤ 0, i = 1, . . . ,m}
Conic Programming 14 / 46
Introduction
Previously we have learnt:• Equivalent SOCP for convex QCQP• SDP relaxation for QCQP• m = 1: optimal rank one solution from SDP relaxation• m = 2: optimal solution from non-binding solution of SDP relaxation• m = 2: optimal solution from SDP relaxation for HQCQP• Optimal solution for Ext-QCQP with all nonpositive off-diagonal
coefficients
Conic Programming 15 / 46
Recall
• Nonnegative quadratic functions over F
f(x) = xTAx+ 2bTx+ c ≥ 0,∀x ∈ F
f ⇔[c bT
b A
]• DF = {
[c bT
b A
]∈ Sn+1|
[1x
]T [c bT
b A
] [1x
]≥ 0,∀x ∈ F} is a closed,
convex cone, containing Sn+1+ .
• D∗F = cl cone{[
1 xT
x xxT
]|x ∈ F
}• (D∗F )∗ = DF and (DF )∗ = D∗F• F = Rn =⇒ DF = Sn+1
+ , D∗F = Sn+1+
• F = Rn+ =⇒ DF = Cn+1, D∗F = C∗n+1
Conic Programming 16 / 46
Question
How to use the cone of nonnegative quadratic functions to solve QCQP?
• Relation to KKT solution• Adaptively approximation
Conic Programming 17 / 46
KKT solution and Conic Relaxation forQCQP
Ideas:
Conic Programming 18 / 46
Constraint Qualification: LICQ
Active setGiven a feasible solution x, the active set I(x) is defined as
I(x) = {1 ≤ i ≤ m | gi(x) = 0}.
Linear independence constraint qualification (LICQ)Given a feasible solution x, if the vectors ∇gi(x), i ∈ I(x), are linearlyindependent, then the LICQ condition holds at x.
Conic Programming 19 / 46
Lagrangian Function
Define
A(λ) = A0 +m∑i=1
λiAi, b(λ) = b0 +m∑i=1
λibi, c(λ) = c0 +m∑i=1
λici.
Lagrangian function of QCQP
L(x, λ) = f(x) +m∑i=1
λigi(x)
= xTA(λ)x+ 2bT (λ)x+ c(λ)
where λ ∈ Rm+ .
Conic Programming 20 / 46
Karush-Kuhn-Tucker Conditions
Given a feasible solution x∗, if it satisfies the LICQ and is a localminimum, then there exists a Lagrangian multiplier vector λ∗ such that(x∗, λ∗) is a KKT solution satisfying
∇xL(x∗, λ∗) = 2A(λ∗)x∗ + 2b(λ∗) = 0,
λ∗i gi(x∗) = λ∗i ((x
∗)TAix∗ + 2bTi x
∗ + ci) = 0,
gi(x∗) = (x∗)TAix
∗ + 2bTi x∗ + ci ≤ 0, λ∗i ≥ 0, i = 1, . . . ,m.
Conic Programming 21 / 46
“Copositive Cone” w.r.t. the FeasibleDomain (Cone of Nonnegative QuadraticFunctions)
Define
Dn+1 =
{U ∈ Sn+1
∣∣∣∣ U • [ 1 xT
x xxT
]≥ 0,∀x ∈ feas(QCQP)
}D∗n+1 = cl cone
{X =
[1x
] [1x
]T ∣∣∣∣ x ∈ feas(QCQP)
}
Conic Programming 22 / 46
Sufficient Conditions
Lemma (Positive semidefiniteness condition).Let (x∗, λ∗) be a KKT solution. If A(λ∗) ∈ Sn+, then x∗ is a globally optimalsolution of QCQP.
Theorem (Copositiveness condition)Let (x∗, λ∗) be a KKT solution. If M(x∗, λ∗) ∈ Dn+1, then x∗ is an optimalsolution of the QCQP problem, where
M(x∗, λ∗) =
[c(λ∗)− f(x∗) bT (λ∗)
b(λ∗) A(λ∗)
].
Conic Programming 23 / 46
Recall
Conic Programming 24 / 46
An Equivalent Conic ProgrammingProblem of QCQP
Min
[c0 bT0b0 A0
]• Y
s.t. Y11 = 1,Y ∈ D∗n+1.
(CoP)
TheoremCoP and QCQP have the same optimal objective value.
ProofSee Sturm and Zhang’s paper “On Cones of Nonnegative QuadraticFunctions”.
Conic Programming 25 / 46
Adding Valid Inequality Constraints
CoP1:
Min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0,
i = 1, . . . ,m,
Y11 = 1,Y ∈ D∗n+1.
CoD1:
Max σ
s.t.
[c(λ)− σ bT (λ)b(λ) A(λ)
]∈ Dn+1,
λ ≥Rm+
0.
TheoremIf QCQP is bounded below, then the optimal objective values of QCQP,CoP1 and CoD1 are equal.
Conic Programming 26 / 46
Finding the Lagrangian Multiplier Vector λ∗
AssumptionThere exists a KKT solution (x∗, λ∗) satisfying the LICQ condition andcopositiveness condition.
LemmaUnder the copositiveness condition assumption, (σ∗, λ∗) = (f(x∗), λ∗) isan optimal solution of CoD1. Furthermore, if the LICQ assumption holds,then for any (σ, λ) being an optimal solution of the CoD1 problem, wehave λ ≤Rn
+λ∗.
Conic Programming 27 / 46
Finding the Lagrangian Multiplier Vector λ∗
TheoremLet (x∗, λ∗) be a KKT solution. If it satisfies the LICQ condition and thecopositiveness condition, then λ∗ is the unique optimal solution of themaximization problem CoD2
Max∑mi=1 λi
s.t.
[c(λ)− σ∗ bT (λ)b(λ) A(λ)
]∈ Dn+1,
λ ≥Rn+
0,
(CoD2)
where σ∗ is the optimal objective value obtained by solving CoD1.Furthermore, if A(λ∗) is invertible, then
x∗ = −A−1(λ∗)b(λ∗).
Conic Programming 28 / 46
Relaxing the Conic Constraint
Choose a computable cone Kn+1 such that Sn+1+ ⊂ Kn+1 ⊂ Dn+1.
CoD1’:Max σ
s.t.
[c(λ)− σ bT (λ)b(λ) A(λ)
]∈ Kn+1,
λ ≥Rn+
0.
CoD2’:Max
∑mi=1 λi
s.t.
[c(λ)− σ∗ bT (λ)b(λ) A(λ)
]∈ Kn+1,
λ ≥Rn+
0.
Conic Programming 29 / 46
Algorithm (QCQP)
STEP 1: Given a QCQP problem, solve the corresponding CoD1’ problem andget the optimal value σ∗. If failed, then stop and the problem cannot besolved by the current scheme.
STEP 2: Solve CoD2’ to get the optimal λ∗.STEP 3: Compute x∗ = −A+(λ∗)b(λ∗).STEP 4: If (x∗, λ∗) is a KKT solution, then return x∗ as a global optimal solution
of QCQP with the objective value f(x∗) = σ∗. Otherwise, return σ∗ asa lower bound of QCQP.
Here A+ is the Moore-Penrose generalized inverse for a given squarematrix A ∈ Rn×n.
Conic Programming 30 / 46
Example
Min f(x) = xTA0x+ 2bT0 x
s.t. g1(x) = xTA1x+ 2bT1 x+ c1 ≤ 0,
g2(x) = xTA2x+ 2bT2 x+ c2 ≤ 0,
where
A0 =
−2 10 210 4 12 1 −7
, b0 =
−12−656
,A1 =
2 0 00 −2 00 0 8
, b1 =
0−2−64
, c1 = 512,
A2 =
2 0 00 2 00 0 2
, b2 =
−20−16
, c2 = 128.
Conic Programming 31 / 46
Example
• The feasible domain is a subset of R3+.
• If we choose K4 = S4+ (SDP relaxation), then the optimal value of the relaxed problem
is 222.88.• Instead, choose K4 = S4
+ +N4.
• x∗ = (0, 0, 8)T with f(x∗) = 224 and λ∗ = (1, 2)T .
M(x∗, λ∗) =
320 −16 −8 −40−16 4 10 2−8 10 6 1−40 2 1 5
=
320 −16 −8 −40−16 4 0 2−8 0 6 1−40 2 1 5
+
0 0 0 00 0 10 00 10 0 00 0 0 0
∈ S4+ +N4,
A0 + λ∗1A1 + λ
∗2A2 =
4 10 210 6 12 1 5
/∈ S3+.
Conic Programming 32 / 46
Summary
• Equivalent conic programming problems.
• The copositiveness condition which ensures a KKT solution beingoptimal.
• The maximal property of the Lagrangian multiplier vector under theLICQ condition and the copositiveness condition.
• Conic programming relaxation.• A way to design polynomial time algorithm to solve a class of QCQP.
Conic Programming 33 / 46
Adaptive Approximation to QCQP
Consider a box constrained quadratic program
Min xTA0x+ 2bT0 x+ c0
s.t. x ∈ G = [0, 1]n(BQP)
This problem is in general NP-hard.
Is there any polynomial time approximation?
Find an ellipsoid cover F1 ∪ · · · ∪ Fk ⊃ G such that the problem
Min xTA0x+ 2bT0 x+ c0
s.t. x ∈ F1 ∪ · · · ∪ Fk(BQPR)
satisfies opt(BQP) ≤ opt(BQPR) + ε.
Conic Programming 34 / 46
Conic Reformulation
Let H =
[c0 bT0b0 A0
], BQP becomes
Min H • Y
s.t. Y11 = 1
Y ∈ D∗G
(BCoP)
and BQPR becomes
Min H • Y
s.t. Y11 = 1
Y ∈ D∗F1∪···∪Fk
(BCoPR)
Max σ
s.t. σ
[1 00 0
]+ S = H
S ∈ DF1∪···∪Fk
Conic Programming 35 / 46
Conic Reformulation
Lemma
DF1∪···∪Fk= DF1 ∩ · · · ∩ DFk
and
D∗F1∪···∪Fk= D∗F1
+ · · ·+D∗Fk
Rewrite the relaxation BCoPR as
Min H • Y
s.t. Y11 = 1
Y = Y1 + · · · , YkYi ∈ D∗Fi
, i = 1, . . . , k
(BCRP)
What is DF1∪···∪Fkfor the dual problem of BCRP?
Conic Programming 36 / 46
Conic Reformulation: Dual Problem
Lemma (Linear Matrix Inequalities (LMI))Suppose each ellipsoid Fi = {x ∈ Rn|xTAix+ 2bTi x+ ci 6 0} with someAi � 0 and Int(Fi) 6= ∅. Then, M ∈ DF1∪···∪Fk
if and only if the followingsystem is feasible:
M + λi
[ci bTibi Ai
]� 0, λi > 0 for i = 1, 2, ..., k.
Dual of BCoRP:Max σ
σ
[1 00 0
]+ S = H
S + λi
[ci bTibi Ai
]� 0, λi ≥ 0,
i = 1, . . . , k.
(BCRD)
Conic Programming 37 / 46
Optimality Condition
(Y ∗, Y ∗1 , ..., Y∗k ) is feasible to BCRP,
(σ∗, λ∗, S∗) is feasible to BCRD,
S∗ · Y ∗i = 0 and λ∗i
[bi cTici Qi
]• Y ∗i = 0, i = 1, 2, ...k.
Conic Programming 38 / 46
Some Properties
LemmaS∗ is on the boundary of DF1∪···∪Fk
.
Lemma (Rank-one decomposition)Y ∗ can be decomposed in the form of
Y ∗ =k∑i=1
ni∑j=1
µij
[1xij
] [1xij
]T
with xij ∈ Fi, µij > 0 and∑ki=1
∑ni
j=1 µij = Y ∗11 = 1.Moreover, [
1xij
]TS∗[
1xij
]= 0,∀xij
Conic Programming 39 / 46
Improving The Relaxation
TheoremIf any xij decomposed from Y ∗ is feasible for BQP, then it is an optimalsolution of BQP.
However, in general, they are not feasible.
Let x = xij = arg mini,j
[1xij
]TH
[1xij
]with corresponding Fi.
Notice that for Fi, the feasible part is Fi ∩ G. If we could cover Fi ∩ G bysmaller sets, whose union does not contain x, then the solution of thenew cover may be improved.
Conic Programming 40 / 46
Improving The Relaxation
For simplicity, the ellipsoids have to be constructed carefully.
DefinitionLet T = {x ∈ Rn|ui 6 xi 6 vi} be a rectangular set in Rn, define aquadratic function
gT (x) =
n∑i=1
(2xi − vi − ui)2
(vi − ui)2− n for x ∈ Rn,
and an ellipsoidal set FT = {x ∈ Rn|gT (x) 6 0}. Then we say FT is thecorresponding ellipsoid of T .
Conic Programming 41 / 46
Algorithm
STEP 1: Set T = G and Nmax. Set k = 1 and let the cover be F1 = FT .STEP 2: Solve BCRP and BCRD defined by the cover F1 ∪ · · · ∪ Fk to obtain Y ∗
and (σ∗, λ∗, S∗). Set lk = σ∗.STEP 3: Decompose Y ∗ to obtain x and Fi.STEP 4: If x ∈ G, then x is optimal. Stop and return.
Otherwise, set T be the rectangle covered by Fi.Find i∗ = arg maxi{max(xi − 1,−xi)}.Partition T into T1 = {x ∈ T |xi∗ ≤ ui∗+vi∗
2} and
T2 = {x ∈ T |xi∗ ≥ ui∗+vi∗2}.
Drop Fk from the cover, and add FT1 and FT2 into the cover.Set k = k + 1.
STEP 5: If k < Nmax, then go to STEP 2. Otherwise return max{l1, ..., lk} as alower bound for BQP.
Conic Programming 42 / 46
Convergence
TheoremLet l∗k = maxi=1,...,k{li} be the largest lower bound up to the kth iteration,then limk→+∞ l∗k = opt(BQP ).
Conic Programming 43 / 46
Example
Min xTA0x+ 2bT0 x
s.t. x ∈ [0, 1]10
with
A0 =
54 24 −79 88 11 −109 −76 −14 250 8124 303 −24 −61 24 −6 −5 112 85 76−79 −24 72 28 40 −9 64 −25 25 −388 −61 28 −81 −132 16 71 7 148 4411 24 40 −132 −86 −2 69 −7 14 22−109 −6 −9 16 −2 153 −34 −52 24 65−76 −5 64 71 69 −34 −149 −108 −11 113−14 112 −25 7 −7 −52 −108 49 −21 −106250 85 25 148 14 24 −11 −21 22 −8081 76 −3 44 22 65 113 −106 −80 −179
and
b0 =1
2[84 − 89 10 − 54 30 − 60 49 74 171 − 19]T .
Conic Programming 44 / 46
Example
The optimal value is -611.
0 10 20 30 40 50−670
−660
−650
−640
−630
−620
−610
Iterations
Low
er B
ound
s
Adaptive BoundSDP Bound
Conic Programming 45 / 46
Summary
• Developed an adaptive relaxation for BQP.• It applies to convex constrained QCQP too.• It may be applied to general QCQP, but needs well-designed covers.• Other techniques such as RLT can be involved in the relaxation.
Conic Programming 46 / 46
LCoP Part VII – Practical LCoP
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
December 2017
Conic Programming 1 / 40
Practical LCoP
Content• Convex Cones and Properties• Different Forms of LCoP• SOCP Formulation• SDP Formulation
Conic Programming 2 / 40
Convex Cones and Properties
K, K1, K2 are convex cones in Rn.• (K∗)∗ = cl(K).• K1 ⊆ K2 ⇒ K∗2 ⊆ K∗1 .• K1 ∩K2, K1 ∪K2, K1 +K2, K1 ×K2 are all cones.• (K1 +K2)∗ = K∗1 ∩K∗2 .• K1, K2 closed⇒ K1 +K2 and K1 ×K2 are closed.• ri(K1 +K2) = ri(K1) + ri(K2).• ri(K1 ×K2) = ri(K1)× ri(K2).• The supporting hyperplane of K always contains the origin• If K is solid, then K∗ is pointed.• If K is pointed, then K∗ is solid.• If K is proper, then K∗ is proper.
Conic Programming 3 / 40
Convex Cones and Properties
K, K1, K2 are convex cones in Rn.• Rn+ ⊆ Rn, Ln ⊆ Rn, Sn+ ⊆ Rn×n are pointed, closed, convex and
solid cones, i.e., proper cones.• (Rn+)∗ = Rn+, (Ln)∗ = Ln, (Sn+)∗ = Sn+.
• Rn1+ ×R
n2+ × · · · ×R
nk+ is a proper cone in R
k∑i=1
ni
.
• Ln1 × Ln2 × · · · × Lnk is a proper cone in Rk∑
i=1ni
.
• Sn1+ × S
n2+ × · · · × S
nk+ is a proper cone in R
k∑i=1
ni×ni
.• Rn1
+ × Ln2 × Sn3+ is a proper cone in Rn1+n2+n3×n3 .
Conic Programming 4 / 40
Different Forms of LCoP
• LP (Given A ∈ Rm×n,b ∈ Rm, ai ∈ Rn, bi ∈ R and c ∈ Rn)
Min cTxs.t. Ax = b
x ≥ 0(LP)
Max bT ys.t. AT y ≤ c
y ∈ Rm(LD)
Min c • xs.t. ai • x = bi, i = 1, . . . ,m
x ∈ Rn+(ai ∈ Rn, bi ∈ R)
(LP)Max bT ys.t. AT y + s = c
y ∈ Rm, s ∈ Rn+(LD)
Max b • ys.t. c−AT y ∈ Rn+
y ∈ Rm(LD)
Conic Programming 5 / 40
Different Forms of LCoP
• SOCP (Given A ∈ Rm×n, b ∈ Rm, c ∈ Rn, ai ∈ Rn, bi ∈ R)
Min cTxs.t. Ax = b
x ∈ Ln(SOCP)
Max bT ys.t. AT y + s = c
y ∈ Rm, s ∈ Ln(SOCD)
Min c • xs.t. ai • x = bi, i = 1, . . . ,m
x ≥Ln 0(SOCP)
Max bT ys.t. c−AT y ∈ Ln
y ∈ Rm(SOCD)
Conic Programming 6 / 40
Different Forms of LCoP
• SDP (Given Ai ∈ Rn×n, bi ∈ R and C ∈ Rn×n)
Min C •Xs.t. Ai •X = bi, i = 1, · · · ,m
X � 0(SDP)
Max bT y
s.t.m∑i=1
yiAi + S = C
y ∈ Rm, S ≥Sn+
0
(SDD)
Min C •Xs.t. Ai •X = bi, i = 1, · · · ,m
X ∈ Sn+(SDP)
Max bT y
s.t. C −m∑i=1
yiAi � 0
y ∈ Rm(SDD)
Max bT y
s.t. C −m∑i=1
yiAi ∈ Sn+yi ∈ R, i = 1, . . . .m
(SDD)
Conic Programming 7 / 40
Different Forms of LCoP
• Combinations
Min pTx+ qT ys.t. Ax+By = b
x ≥ 0, y ∈ Lm
Max pTx+ qT y + rT z (linear objective)s.t. Ax+By ≤ b (linear inequalities)
C −k∑i=1
ziAi � 0 (linear matrix inequalities)
x ≥ 0, y ∈ Lm, z ∈ Rk
Conic Programming 8 / 40
Second Order Cone Program (SOCP)Formulation
• Combinations
Min c • xs.t. ai • x = bi, i = 1, . . . ,m
x ∈ K
where K = Ln1 × · · · × Lnr = {x ∈ Rn |r∑i=1
ni = n, (x1, · · · , xn1) ∈
Ln1 , · · · , (xn−nr+1, · · · , xn) ∈ Lnr}, c, ai ∈ Rn, bi ∈ R.
Conic Programming 9 / 40
Second Order Cone Program (SOCP)Formulation
• For easy thinking, take r = 1 and n = n1.
Min cTxs.t. Ax = b
x ∈ K[Conic variables]
(SOCP)
Max bT ys.t. AT y + S = C
y ∈ Rm, S ∈ K(i.e., C −AT y ∈ K)[Conic constraints]
(SOCD)
Conic Programming 10 / 40
SOCP Formulation
• Generalization:Since linear constraints are allowed, instead of considering x ∈ Ln,we may consider
y = Mx+ q ∈ Lm
i.e., a linear transformation of x in an SOC.
• Denote M =
(Am−1×ncT1×n
), q =
(bm−1×1d1×1
), we have
y =
(AcT
)x+
(bd
)=
(Ax+ bcTx+ d
)∈ Lm
⇔ ‖Ax+ b‖ ≤ cTx+ d
Conic Programming 11 / 40
SOCP Formulation
• (SOCP) becomes
Min cTxs.t. ‖Ax+ b‖ ≤ cTx+ d
x ∈ Rn
where A ∈ R(m−1)×n, b ∈ Rm−1, c ∈ Rn, d ∈ R.• More generally, (SOCP) may be represented as
Min cTxs.t. ‖Aix+ bi‖ ≤ cTi x+ di, i = 1, · · · , r
x ∈ Rn
where Ai ∈ R(ni−1)×n, b ∈ Rni−1, ci ∈ Rn, di ∈ R, and x ∈ Rn with
n =r∑i=1
ni.
Conic Programming 12 / 40
Questions
• Are the following problems SOCP?
Minp∑i=1
1/(aTi x+ bi)
s.t. aTi x+ bi > 0, i = 1, · · · , pcTj x+ dj ≥ 0, j = 1, · · · , q
Min xTP0x+ 2qT0 x+ r0s.t. xTPix+ 2qTi x+ ri ≤ 0, i = 1, · · · , p
P0, P1, · · · , Pp ∈ Sn++
Conic Programming 13 / 40
SOC-Representable Sets and Functions
• Definition (SOC-Representable Set)
A set S ⊆ Rn is SOC-representable if there exists a linear transformation
L(x) = Mx+ q for x ∈ Swith M = (·)m×n and q ∈ Rm, such that
L(S) =
K∏i=1
Lmi for some K > 0 andK∑i=1
mi = m.
• Note that S is SOC-representable if and only if S is the x-spaceprojection of the solution set of K(> 0) SOC inequalities:
Mi
(xµ
)+ qi ∈ Lmi , i = 1, ...,K,
i.e., S = {x ∈ Rn|Mi
(xµ
)+ qi ∈ Lmi , i = 1, ...,K}.
Conic Programming 14 / 40
SOC-Representable Sets and Functions
• Theorem: If S1,S2 ⊆ Rn are SOC-representable sets, then
αS1(α ≥ 0),S1 ∩ S2,S1 × S2 are SOC-representable.
• Definition (SOC-Representable Function)
A function g(·) : Rn → R is SOC-representable if its epigraph
Epi(g) = {(x, t) ∈ Rn+1 | g(x) ≤ t} is an SOC representable
set in Rn+1.
Conic Programming 15 / 40
SOC-Representable Sets and Functions
• Theorem
If a function g(·) : Rn → R is SOC-representable, then any of its
lower level sets Gα = {x ∈ Rn | g(x) ≤ α} is an SOC-representable set.
Conic Programming 16 / 40
SOCP for NLP
min f(x)s.t. gj(x) ≤ 0 j = 1, ..., J (NLP )
x ∈ Rn
⇐⇒min ts.t. f(x) ≤ t
gj(x) ≤ 0 j = 1, ..., Jx ∈ Rn, t ∈ R
Conic Programming 17 / 40
SOCP for NLP
• TheoremIf f(·) and gj(·), j = 1, 2, ..., J , are all SOC-representable functions, then(NLP) becomes an SOCP.
Conic Programming 18 / 40
Identify SOC-Representable Functions
• TheoremIf f1 and f2 are SOC-representable functions, then αf1(α ≥ 0); f1 + f2and max {f1, f2} are SOC-representable.
• TheoremIf f1 and f2 are convex and SOC-representable with f1(x) beingmonotone nondecreasing, then g(x) = f1(f2(x)) is convex andSOC-representable.
Conic Programming 19 / 40
Identify SOC-Representable Functions
• Hyperbolic Constriants
Theorem: For w, x, y ∈ R,
w2 ≤ xy, x ≥ 0, y ≥ 0⇔∥∥∥∥( 2wx− y
)∥∥∥∥ ≤ x+ y ⇔
2wx− yx+ y
∈ L3.
Theorem: For w ∈ Rn, x, y ∈ R,
wTw ≤ xy, x ≥ 0, y ≥ 0⇔∥∥∥∥( 2wx− y
)∥∥∥∥ ≤ x+ y ⇔
2wx− yx+ y
∈ Ln+2.
• Quadratic Constraints
Theorem: For Q ∈ Sn++, q ∈ Rn, r ∈ R,
xTQx+ 2qTx+ r ≤ 0⇔∥∥∥Q1/2x+Q−1/2q
∥∥∥ ≤ (qTQ−1q − r)1/2.
Conic Programming 20 / 40
SOC-Representable Functions
• Constant function
g(x) ≡ c, x ∈ Rn, c ∈ R
Consider the epigraph{(
xt
)∈ Rn+1 | c ≤ t
}.
Set A = (0)m×n, then ||Ax|| ≤ t− c, i.e.,(
Axt− c
)∈ Lm+1.
Conic Programming 21 / 40
SOC-Representable Functions
• Linear function
g(x) = aTx+ b, a ∈ Rn, b ∈ R
Consider the epigraph{( x
t
)| aTx+ b ≤ t
}.
Set C = (0)p×n, then the set can be represented by||Cx|| ≤ t− aTx− b.
Conic Programming 22 / 40
SOC-Representable Functions
• Square Root of Homogeneous Convex Quadratic Function
g(x) =√xTAx,A ∈ Sn+
The epigraph is{( x
t
)|√xTAx ≤ t
}.
Since A ∈ Sn+, there exists matrix B satisfying A = BTB, we have√(Bx)T (Bx) ≤ t, i.e., ‖Bx‖ ≤ t.
Conic Programming 23 / 40
SOC-Representable Functions
• Convex Quadratic Function
g(x) = xTAx+ bTx+ c, A ∈ Sn+, b ∈ Rn, c ∈ R
g(x) ≤ t is equivalent to√(Bx)TBx+
(t− bTx− c− 1)2
4≤ t− bTx− c+ 1
2,
where B is the matrix satisfying A = BTB, i.e.,∥∥∥∥ Bxt−bT x−c−1
2
∥∥∥∥ ≤ t− bTx− c+ 1
2.
Conic Programming 24 / 40
SOC-Representable Functions
• Reciprocal Function
g(x) = 1x , x > 0
g(x) ≤ t is equivalent to xt ≥ 1, x > 0, hence we have√1 +
(x− t)24
≤ x+ t
2, x > 0, t ≥ 0.
i.e., ∥∥∥∥ 1x−t2
∥∥∥∥ ≤ x+ t
2.
Conic Programming 25 / 40
SOC-Representable Functions
• Maximum Function
g(x) = max{x1, x2, ..., xn}
The epigraph{( x
t
)| max{x1, x2, ..., xn} ≤ t
}={( x
t
)|
t− x1 ≥ 0, ..., t− xn ≥ 0}
.
Conic Programming 26 / 40
SOC-Representable Functions
• Sum of the norms
g(x) =∑pi=1 ||Fix+ gi||, Fi ∈ Rni×n, gi ∈ Rni(
xt
)satisfies g(x) ≤ t is equivalent to ∃ t1, t2, ..., tp such that
||Fjx+ gj || ≤ tj , j = 1, ..., p and t1 + ...+ tp ≤ t.
• Max of the norms
g(x) = maxi=1,...,p ||Fix+ gi||, Fi ∈ Rni×n, gi ∈ Rni
Conic Programming 27 / 40
SOC-Representable Functions
• Sum of the fractional functions
g(x) =∑pi=1 1/(aTi x+ bi), where aTi x+ bi > 0, i = 1, ..., p(
xt
)satisfies g(x) ≤ t is equivalent to ∃ t1, t2, ..., tp such that
ti(aTi x+ bi) ≥ 1, i = 1, ..., p and t1 + ...+ tp ≤ t, i.e., ∃ t1, ..., tp such
that∥∥∥∥( 2
aTi x+ bi − ti
)∥∥∥∥ ≤ aTi x+ bi + ti, i = 1, ..., p and
t1 + ...+ tp ≤ t.
Conic Programming 28 / 40
SOC-Representable Functions
• Sum of the fractional functions
g(x) =∑pi=1
||Fix+gi||2aTi x+bi
, where aTi x+ bi > 0, i=1,...,p(xt
)satisfies g(x) ≤ t is equivalent to ∃ t1, t2, ..., tp such that
(Fix+ gi)T (Fi + gi) ≤ ti(aTi x+ bi), i = 1, ..., p and t1 + ...+ tp ≤ t,
i.e., ∃ t1, ..., tp such that∥∥∥∥( Fix+ giaTi x+ bi − ti
)∥∥∥∥ ≤ aTi x+ bi + ti, i = 1, ..., p and t1 + ...+ tp ≤ t.
Conic Programming 29 / 40
SOC-Representable Functions
• Geometrical mean
g(x) = −(x1...xn)1/n, x ∈ Rn+, where n = 2q, q > 0
• Geometrical mean of nonnegative affine functions
g(x) = −
(p∏i=1
(aTi x+ bi)
)1/p
with aTi x+ bi ≥ 0
Conic Programming 30 / 40
Semidefinite Programming (SDP)Formulation
• Let C,Ai ∈ Sn×n, b ∈ Rm, bi ∈ R.
Min C •Xs.t. Ai •X = bi, i = 1, · · · ,m
X ∈ Sn+(SDP)
Max bT y
s.t.m∑i=1
yiAi + S = C
y ∈ Rm, S ∈ Sn+
(SDD)
Min C •Xs.t. Ai •X = bi, i = 1, . . . ,m
X � 0(SDP)
Max bT y
s.t. C −m∑i=1
yiAi � 0
y ∈ Rm(SDD)
Conic Programming 31 / 40
SDP Formulation
• SDD is often easier to “see” and “work” on, since it has only mvariables. We may have different interpretations of SDD.
- Linear Matrix Inequality Constraint
C −m∑i=1
yiAi � 0
y ∈ Rm
- PSD conic constraint
C −m∑i=1
yiAi ∈ Sn+
y ∈ Rm
Conic Programming 32 / 40
SDP Formulation
• SDD is often easier to “see” and “work” on, since it has only mvariables. We may have different interpretations of SDD.
- Linear transformation of variables in the cones11 = c11 −A1(1, 1)y1 −A2(1, 1)y2 − · · · −Am(1, 1)ym
sij = cij −A1(i, j)y1 −A2(i, j)y2 − · · · −Am(i, j)ym
snn = cnn −A1(n, n)y1 −A2(n, n)y2 − · · · −Am(n, n)ym
S = (sij) ∈ Sn+
Conic Programming 33 / 40
SDP Formulation
• Useful Facts
1.(xt
)∈ Ln+1 ⇔
[t xT
x tIn
]∈ Sn+1
+
Any SOCP is an SDP!
2. Let M =
(P vvT d
), P � 0, v ∈ Rn, d ∈ R.
M � (�)0 if and only if d− vTP−1v ≥ (>) 0.
3. Let M =
(P BBT C
), P � 0, B,C proper size.
M � (�)0 if and only if S = C −BTP−1B � (�) 0.
Conic Programming 34 / 40
SDP Formulation
• Useful Facts
4. For x ∈ Rn, M = xxT ∈ Sn+.
5. For x ∈ Rn and W ∈ Sn, then(
1 xT
x W
)∈ Sn+1
+ if and only if
W � xxT .
Conic Programming 35 / 40
SDP Identification
• Types of constraints that can be modeled as SDP
1. Convex quadratic constraints (Q ∈ Sn+, q ∈ Rn, c ∈ R)
xTQx+ qTx+ c ≤ 0⇔ xT (MTM)x+ qTx+ c ≤ 0
⇔(
I MxxTMT −qTx− v
)� 0
• Convex QCQPMin xTQ0x+ qT0 x+ c0s.t. xTQix+ qTi x+ ci ≤ 0 i = 1, . . . ,m
x ∈ Rn
Conic Programming 36 / 40
SDP Identification
• Convex QCQP
Min θs.t. xTQ0x+ qT0 x+ c0 ≤ θ
xTQix+ qTi x+ ci ≤ 0 i = 1, . . . ,mx ∈ Rn
Min θ
s.t.
(I M0x
xTM0 −qTx− c0 + θ
)� 0(
I MixxTM0 −qTx− ci
)� 0, i = 1, . . . ,m
x ∈ Rn, θ ∈ R
Conic Programming 37 / 40
SDP Identification
2. Binary constraints (x ∈ Rn)
xj ∈ {−1, 1}, j = 1, . . . , n
Y = xxT � 0 (rank-one matrix constraint)
Yjj = 1
3. Distance constraints
‖x− a‖2 ≤ 1
⇔ (x− a)T (x− a) ≤ 1
⇔(
I x− a(x− a)T 1
)� 0
Conic Programming 38 / 40
SDP Identification
4. Elliposid constraints (Q � 0)
(x− a)TQ(x− a) ≤ 1
⇔ (x− a)T√Q√Q(x− a) ≤ 1
⇔(
I√Qx−
√Qa
(√Qx−
√Qa)T 1
)� 0
5. Eigenvalue constraints
λI � S � µI (µ > λ ≥ 0)
a psd with eigenvalues between λ and µ.
Conic Programming 39 / 40
SDP Identification
6. Fractional constraints
(cTx)2
dTx≤ t
⇔(
t cTxcTx dTx
)� 0
Conic Programming 40 / 40
Top Related