Agenda

1 Duality for LP2 Theorem of alternatives3 Conic Duality4 Dual cones5 Geometric view of cone programs6 Conic duality theorem7 Examples

Lower bounds on LPs

By eliminating variables (if needed) any LP can be put in inequality form

minimize c

T

x

subject to Ax � b

We use this form for simplicity

Purpose of Lagrange duality: get a lower bound on optimal value p

?

Lagrange duality

Lagrangian: L : Rn ⇥ Rm ! R

L(x,�) = c

T

x� �

T (Ax� b) �: Lagrange multipliers

(Lagrange) dual function

g(�) = infx2Rn L(x,�) =

(b

T

� A

T

� = c

�1 otherwise

Suppose � � 0 & x is feasible

L(x,�) = c

T

x� �

T (Ax� b) c

T

x

This implies that g(�) c

T

x for all feasible x

) g(�) infx:Ax�b

c

T

x = p

?

! Dual function always gives a lower bound on optimal value p

?

Dual problem

Dual problem: best lower bound on p

?

maximize g(�)subject to � � 0

()maximize b

T

�

subject to A

T

� = c

� � 0

(Lagrange) dual problem

This is an LP

Optimal value d

? obeys d? p

?: weak duality

LP duality

Theorem

For LP, strong duality always holds (remarkable and very useful fact)

d

? = p

?

p

? = 1 (primal infeasible) =) d

? = p

?

p

? = �1 =) d

? = �1 (dual infeasible)

Theorem of alternatives and strong LP duality

min {cTx : Ax � b}

How do we decide if LP is feasible?

How can we certify feasible set is empty?

One possible approach: combine inequalities and get a contradiction

� � 0 and Ax � b =) �

T

Ax � �

T

b

Get such a contradiction if AT

� = 0 and b

T

� > 0 (only way to be sure)

Remarkably, this is su�cient but also necessary

Theorem (Theorem of alternatives)

Exactly one of (1) and (2) is true

(1) 9x s.t. Ax � b

(2) 9� � 0 s.t. AT

� = 0 and b

T

� > 0

Lower bounds for LPs

min {cTx : Ax � b}

a p

? , {x : Ax � b and c

T

x < a} = ;

Theorem of alternatives: a p

? i↵

9(�0,�) � 0 s .t.��0c+A

T

� = 0��0 + b

T

� � 0

If LP is feasible, can take �0 > 0, hence a p

? i↵ system below has a solution

(D)� � 0A

T

� = c

b

T

� � a

Proposition

(D) has a sol (�, a) =) a p

?

LP is feasible and a p

? =) existence of a sol to (D)

Strong duality

Proposition

(D) has a sol (�, a) =) a p

?

LP is feasible and a p

? =) existence of a sol to (D)

(�, a) sol to (D) =) (�, bT�) sol to (D)

Since b

T

� � a, best lower bound

max {bT� : AT

� = c,� � 0}

ConclusionThis proves strong duality

� feasible for dual problem =) b

T

� p

? (d? p

?)

If LP feasible, then for every a p

?, 9 dual feasible � with b

T

� � a

) d

? = p

?

Proof of theorem of alternatives

Similar notion for cone programs?

Conic programminimize c

T

x

subject to Ax ⌫K

b

Interested in something similar to LP’s

� � 0 and Ax� b � 0 =) h�, Ax� bi � 0

For cones, we would like a set C such that

� 2 C and a ⌫K

0 =) h�, ai � 0

If � 2 C and x is feasible, we get a lower bound

hc, xi � h�, Ax� bi hc, xi

Or similarly

min {cTx : Ax ⌫K

0}

For LP’s: � � 0 gives

Ax � b =) �

T

Ax � �

T

b

If AT

� = c, we get a lower bound

For cones, we would like a set C such that

� 2 C and a ⌫K

0 =) h�, ai � 0

Best lower boundmax {bT� : � 2 C and A

T

� = c}

Dual cones

DefinitionDual cone K

⇤

K

⇤ = {� 2 Rn : �T

a � 0 8a 2 K}

Properties of dual cone

K 6= ; (nonempty set)

1K

⇤ is a closed convex cone2 int(K) 6= ; =) K

⇤ is pointed3

K closed, convex and pointed =) int(K⇤) 6= ;

4K closed, convex and pointed =) K

⇤ closed and convex(K⇤)⇤ = K

Corollary: K proper () K

⇤ proper

Dual of a conic problem

Primal

(P)minimize c

T

x

subject to Ax ⌫K

b

LagrangianL(x,�) = c

T

x� �

T (Ax� b) � 2 K

⇤

Dual

(D)maximize b

T

�

subject to A

T

� = c

� 2 K

⇤

Weak duality: d⇤ p

⇤

x feasible, � feasible =) b

T

� c

T

x

Dual of conic problem, continued

(P)minimize c

T

x

subject to A

i

x ⌫Ki bi, i = 1, . . . ,m

(D)maximize

Pb

T

i

�

i

subject toP

A

T

i

�

i

= c

�

i

⌫K

⇤i0, i = 1, . . . ,m

Geometrical view of primal/dual pair

(P)minimize c

T

x

subject to Ax ⌫K

b

(D)maximize b

T

�

subject to A

T

� = c

� ⌫K

⇤ 0

Primal: minimize linear function over an a�ne slice of cone K

Dual: maximize linear function over an a�ne slice of cone K

⇤

Same problem: only di↵erence due to how we represent the problem

Assume 9d : c = A

T

d () c 2 row(A)otherwise dual is infeasible: d? = �1 and p

? 2 {±1}Assume for simplicity A has full column rank (so that c = A

T

d as a solution)

Primal feasible set: v = Ax� b

(i) v 2 V � b, V = range(A)

(ii) v ⌫K

0

(iii) Gives cTx = d

T

Ax = d

T

b+ d

T

v

Conic program can be formulated as

min {dT b+ d

T

v : v 2 V � b, v ⌫K

0}

Dual feasible set:

A

T

� = c,� 2 K

⇤ () � 2 V

? + d, � 2 K

⇤

So that dual problem is

max {bT� : � 2 V

? + d,� ⌫K

⇤ 0}

Geometric view of primal/dual pair, continued

K*

K

pri

ma

l fe

as

ible

R - b

dual feasible

Symmetry of duality

(P)minimize c

T

x

subject to Ax ⌫K

b

(D)maximize b

T

�

subject to A

T

� = c

� ⌫K

⇤ 0

If K is proper, then (P) is dual to (D) because K

⇤⇤ = K

dual of (D)minimize �⌫

T

c

subject to v ⌫K

0v = �A⌫ � b

() minimize c

T

x

subject to Ax� b ⌫K

0

Examples of dual cones

The following are all self-dual cones:

(a) K = Rn

+

K

⇤ = {� :X

�

i

a

i

� 0 8a � 0} = Rn

+

(b) K = L ⇢ Rn+1, L = {(x, t) : kxk t}

(y, s) 2 K

⇤ () (y, x) + st � 0 8(x, t) 2 L

t

x

L = {(x, t) : kxk t}

(y, s) 2 L

⇤ () hy, xi+ st � 0 8(x, t) 2 L

Claim: L⇤ = L

(i) L ⇢ L

⇤

Take (y, s) 2 L,hy, xi+ st � �kykkxk+ st � 0

(ii) L

⇤ ⇢ L

Take (y, s) 2 L

⇤ and set (x, t) = (�y, kyk) 2 L

�hy, yi+ skyk � 0 =) kyk s () (y, s) 2 L

(c) K = S

n

+

K

⇤ = {⇤ 2 S

n : Tr(⇤X) � 0 8X ⌫ 0}Claim: K⇤ = K

(i) K

⇤ ⇢ S

n+

Take ⇤ 2 K

⇤. Then for all x 2 Rn

Tr(⇤xxT ) = Tr(xT⇤x) = x

T⇤x � 0

and so ⇤ ⌫ 0

(ii) S

n+ ⇢ K

⇤

Take ⇤ ⌫ 0 and decompose X ⌫ 0 as X =P

�kxkxTk ,with �k � 0

Tr(X⇤) =X

k

�kTr(xkxTk ⇤) =

X

k

�kxTk ⇤xk � 0

Implication of conic self duality

Primal LP SOCP SDP

Dual LP SOCP SDP

Example: minimum total-variation denoising

minimize kxkTVsubject to kx� bk �

()minimize

Pt

ij

subject to kx� bk �

kDij

xk t

ij

Constraints

kx� bk � ()x� b

�

�2 L0 ()

I 00 0

� x

t

�⌫

L0

b

��

�

(Dij

x, t

ij

) 2 L

ij

()D 00 I

� x

t

�⌫

L

0 L =Y

L

ij

Dual

maximize �

T

0

b

��

�

subject to �0 ⌫L0 0 �

ij

⌫Lij 0

�I 00 0

� u0

s0

��D

T 00 I

� u

s

�+

01

�= 0

Dual variables

�0 =

u0

s0

�: ku0k s0 �

ij

=

u

ij

s

ij

�: ku

ij

k s

ij

Dual problem (SOCP)

max hu0, bi � s0�

s. t. ku0k s0, kuij

k s

ij

u0 +D

T

u = 0s = 1

() max �hb,DT

ui � �kDT

uks. t. ku

ij

k 1

Example: SDP

maximize hc, xisubject to x1F1 + . . .+ x

m

F

m

⌫ F0 (Fi

2 S

p)

Dual cone problem

maximize hF0,⇤isubject to ⇤ ⌫ 0

hFi

,⇤i = c

i

, i = 1, . . . ,m

minimize kA0 � (x1A1 + . . .+ x

m

A

m

)k

SDP formulation

minimize t

subject to

tI A(x)

A

T (x) tI

�⌫ 0

with A(x) = A0 � (x1A1 + . . .+ x

m

A

m

)

LMI () t

I 00 I

��X

i

x

i

0 A

i

A

T

i

0

�+

0 A0

A

T

0 0

�⌫ 0

Dual problem

maximize h�0 A0

A

T

0 0

�,⇤i

subject to ⇤ ⌫ 0

h0 A

i

A

T

i

0

�,⇤i = 0

hI 00 I

�,⇤i = 1

With ⇤ =

⇤11 ⇤12

⇤T

12 ⇤22

�, this gives

maximize �2Tr(AT

0 ⇤12)subject to Tr(AT

i

⇤12) = 0Tr(⇤) = 1⇤ ⌫ 0

()

maximize �2Tr(AT

0 ⇤12)subject to Tr(AT

i

⇤12) = 0Tr(⇤11) + Tr(⇤22) = 1⇤11 ⇤12

⇤T

12 ⇤22

�⌫ 0

Conic duality theorem

Can get results paralleling LP duality provided primal is strictly feasible

Strict feasibility means that V � b \ int(K) 6= ;

Theorem

(1) Assume (P) is strictly feasible (9x : Ax �K

b) and p

? 6= �1. Then (D) issolvable and d

? = p

?.

(2) Assume (D) is strictly feasible (9� : � �K

⇤ 0, AT

� = c) and d

? 6= 1. Then(P) is solvable and d

? = p

?.

(3) Assume one of the problems is bounded and strictly feasible. Then (x,�)primal/dual optimal i↵

b

T

� = c

T

x () �

T (Ax� b) = 0

Proof(1) Claim: If (P) is strictly feasible and 9x : Ax �

K

b

then (D) is solvable and d

? = p

?

Enough to show that 9� dual feasible s.t. bT� � p

?

If c = 0, then p

? = 0 and �

? = 0 does the job. So we assume c 6= 0.

M = {Ax� b : x 2 Rn

, c

T

x p

?}

M 6= ; (image of halfspace)

K

R - b

M d

M \ int(K) = ;Why? Assume v 2 M \ int(K) () v = Ax0 � b with Ax0 � b �

K

0 andc

T

x0 p

? and hence x0 is primal optimal9✏ ⌫ 0 s.t. Ax� b � 0 8x : kx� x0k ✏

c 6= 0 =) 9x s.t. kx� x0k ✏ and c

Tx < c

Tx0 p

?

=)(=

M is convex, nonempty, and M \ int(K) = ;

Separation theorem for convex sets: 9� 6= 0 s.t

supv2M

h�, vi infv2int(K)

h�, vi

hence �

T

v bounded below on int(K)

=) �

T

y � 0 8y 2 int(K)=) �

T

y � 0 8y 2 K

() � 2 K

⇤

In summary

supv2M

h�, vi 0 & � 2 K

⇤ () � 2 K

⇤

�

T (Ax� b) 0 8x : cTx p

?

! Linear form bounded above on a half-plane. This is possible only if

A

T

� = µc µ � 0

We claim µ > 0Why? µ = 0 =) A

T

� = 0, �

T

b � 0Take x0 s.t. Ax0 � b �

K

0

� ⌫K

⇤ 0 and � 6= 0 =) �

T (Ax0 � b) > 0 =) ��

T

b > 0

This is a contradiction

�

? = µ

�1� obeys

�

? 2 K

?

, A

T

�

? = c, hc, xi h�?

, bi 8x : cTx p

?

Taking an x such that cTx = p

? =) value of dual objective is at least p?

End of proof

(2) Same by symmetry

(3) x primal feasible, � dual feasible =) c

T

x� b

T

� = (Ax� b)T�

! Complementary slackness i↵ zero duality gap

Enough to show (x,�) optimal () DG = 0

DG is zero gives

c

T

x� b

T

� = c

T

x� p

? + d

? � b

T

�+ p

? � d

? = 0

since c

T

x� p

? = 0 and d

? � b

T

� = 0 must holdIn conclusion,

x primal optimal� dual optimal

nota bene: no strict feasibility required

Conversely, p? = d

? =) c

T

x = b

T

�

Example

(P) minimize c

T

x

subject to x1F1 + . . .+ x

n

F

n

⌫ F0

(D) maximize hF0,⇤isubject to ⇤ ⌫ 0

hFi

,⇤i = c

i

Primal / dual strict feasibility

(x,⇤) optimal () c

T

x� hF0,⇤i = hX

x

i

F

i

� F0,⇤i = 0

Example for which p? > d?

minimize x1

subject to

2

40 x1 0x1 x1 + x2 00 0 x1 + 1

3

5 ⌫ 0

Feasible set:0 x1

x1 x1 + x2

�⌫ 0

x1 + 1 � 0()

x1 + x2 � 0�x

21 � 0

x1 � �1() x1 = 0

x2 � 0

) p

? = 0

Dual 2

40 0 00 0 00 0 1

3

5+ x1

2

40 1 01 1 00 0 1

3

5+ x2

2

40 0 00 1 00 0 0

3

5 ⌫ 0

which givesmaximize ��33

subject to

2

4�11

1��332 ⇤

1��332 0 ⇤⇤ ⇤ �33

3

5 ⌫ 0

⇤ dual feasible =) �33 = 1) d

? = �1

SDP violates conditions of conic duality theorem

Both (P) and (D) are feasible, but not strictly

6= LP where strong duality always holds

Nuclear norm

TheoremOperator norm and nuclear norm are dual

Proof: want to show kXk⇤ = maxkY k1

hY,Xi

(a) Take SVD of X = U⌃V T and set Y = UV

T . Then

Tr(Y T

X) = Tr(⌃) = kXk⇤ =) maxkY k1

hY,Xi � kXk⇤

(b) In the other direction

(D)maximize hY,Xi

subject to

I Y

Y

T

I

�⌫ 0

duality$ (P)minimize 1

2Tr(⇤1) +12Tr(⇤2)

subject to

⇤1 X

X

T ⇤2

�⌫ 0

(D) has variable Y and (P) has variables ⇤1,⇤2

⇤1 = U⌃UT ⇤2 = V ⌃V T =)⇤1 X

X

T ⇤2

�=

U

V

� ⌃ 00 ⌃

� U

V

�T

val(D) val(P) Tr(⌃) = kXk⇤

Consequence

minimize kXk⇤subject to X 2 C

If C is SDP-representable, this is an SDP; e.g.A(X) = b

kA(X)� bk2 ✏

kA⇤(A(X)� b)k �

. . .

CVX knows about this