Optimization - Lecture 4, Part 1

M. Pawan Kumar

http://www.robots.ox.ac.uk/~oval/

Slides available online http://mpawankumar.info

“Recap” of Linear Programming

Polyhedron

Ax ≤ b

A : m x n matrix

b: m x 1 vector

Bounded Polyhedron = Polytope

Ax ≤ b

A : m x n matrix

b: m x 1 vector

Vertex

z is a vertex of P = {x, Ax ≤ b}

z is not a convex combination of two points in P

There does not exist x, y P and 0 < λ < 1∈

x ≠ z and y ≠ z

such that z = λ x + (1-λ) y

Vertex

z is a vertex of P = {x, Ax ≤ b}

Az is a submatrix of A

Contains all rows of A such that aiTz = bi

Recall A is an m x n matrix

Vertex

z is a vertex of P

Rank of Az = n

⟺Proof?

See “hidden” slides

• Linear Programming

• Duality

• Solving the LP

Outline

Linear Program

Maximize a linear function

Over a polyhedral feasible region

s.t. A x ≤ b

maxx cTx

A: m x n matrix

b: m x 1 vector

c: n x 1 vector

Objective function

Constraints

x: n x 1 vector

Example

4x1 – x2 ≤ 8

maxx x1 + x2

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

s.t.

What is c? A? b?

Example

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 = 8

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 ≤ 8

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

Example

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x1 + x2x1 + x2 = 0

Example

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x1 + x2

x1 + x2 = 8 Optimal solution

• Linear Programming

• Duality

• Solving the LP

Outline

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

3 x

2 x 7 x

Scale the constraints, add them up

3x1 + x2 + 2x3 ≤ 90 Upper bound on solution

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

1 x

1 x

Scale the constraints, add them up

3x1 + x2 + 2x3 ≤ 36 Upper bound on solution

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

1 x

1 x

Scale the constraints, add them up

3x1 + x2 + 2x3 ≤ 36 Tightest upper bound?

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

y1, y2, y3, y4, y5, y6 ≥ 0

We should be able to add up the inequalities

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

-y1 + y4 + 2y5 + 4y6 = 3

Coefficient of x1 should be 3

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

-y2 + y4 + 2y5 + y6 = 1

Coefficient of x2 should be 1

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

-y3 + 3y4 + 5y5 + 2y6 = 2

Coefficient of x3 should be 2

Example

2x1 + 2x2 + 5x3 ≤ 24

maxx 3x1 + x2 + 2x3

4x1 + x2 + 2x3 ≤ 36

-x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0

x1 + x2 + 3x3 ≤ 30

s.t.

y4

y1 y2 y3

y5

y6

miny 30y4 + 24y5 + 36y6

Upper bound should be tightest

Dual

miny 30y4 + 24y5 + 36y6

s.t. y1, y2, y3, y4, y5, y6 ≥ 0

-y1 + y4 + 2y5 + 4y6 = 3

-y2 + y4 + 2y5 + y6 = 1

-y3 + 3y4 + 5y5 + 2y6 = 2

Original problem is called primal

Dual of dual is primal

Dual

s.t. A x ≤ b

maxx cTx

Dual

- yT(A x – b)maxx cTxminy≥0

KKT Condition? ATy = c

miny≥0 bTy

s.t. ATy = c

miny≥0 bTy

s.t. ATy = c

s.t. A x ≤ b

maxx cTxPrimal

Dual

Strong Duality

miny≥0 bTy

s.t. ATy = c

s.t. A x ≤ b

maxx cTxPrimal

Dual

p =

d =

If p ≠ ∞ or d ≠ ∞, then p = d.

Skipping the proof

Think back to the intuition of dual

Question

s.t. A1 x ≤ b1

maxx cTx

A2 x ≥ b2

A3 x = b3

Dual?

• Linear Programming

• Duality

• Solving the LP

Outline

Graphical Solution

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x1 + x2

x1 + x2 = 8

Optimal solution at a vertex

Graphical Solution

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x1

x1 = 3

Optimal solution at a vertex

Graphical Solution

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

maxx x2

x2 = 6

Optimal solution at a vertex

Graphical Solution

4x1 – x2 ≤ 8

2x1 + x2 ≤ 10

5x1 - 2x2 ≥ -2

x1 ≥ 0

x2 ≥ 0

An optimal solution always at a vertex

Proof? maxx cTx

• Dantzig (1951): Simplex Method– Search over vertices of the polyhedra– Worst-case complexity is exponential– Smoothed complexity is polynomial

• Khachiyan (1979, 1980): Ellipsoid Method– Polynomial time complexity– LP is a P optimization problem

• Karmarkar (1984): Interior-point Method– Polynomial time complexity– Competitive with Simplex Method

Solving the LP

• Plenty of standard software available

• Mosek (http://www.mosek.com)– C++ API– Matlab API– Python API– Free academic license

Solving the LP

Questions?