Lecture 9: Dantzig-Wolfe Decomposition

(3 units)

Outline

I Dantzig-Wolfe decomposition

I Column generation algorithm

I Relation to Lagrangian dual

I Branch-and-price method

I Generated assignment problem and multi-commodity flowproblem

I References

1 / 30

Dantzig-Wolfe decomposition

I Consider the following integer programming problem:

(P) min (c1)T x1 +(c2)T x2 + · · · +(ck)T xk

s.t. A1x1 +A2x2 + · · · +AKxK = bD1x1 ≤ d1

· · · ≤ ·· · · ≤ ·

DKxK ≤ dK

x1 ∈ Zn1+ , . . . , xK ∈ Znk

+

I The constraint matrix has a block angular structure. Let

X k = {xk ∈ Znk | Dkxk ≤ dk}.

2 / 30

I The sets are independent for k = 1, . . . ,K , only the jointconstraint

∑Kk=1 Akxk = b link together the different sets of

variables.

I Dualizing the joint constraint, we have the followingLagrangian dual of (P):

(D) maxu

L(u),

where

L(u) = min{K∑

k=1

((ck)T − uTAk)xk + bTu | xk ∈ X k , ∀k}

=K∑

k=1

Lk(u) + bTu.

where Lk(u) = min{((ck)T − uTAk)xk | xk ∈ X k}.

3 / 30

I We have discussed Lagrangian relaxation and dual search inthe previous lectures. Another way of exploiting theblock-angular structure is Dantzig-Wolfe decomposition,which was invented by Dantzig and Wolfe in1961. Themethod is very closely connected to column generation andthey are often used interchangeably.

I The set X k in (P) can be either continuous (polyhedron) ordiscrete (integer set).

I Minkowski-Weyl’s Theorem: Given the convex setX = {x ∈ Rn | Ax ≤ b}. X can be represented by theextreme points and extreme rays of X :

X = {x =∑

i

λixi +

∑

j

µiyj |

∑

i

λi = 1, λi ≥ 0, µj ≥ 0}.

4 / 30

Changing representation

I When X k is a bounded polyhedron, we can express X k as

X k = {xk =

Tk∑

t=1

λktxkt |

Tk∑

t=1

λkt = 1, λkt ≥ 0},

where xkt , t = 1, . . . ,Tk , are extreme points of X k .

I When X k is a finite integer set, we can express X k as

X k = {xk =

Tk∑

t=1

λktxkt |

Tk∑

t=1

λkt = 1, λkt ∈ {0, 1}},

where xkt , t = 1, . . . ,Tk , list all the points of X k .

5 / 30

IP Master Problem

I Now, we substitute the expression of xk in (P), leading to thefollowing IP Master Problem:

(IPM) minK∑

k=1

Tk∑

t=1

((ck)T xkt)λkt

s.t.K∑

k=1

Tk∑

t=1

(Akxkt)λkt = b

Tk∑

t=1

λkt = 1, k = 1, . . . ,K ,

λkt ∈ {0, 1}, ∀k, t.

I How to solve this equivalent problem? Note that the numberof variables or columns in the constraint matrix is

∑Kk=1 Tk ,

which is usually exponentially large.

6 / 30

LP master problem

I We consider the LP relaxation of (IPM):

(LPM) minK∑

k=1

Tk∑

t=1

((ck)T xkt)λkt

s.t.K∑

k=1

Tk∑

t=1

(Akxkt)λkt = b

Tk∑

t=1

λkt = 1, k = 1, . . . ,K ,

λkt ≥ 0, ∀k, t.

I How can we find all the points (or extreme points in thecontinuous case) of X k? ⇒ We can use the idea of columngeneration.

7 / 30

Dual of the LP master problem

I The dual problem of (LPM) is

(DLPM) maxm∑

i=1

biπi +K∑

k=1

µk

s.t. πAkxkt + µk ≤ (ck)T xkt ,

t = 1, . . . ,Tk , k = 1, . . . ,K ,

where (π, µk) ∈ Rm × RK .

I Notice that a column in (LPM) corresponds a row in (DLPM).

8 / 30

Column generation algorithm

I Initialization. Suppose that a subset of columns P = ∪Kk=1Pk

(at least one for each k) is available. Consider the RestrictedLP Master Problem:

(RLPM) minK∑

k=1

∑

t∈Pk

((ck)T xkt)λkt

s.t.K∑

k=1

∑

t∈Pk

(Akxkt)λkt = b

∑

t∈Pk

λkt = 1, k = 1, . . . ,K ,

λkt ≥ 0, t ∈ Pk , k = 1, . . . ,K .

I Let λ∗ and (π, µ) ∈ Rm × RK be the optimal primal and dualsolutions to (RLPM), respectively.

9 / 30

I Primal feasibility. Any feasible solution of (LPM) can beexpanded to a feasible solution of (RLPM) (setting λkt = 0for those columns not in Pk . So

v(RLPM) =m∑

i=1

πbi +K∑

k=1

µk ≥ v(LPM).

I Optimality check for (LPM). We need to check whether (π, µ)is dual feasible for (LPM). For any x ∈ X k , the correspondingcolumn is

(ck)T xAkxek

.

The reduced cost for this column is (ck)T x − πTAkx − µk .Solve the following subproblem:

(SP) ζk = min{((ck)T − πAk)x − µk | x ∈ X k}.

10 / 30

I Stopping criterion. If ζk ≥ 0, then the solution (π, µ) is dualfeasible to (LPM) and so

K∑

k=1

∑

t∈Pk

((ck)T xkt)λ∗kt =m∑

i=1

πbi +K∑

k=1

µk ≤ v(LPM).

So the expanded solution of λ∗ is optimal to (LPM).

I Generating a new column. If ζk < 0 for some k, the columncorresponding to the optimal solution x̃ of the subproblem(SP) has negative reduced cost. Introducing the column

(ck)T x̃Ak x̃ek

into (RLPM) will lead to a new (RLPM), which can be easilyre-optimized using Primal Simplex Method.

11 / 30

I Dual (lower) bound. From the subproblem, we have

ζk ≤ ((ck)T − πAk)x − µk , ∀x ∈ X k ,

which implies

πAkx + (µk + ζk) ≤ (ck)T x , ∀x ∈ X k .

Therefore, setting ζ = (ζ1, . . . , ζK ), we have that (π, µ + ζ) isdual feasible to (LPM). We thus have

πTb +K∑

k=1

(µk + ζk) ≤ v(LPM).

12 / 30

Relation to Lagrangian dual

I Theorem:

v(LPM) = min{K∑

k=1

(ck)T xk |K∑

k=1

Akxk = b, xk ∈ conv(X k)}.

I Proof: Note that (LPM) is obtained by substituting

xk =

Tk∑

t=1

λktxkt ,

Tk∑

t=1

λkt = 1, λkt ≥ 0,

where xkt , t = 1, . . . ,Tk , are all the integer points in X k .This is equivalent to xk ∈ conv(X k).

I Theorem: v(LPM) = v(D).

I Therefore, the LP relaxation of the IP master problem isactually equivalent to the Lagrangian dual.

13 / 30

I Consider the general IP problem:

min{cT x | Ax ≤ b, Dx ≤ d , x ∈ Zn}.

I Strength of the D-W decomposition and Lagrangian dual:

2CONV( )

2

1

x

xx

Figure: X1 = {x ∈ Zn | Ax ≤ b}, X2 = {x ∈ Zn | Dx ≤ d}.

14 / 30

Branch-and-Price Method

I Consider the general IP problem:

(IP) min cT x

s.t. Ax ≤ b (“complicating′′constraints)Dx ≤ d , (“easy′′constraints)x ∈ Zn.

I We assume that optimization over the setX2 = {x ∈ Zn | Dx ≤ d} is “easy”. This will be used insolving the column generation subproblem.

I Let x1, . . . , xT be all the integer points of X2. Then

X2 = {x =T∑

i=1

λixi |

T∑

i=1

λi = 1, λi ∈ {0, 1}}.

15 / 30

I Use this representation of X2, we can rewrite (IP) as

(IPM) min cT (T∑

i=1

λixi )

s.t. A(T∑

i=1

λixi ) ≤ b,

T∑

i=1

λi = 1,

λi ∈ {0, 1}, i = 1, . . . ,T .

I This is a reformulation of (IP).

16 / 30

I The LP relaxation of (IPM) is at least as strong as the directLP relaxation of (IP). (Because the LP relaxation of (IPM) isequivalent to the Lagrangian relaxation of (IP) by dualizingthe “complicating” constraint Ax ≤ b.

I We solve the LP relaxation of (IPM) using column generation.

I Adding one column to the LP master problem corresponds toadding one cutting plane to the dual problem of (LPM).

I The column generation subproblem is an optimization problemover X2, which can be solved efficiently in many applications.

I We can embed this bounding scheme into a branch and priceframework. “Price” here is referred to find the column withnegative reduced cost by solving the subproblem.

I How to branch?

17 / 30

Branching with Dantzig-Wolfe Decomposition

I Unfortunately, branching on the variables (λi ) of thereformulation doesn’t work well because it’s generally difficultto keep a variable from being generated again after it’s beenfixed to zero.

I Branching must be done in a way that does not destroy thestructure of the column generation master problem andsubproblem.

I We can do this by branching on the original variables, i.e.,before the reformulation.

I In a 0-1 problem, branching on the jth original variable isequivalent to fixing the value of some element of the columnsto be generated. This can usually be incorporated into thecolumn generation subproblem.

18 / 30

Generalized Assignment ProblemI The problem is to assign m tasks to n machines subject to

capacity constraints.I An IP formulation of tho problem is

(GAP) maxm∑

i=1

n∑

j=1

pijzij

s.t.n∑

j=1

zij = 1, i = 1, . . . ,m,

m∑

i=1

wijzij ≤ dj , j = 1, . . . , n,

zij ∈ {0, 1}, i = 1, . . . ,m, j = 1, . . . , n.

I Let’s the columns in (GAP) represent feasible assignments oftasks to the machine.

X2 = {z ∈ {0, 1}n×m |m∑

i=1

wijzij ≤ dj , j = 1, . . . , n}.19 / 30

I The Dantzig-Wolfe decomposition master problem is:

maxn∑

j=1

m∑

i=1

pij

Tj∑

k=1

λjjy

jik

s.t.n∑

j=1

Tj∑

k=1

λjjy

jik = 1, i = 1, . . . ,m,

Tj∑

i=1

λjk = 1, j = 1, . . . , n,

λjk ∈ {0, 1}, j = 1, . . . , n, k = 1, . . . ,Tj .

This is a set-covering problem.

I The column generation subproblems are n knapsack problemwith a single constraint

∑mi=1 wijzij ≤ dj .

20 / 30

Multi-commodity flow problem

I Let D = (V ,A) be a directed graph with nonnegativecapacities ua for all a ∈ A.

I Given k commodities, i.e., node pairs (s1, t1), . . . , (sk , tk). Foreach i = 1, . . . , k, we want to find an si − ti -flow, (x i

a), suchthat the sum of these flows on each arc does not exceed agiven capacity. This flow is called a feasible multicommodityflow and can be expressed by the following constraints:

∑

a∈δ+(v)

x ia −

∑

a∈δ−(v)

x ia = 0, ∀v ∈ V \ {si , ti}, i = 1, . . . , k,

k∑

i=1

x ia ≤ ua, ∀a ∈ A,

where δ+(v) = {(v ,w) | (v ,w) ∈ A} andδ−(v) = {(u, v) | (u, v) ∈ A}.

21 / 30

Edge formulation

I The multicommodity flow problem (MCF) is to maximize thesum of the flow values of each commodes:

maxk∑

i=1

∑

a∈δ+(si )

x ia −

∑

a∈δ−(si )

x ia

s.t.∑

a∈δ+(v)

x ia −

∑

a∈δ−(v)

x ia = 0, ∀v ∈ V \ {si , ti}, i = 1, . . . , k,

k∑

i=1

x ia ≤ ua, ∀a ∈ A.

I This formulation is called edge formulation of MCF. It is alinear program and can be solved polynomially.

I However, if we require the flow to be integer, i.e., x ia ∈ Z+,

then the MCF is NP-hard.

22 / 30

Path formulation

I We now give a path formulation for MCF, which can beviewed as the reformulation of the edge formulation usingDantzig-Wolfe decomposition.

I Let Pst be the set of all s − t paths. Let

P = Ps1,t1 ∪ · · · ∪ Psk ,tk .

Let Pa denote the set of paths that use arc a, i.e.,Pa = {p ∈ P | a ∈ p}.

I For each p ∈ Psi ,ti , we define a flow variable fp. Themulti-commodity flow problem can be expressed as

(MCF ) max∑

p∈Pfp

s.t.∑

p∈Pa

fp ≤ ua, ∀a ∈ A,

fa ≥ 0, ∀a ∈ A.

23 / 30

Column generation for MCF

I Let P ′ ⊂ P be a subset of all paths. Consider the restrictedLP master problem:

(MCF ′) max∑

p∈P ′fp

s.t.∑

p∈P ′afp ≤ ua, ∀a ∈ A,

fa ≥ 0, ∀a ∈ A.

I Note that a feasible solution (f ′p)p∈P ′ for (MCF ′) can beexpanded to a feasible solution (fp)p∈P to (MCF ) (settingfp = 0 for p ∈ P \ P ′. Thus,

v(MCF ′) ≤ v(MCF ).

24 / 30

I The dual of (MCF ′) is

(DMCF ′) min∑

a∈A

uaµa

s.t.∑a∈p

µa ≥ 1, ∀p ∈ P ′,

µa ≥ 0, ∀a ∈ A.

I Reduced cost≤ 0 ⇔ Dual feasible. So, if

∑a∈p

µa ≥ 1, ∀p ∈ P,

then the current solution is optimal to (MCF ). Otherwise, weneed to find a new column to improve (MCF ′).

25 / 30

I Checking whether the above inequality is satisfied for all pathsis called pricing subproblem. This can be done efficiently bycomputing the shortest path w.r.t. the weight vector µa

(a ∈ A) for each commodity (si , ti ) (i = 1, . . . , k). If theshortest paths are all at least 1, then we mush have

∑a∈p

µa ≥ 1, ∀p ∈ P,

Otherwise, we find a new path p that has the positive reducedcost and we can add this path (column) to (MCF ′).

I Finding a shortest (si , ti )-path in graph (V ,A) withnonnegative weights µa ≥ 0 (a ∈ A) is polynomial (e.g., byDijkstra’s algorithm).

26 / 30

I Column generation algorithm for MCF

Repeat1. Solve (MCF ′) for P ′2. Let µ be the optimal multiplier vector of

(MCF ′)3. for i = 1, . . . , k, do

I Find a shortest path si − ti -path w.r.t. weightvector (µa)

I If weight of the shortest path p is less than 1, addp to (MCF ′)

4. end for

Until no path has been added

I How to get an integer solution?

27 / 30

Dantzig-Wolfe decomposition for MCFI We now show that the path formulation can be obtained from

Dantzig-Wolfe decomposition. We first write the edgeformulation in a simple form:

max cT x

s.t. Nx = 0, (flow conservation)

Ux ≤ u, (capacity constraints)x ≥ 0.

I LetP = {x ≥ 0 | Nx = 0}.

This polyhedron is a cone and has a single vertex 0 and afinite number of rays r1, . . . , r s . By the Minkowski-Weyltheorem, we have

P = cone{r1, . . . , r s} = {s∑

i=1

λi ri | λi ≥ 0}.

28 / 30

I Using the above expression of P, we can rewrite the edgeformulation as

maxs∑

i=1

λi (cT r i )

s.t.s∑

i=1

λi (Nr i ) ≤ u,

λi ≥ 0, i = 1, . . . , s.

I Analyzing the structure of c and U, the above problem can bereduced to

maxs∑

i=1

yi

s.t.∑

i :a∈pi

yi ≤ ua, a ∈ A

yi ≥ 0, i = 1, . . . , s,

which is equivalent to the path formulation (refer toNumhauser and Wolsey (1988) for more details)

29 / 30

References

1. F. Vanderbeck, M. W. P. Savelsbergh, A generic view ofDantzig-Wolfe decomposition in mixed-integer programming,Vol. 34(3), 296-306, Operations Research Letters, 2006.

2. F. Vanderbeck, L. A. Wolsey, Reformulation anddecomposition of integer programs, in 50 Years of IntegerProgramming 1958-2008, Part 2, pp. 431-502, 2010.

3. Cynthia Barnhart, Ellis L. Johnson, George L. Nemhauser,Martin W. P. Savelsbergh and Pamela H. Vance,Branch-and-Price: Column Generation for Solving HugeInteger Programs, Operations Research, Vol. 46(3), 316-329,1998.

4. Marc Pfetsch, Multi-commodity flow and column generation,Lecture Notes, ZIB, 2006.

30 / 30