· Introduction Gauge problems Optimality conditions and examples A few examples A few ideas on...
Transcript of · Introduction Gauge problems Optimality conditions and examples A few examples A few ideas on...
IntroductionGauge problems
Optimality conditions and examples
MATHEMATICAL PROGRAMMING (I)
Emilio Carrizosa
Doc-Course, March 2010
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector
U = u1, . . . , uN (records)
Each ui : associated ci > 0.
Sought: for each i , πi , with
0 ≤ πi ≤ 1 for all i = 1, 2, . . . ,N.∑Ni=1 πi = n0.
πi ∝ ci for all i = 1, 2, . . . ,N.
min∑N
i=1
(πici− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector
U = u1, . . . , uN (records)
Each ui : associated ci > 0.
Sought: for each i , πi , with
0 ≤ πi ≤ 1 for all i = 1, 2, . . . ,N.∑Ni=1 πi = n0.
πi ∝ ci for all i = 1, 2, . . . ,N.
min∑N
i=1
(πici− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector
U = u1, . . . , uN (records)
Each ui : associated ci > 0.
Sought: for each i , πi , with
0 ≤ πi ≤ 1 for all i = 1, 2, . . . ,N.∑Ni=1 πi = n0.
πi ∝ ci for all i = 1, 2, . . . ,N.
min∑N
i=1
(πici− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector
U = u1, . . . , uN (records)
Each ui : associated ci > 0.
Sought: for each i , πi , with
0 ≤ πi ≤ 1 for all i = 1, 2, . . . ,N.∑Ni=1 πi = n0.
πi ∝ ci for all i = 1, 2, . . . ,N.
min∑N
i=1
(πici− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector
U = u1, . . . , uN (records)
Each ui : associated ci > 0.
Sought: for each i , πi , with
0 ≤ πi ≤ 1 for all i = 1, 2, . . . ,N.∑Ni=1 πi = n0.
πi ∝ ci for all i = 1, 2, . . . ,N.
min∑N
i=1
(πici− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector
U = u1, . . . , uN (records)
Each ui : associated ci > 0.
Sought: for each i , πi , with
0 ≤ πi ≤ 1 for all i = 1, 2, . . . ,N.∑Ni=1 πi = n0.
πi ∝ ci for all i = 1, 2, . . . ,N.
min∑N
i=1
(πici− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector
U = u1, . . . , uN (records)
Each ui : associated ci > 0.
Sought: for each i , πi , with
0 ≤ πi ≤ 1 for all i = 1, 2, . . . ,N.∑Ni=1 πi = n0.
πi
ci: constant.
min∑N
i=1
(πici− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
The Fermat-Weber problem
Let A ⊂ Rn, finite (set of users, asking for service). For eacha ∈ A, let ωa > 0 be the demand power of a.
minx∈Rn
f (x) :=∑a∈A
ωa‖x − a‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
The Fermat-Weber problem
Let A ⊂ Rn, finite (set of users, asking for service). For eacha ∈ A, let ωa > 0 be the demand power of a.
minx∈Rn
f (x) :=∑a∈A
ωa‖x − a‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
The Fermat-Weber problem
Let A ⊂ Rn, finite (set of users, asking for service). For eacha ∈ A, let ωa > 0 be the demand power of a.
minx∈Rn
f (x) :=∑a∈A
ωa‖x − a‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Ingredients:
A, finite set of users. Each a ∈ A, with demand ωa.
Finite set F of possible locations to open facilities.
The locations of p identical facilities are sought.
Distance d(a, f ) from user at a and potential location f :known.
Each user goes to his/her nearest open facility.
Objective: minimize the total transportion cost (or,equivalently, the average from a random user to his/hernearest open facility).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Ingredients:
A, finite set of users. Each a ∈ A, with demand ωa.
Finite set F of possible locations to open facilities.
The locations of p identical facilities are sought.
Distance d(a, f ) from user at a and potential location f :known.
Each user goes to his/her nearest open facility.
Objective: minimize the total transportion cost (or,equivalently, the average from a random user to his/hernearest open facility).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Ingredients:
A, finite set of users. Each a ∈ A, with demand ωa.
Finite set F of possible locations to open facilities.
The locations of p identical facilities are sought.
Distance d(a, f ) from user at a and potential location f :known.
Each user goes to his/her nearest open facility.
Objective: minimize the total transportion cost (or,equivalently, the average from a random user to his/hernearest open facility).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Ingredients:
A, finite set of users. Each a ∈ A, with demand ωa.
Finite set F of possible locations to open facilities.
The locations of p identical facilities are sought.
Distance d(a, f ) from user at a and potential location f :known.
Each user goes to his/her nearest open facility.
Objective: minimize the total transportion cost (or,equivalently, the average from a random user to his/hernearest open facility).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Ingredients:
A, finite set of users. Each a ∈ A, with demand ωa.
Finite set F of possible locations to open facilities.
The locations of p identical facilities are sought.
Distance d(a, f ) from user at a and potential location f :known.
Each user goes to his/her nearest open facility.
Objective: minimize the total transportion cost (or,equivalently, the average from a random user to his/hernearest open facility).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Ingredients:
A, finite set of users. Each a ∈ A, with demand ωa.
Finite set F of possible locations to open facilities.
The locations of p identical facilities are sought.
Distance d(a, f ) from user at a and potential location f :known.
Each user goes to his/her nearest open facility.
Objective: minimize the total transportion cost (or,equivalently, the average from a random user to his/hernearest open facility).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Binary variables
For each f ∈ F , define y(f ) =
1, if plant at f is open0, else
For each f ∈ F , a ∈ A, define variable
x(a, f ) =
1, if a goes to f0, else
Cost:
Cost for a, if (s)he goes to f : ωad(a, f )
Cost for a : ωa
∑f∈F d(a, f )x(a, f )
Total cost:∑
a∈A
(ωa
∑f∈F d(a, f )x(a, f )
)Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Binary variables
For each f ∈ F , define y(f ) =
1, if plant at f is open0, else
For each f ∈ F , a ∈ A, define variable
x(a, f ) =
1, if a goes to f0, else
Cost:
Cost for a, if (s)he goes to f : ωad(a, f )
Cost for a : ωa
∑f∈F d(a, f )x(a, f )
Total cost:∑
a∈A
(ωa
∑f∈F d(a, f )x(a, f )
)Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Binary variables
For each f ∈ F , define y(f ) =
1, if plant at f is open0, else
For each f ∈ F , a ∈ A, define variable
x(a, f ) =
1, if a goes to f0, else
Cost:
Cost for a, if (s)he goes to f : ωad(a, f )
Cost for a : ωa
∑f∈F d(a, f )x(a, f )
Total cost:∑
a∈A
(ωa
∑f∈F d(a, f )x(a, f )
)Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Binary variables
For each f ∈ F , define y(f ) =
1, if plant at f is open0, else
For each f ∈ F , a ∈ A, define variable
x(a, f ) =
1, if a goes to f0, else
Cost:
Cost for a, if (s)he goes to f : ωad(a, f )
Cost for a : ωa
∑f∈F d(a, f )x(a, f )
Total cost:∑
a∈A
(ωa
∑f∈F d(a, f )x(a, f )
)Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Binary variables
For each f ∈ F , define y(f ) =
1, if plant at f is open0, else
For each f ∈ F , a ∈ A, define variable
x(a, f ) =
1, if a goes to f0, else
Cost:
Cost for a, if (s)he goes to f : ωad(a, f )
Cost for a : ωa
∑f∈F d(a, f )x(a, f )
Total cost:∑
a∈A
(ωa
∑f∈F d(a, f )x(a, f )
)Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Binary variables
For each f ∈ F , define y(f ) =
1, if plant at f is open0, else
For each f ∈ F , a ∈ A, define variable
x(a, f ) =
1, if a goes to f0, else
Exactly p plants are open: ∑f ∈F
y(f ) = p
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Binary variables
For each f ∈ F , define y(f ) =
1, if plant at f is open0, else
For each f ∈ F , a ∈ A, define variable
x(a, f ) =
1, if a goes to f0, else
Each user goes to just one facility:∑f ∈F
x(a, f ) = 1 ∀a ∈ A
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
Binary variables
For each f ∈ F , define y(f ) =
1, if plant at f is open0, else
For each f ∈ F , a ∈ A, define variable
x(a, f ) =
1, if a goes to f0, else
Nobody can go to closed facilities:
x(a, f ) ≤ y(f ) for all a ∈ A, ∀f ∈ F
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Discrete p-median problem
min∑
a∈A
(ωa∑
f ∈F d(a, f )x(a, f ))
s.t.∑
f ∈F y(f ) = p∑f ∈F x(a, f ) = 1 ∀a ∈ A
x(a, f ) ≤ y(f ) ∀a ∈ A, f ∈ Fx(a, f ) ∈ 0, 1 ∀a ∈ A, f ∈ Fy(f ) ∈ 0, 1 ∀f ∈ F
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
(`∞) Distance to co-hyperplanarity
A ⊂ Rn, finite nonempty.
Aim: perturb as few as possible the points in A so that theperturbed set is co-hyperplanar
For hyperplane H(u, β) := u>x + β = 0, the perturbation is
δ(u, β) = max |u>a+β|‖u‖
H(u, β) is sought minimizing the maximum perturbationδ(u, β)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
(`∞) Distance to co-hyperplanarity
A ⊂ Rn, finite nonempty.
Aim: perturb as few as possible the points in A so that theperturbed set is co-hyperplanar
For hyperplane H(u, β) := u>x + β = 0, the perturbation is
δ(u, β) = max |u>a+β|‖u‖
H(u, β) is sought minimizing the maximum perturbationδ(u, β)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
(`∞) Distance to co-hyperplanarity
min(u,b)∈Rn×R, u 6=0
maxa∈A
|u>a + b|‖u‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
(`∞) Distance to co-hyperplanarity
min(u,b)∈Rn×R, u 6=0
maxa∈A
|u>a + b|‖u‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Population: O.Classes: C = −1, 1.
u ∈ O →
xu ∈ Rn ( features )yu ∈ C ( class labels )
I ⊂ O : training sample: individuals u with (xu, yu) known.
I+ := u ∈ I : yu = +1, I− := u ∈ I : yu = −1. Bothassumed to be non-empty
score function f (x) := ω>x + β sought to assign labels toobjects u for which just xu (and not yu!) is known:
x ∈ Rn → f (x) :=
1, if ω>x + β > 0−1, if ω>x + β < 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Population: O.Classes: C = −1, 1.
u ∈ O →
xu ∈ Rn ( features )yu ∈ C ( class labels )
I ⊂ O : training sample: individuals u with (xu, yu) known.
I+ := u ∈ I : yu = +1, I− := u ∈ I : yu = −1. Bothassumed to be non-empty
score function f (x) := ω>x + β sought to assign labels toobjects u for which just xu (and not yu!) is known:
x ∈ Rn → f (x) :=
1, if ω>x + β > 0−1, if ω>x + β < 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Population: O.Classes: C = −1, 1.
u ∈ O →
xu ∈ Rn ( features )yu ∈ C ( class labels )
I ⊂ O : training sample: individuals u with (xu, yu) known.
I+ := u ∈ I : yu = +1, I− := u ∈ I : yu = −1. Bothassumed to be non-empty
score function f (x) := ω>x + β sought to assign labels toobjects u for which just xu (and not yu!) is known:
x ∈ Rn → f (x) :=
1, if ω>x + β > 0−1, if ω>x + β < 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Population: O.Classes: C = −1, 1.
u ∈ O →
xu ∈ Rn ( features )yu ∈ C ( class labels )
I ⊂ O : training sample: individuals u with (xu, yu) known.
I+ := u ∈ I : yu = +1, I− := u ∈ I : yu = −1. Bothassumed to be non-empty
score function f (x) := ω>x + β sought to assign labels toobjects u for which just xu (and not yu!) is known:
x ∈ Rn → f (x) :=
1, if ω>x + β > 0−1, if ω>x + β < 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Population: O.Classes: C = −1, 1.
u ∈ O →
xu ∈ Rn ( features )yu ∈ C ( class labels )
I ⊂ O : training sample: individuals u with (xu, yu) known.
I+ := u ∈ I : yu = +1, I− := u ∈ I : yu = −1. Bothassumed to be non-empty
score function f (x) := ω>x + β sought to assign labels toobjects u for which just xu (and not yu!) is known:
x ∈ Rn → f (x) :=
1, if ω>x + β > 0−1, if ω>x + β < 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Population: O.Classes: C = −1, 1.
u ∈ O →
xu ∈ Rn ( features )yu ∈ C ( class labels )
I ⊂ O : training sample: individuals u with (xu, yu) known.
I+ := u ∈ I : yu = +1, I− := u ∈ I : yu = −1. Bothassumed to be non-empty
score function f (x) := ω>x + β sought to assign labels toobjects u for which just xu (and not yu!) is known:
x ∈ Rn → f (x) :=
1, if ω>x + β > 0−1, if ω>x + β < 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
Support Vector Machines
How to chose then one linear classifier?
?
Support Vector Machines
How to chose then one linear classifier?
?
Support Vector Machines
How to chose then one linear classifier?
?
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Halfspace of misclassification
for i ∈ I+ : H(ω, β)− := x ∈ Rn : ω>x + β ≤ 0for i ∈ I− : H(ω, β)+ := x ∈ Rn : ω>x + β ≥ 0
δi (ω, β) := distance from xi to halfspace of misclassification
margin: mini∈I δi (ω, β)
Maximize the margin
maxω∈Rn\0
(mini∈I
δi (ω, β)
):= max
yi (ω
>xi + β)
‖ω‖, 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Halfspace of misclassification
for i ∈ I+ : H(ω, β)− := x ∈ Rn : ω>x + β ≤ 0for i ∈ I− : H(ω, β)+ := x ∈ Rn : ω>x + β ≥ 0
δi (ω, β) := distance from xi to halfspace of misclassification
margin: mini∈I δi (ω, β)
Maximize the margin
maxω∈Rn\0
(mini∈I
δi (ω, β)
):= max
yi (ω
>xi + β)
‖ω‖, 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
Halfspace of misclassification
for i ∈ I+ : H(ω, β)− := x ∈ Rn : ω>x + β ≤ 0for i ∈ I− : H(ω, β)+ := x ∈ Rn : ω>x + β ≥ 0
δi (ω, β) := distance from xi to halfspace of misclassification
margin: mini∈I δi (ω, β)
Maximize the margin
maxω∈Rn\0
(mini∈I
δi (ω, β)
):= max
yi (ω
>xi + β)
‖ω‖, 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
maxω∈Rn\0
max
yi (ω
>xi + β)
‖ω‖, 0
For (xi : i ∈ I+, xi : i ∈ I−) : separable, equivalent to
maxω∈Rn\0
mini∈I
yi (ω>xi + β)
‖ω‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
maxω∈Rn\0
max
yi (ω
>xi + β)
‖ω‖, 0
For (xi : i ∈ I+, xi : i ∈ I−) : separable, equivalent to
maxω∈Rn\0
mini∈I
yi (ω>xi + β)
‖ω‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
maxω∈Rn\0
max
yi (ω
>xi + β)
‖ω‖, 0
For (xi : i ∈ I+, xi : i ∈ I−) : separable, equivalent to
maxω∈Rn\0
mini∈I
yi (ω>xi + β)
‖ω‖
min ‖ω‖s.t. yi (ω
>xi + β) ≥ 1 ∀i ∈ Iω ∈ Rn, β ∈ R
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
maxω∈Rn\0
max
yi (ω
>xi + β)
‖ω‖, 0
For (xi : i ∈ I+, xi : i ∈ I−) : separable, equivalent to
maxω∈Rn\0
mini∈I
yi (ω>xi + β)
‖ω‖
Hard-margin SVM
min ω>ωs.t. yi (ω
>xi + β) ≥ 1 ∀i ∈ Iω ∈ Rn, β ∈ R
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Support Vector Machines
maxω∈Rn\0
max
yi (ω
>xi + β)
‖ω‖, 0
For (xi : i ∈ I+, xi : i ∈ I−) : separable, equivalent to
maxω∈Rn\0
mini∈I
yi (ω>xi + β)
‖ω‖
Soft-margin SVM
min ω>ω + C∑
i∈I ηi
s.t. yi (ω>xi + β) + ηi ≥ 1 ∀i ∈ I
ω ∈ Rn, β ∈ Rηi ≥ 0 ∀i ∈ I
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector(revisited)
πi
ci: constant.
min∑N
i=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0
k ≥ 0
πi
ci: constant.
min maxNi=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0
k ≥ 0
min
(∑Ni=1
(πi
ci− k)2
,maxNi=1
(πi
ci− k)2)
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector(revisited)
πi
ci: constant.
min∑N
i=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0
k ≥ 0
πi
ci: constant.
min maxNi=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0
k ≥ 0
min
(∑Ni=1
(πi
ci− k)2
,maxNi=1
(πi
ci− k)2)
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Probabilities approximately proportional to a vector(revisited)
πi
ci: constant.
min∑N
i=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0
k ≥ 0
πi
ci: constant.
min maxNi=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i∑Ni=1 πi = n0
k ≥ 0
min
(∑Ni=1
(πi
ci− k)2
,maxNi=1
(πi
ci− k)2)
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Summary of examples
1 Probabilities approximately proportional to a vector
2 Fermat-Weber problem
3 Discrete p-median
4 (`∞) distance to co-hyperplanarity
5 Support Vector Machines
6 Probabilities approximately proportional to a vector (revisited)
minx∈X
f (x)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Summary of examples
1 Probabilities approximately proportional to a vector
2 Fermat-Weber problem
3 Discrete p-median
4 (`∞) distance to co-hyperplanarity
5 Support Vector Machines
6 Probabilities approximately proportional to a vector (revisited)
minx∈X
f (x)
Optimal solution (f : X −→ R)
x∗ : f (x∗) ≤ f (x) ∀x ∈ X
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Summary of examples
1 Probabilities approximately proportional to a vector
2 Fermat-Weber problem
3 Discrete p-median
4 (`∞) distance to co-hyperplanarity
5 Support Vector Machines
6 Probabilities approximately proportional to a vector (revisited)
minx∈X
f (x)
ε-Optimal solution (f : X −→ R)
x∗ : f (x∗) ≤ f (x) + ε∀x ∈ X
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Summary of examples
1 Probabilities approximately proportional to a vector
2 Fermat-Weber problem
3 Discrete p-median
4 (`∞) distance to co-hyperplanarity
5 Support Vector Machines
6 Probabilities approximately proportional to a vector (revisited)
minx∈X
f (x)
Ideal solution (f : X −→ Rn)
x∗ : fi (x∗) ≤ fi (x) ∀i , ∀x ∈ X
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Summary of examples
1 Probabilities approximately proportional to a vector
2 Fermat-Weber problem
3 Discrete p-median
4 (`∞) distance to co-hyperplanarity
5 Support Vector Machines
6 Probabilities approximately proportional to a vector (revisited)
minx∈X
f (x)
Weakly efficient solution (f : X −→ Rn)
x∗ : 6 ∃x ∈ X : fi (x) < fi (x∗)∀i
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Summary of examples
1 Probabilities approximately proportional to a vector
2 Fermat-Weber problem
3 Discrete p-median
4 (`∞) distance to co-hyperplanarity
5 Support Vector Machines
6 Probabilities approximately proportional to a vector (revisited)
minx∈X
f (x)
Efficient solution (f : X −→ Rn)
x∗ : 6 ∃x ∈ X :fi (x) ≤ fi (x∗)∀if (x∗) 6= f (x)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
The practitioner’s perspective
http://www-neos.mcs.anl.gov
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Convexity
Convex sets
S ⊂ Rn : convex if ∀x , y ∈ S , the segment with endpoints x , y isincluded in S :
x , y ∈ S ⇒ (1− λ) x + λy ∈ S ∀λ ∈ [0, 1].
Operations preserving convexity
Given C ⊂ Rn : convex, int(C ), cl(C ) andA · C + µ := Ac + µ : c ∈ C are convex sets.
If Cii∈I are convex in Rn, the sets ∩i∈I Ci ,∏
i∈I Ci ,∑
i∈I Ci
are also convex.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Convexity
Convex sets
S ⊂ Rn : convex if ∀x , y ∈ S , the segment with endpoints x , y isincluded in S :
x , y ∈ S ⇒ (1− λ) x + λy ∈ S ∀λ ∈ [0, 1].
Operations preserving convexity
Given C ⊂ Rn : convex, int(C ), cl(C ) andA · C + µ := Ac + µ : c ∈ C are convex sets.
If Cii∈I are convex in Rn, the sets ∩i∈I Ci ,∏
i∈I Ci ,∑
i∈I Ci
are also convex.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Convex envelop
Given X ⊂ Rn, its convex envelop conv(X ) is the smallest convexset containing X : conv(X ) =
⋂C⊃X ,C :convex C
Example
For E ⊂ Rn : finite not empty,conv(E ) =
∑e∈E λee : λe ≥ 0 ∀e ∈ E ,
∑e∈E λe = 1
Caratheodory
Let X ⊂ Rn. For any x ∈ conv(X ), ∃X ∗ ⊂ X , |X ∗| ≤ n + 1 s.t.x ∈ conv(X ∗).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Convex envelop
Given X ⊂ Rn, its convex envelop conv(X ) is the smallest convexset containing X : conv(X ) =
⋂C⊃X ,C :convex C
Example
For E ⊂ Rn : finite not empty,conv(E ) =
∑e∈E λee : λe ≥ 0 ∀e ∈ E ,
∑e∈E λe = 1
Caratheodory
Let X ⊂ Rn. For any x ∈ conv(X ), ∃X ∗ ⊂ X , |X ∗| ≤ n + 1 s.t.x ∈ conv(X ∗).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Convex envelop
Given X ⊂ Rn, its convex envelop conv(X ) is the smallest convexset containing X : conv(X ) =
⋂C⊃X ,C :convex C
Example
For E ⊂ Rn : finite not empty,conv(E ) =
∑e∈E λee : λe ≥ 0 ∀e ∈ E ,
∑e∈E λe = 1
Caratheodory
Let X ⊂ Rn. For any x ∈ conv(X ), ∃X ∗ ⊂ X , |X ∗| ≤ n + 1 s.t.x ∈ conv(X ∗).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Cones
K ⊂ Rn, K 6= ∅ : cone if K = R+ · K :
x ∈ K ⇒ λx ∈ K ∀λ ∈ R+.
Any cone contains the origin.
Convex cones
A cone K ⊂ Rn is convex iff K + K = K .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Cones
K ⊂ Rn, K 6= ∅ : cone if K = R+ · K :
x ∈ K ⇒ λx ∈ K ∀λ ∈ R+.
Any cone contains the origin.
Convex cones
A cone K ⊂ Rn is convex iff K + K = K .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Cones
K ⊂ Rn, K 6= ∅ : cone if K = R+ · K :
x ∈ K ⇒ λx ∈ K ∀λ ∈ R+.
Any cone contains the origin.
Convex cones
A cone K ⊂ Rn is convex iff K + K = K .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Normal cone
Given X ⊂ Rn, x ∈ X , the normal cone of X at x is the closedconvex cone NX (x),
NX (x) =
p ∈ Rn : p>(y − x) ≤ 0 ∀y ∈ X.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Normal cone
Given X ⊂ Rn, x ∈ X , the normal cone of X at x is the closedconvex cone NX (x),
NX (x) =
p ∈ Rn : p>(y − x) ≤ 0 ∀y ∈ X.
Example
For X = [a, b] ⊂ R, a < b,
NX (x) =
R−, if x = a0, if a < x < bR+, if x = b.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Normal cone
Given X ⊂ Rn, x ∈ X , the normal cone of X at x is the closedconvex cone NX (x),
NX (x) =
p ∈ Rn : p>(y − x) ≤ 0 ∀y ∈ X.
Example
For X = x ∈ Rn : Ax ≤ b,
NX (x) =
A>u : u>(Ax − b) = 0, u ≥ 0.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Separation
Given C ⊂ Rn, C 6= ∅, convex, closed, x ∈ Rn \ C , ∃u ∈ Rn s.t.
u>y < u>x ∀y ∈ C .
Minimax theorem
For closed convex sets X ,Y ⊂ Rn, with Y : compact,
minx∈X
maxy∈Y
x>y = maxy∈Y
minx∈X
x>y
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Separation
Given C ⊂ Rn, C 6= ∅, convex, closed, x ∈ Rn \ C , ∃u ∈ Rn s.t.
u>y < u>x ∀y ∈ C .
Minimax theorem
For closed convex sets X ,Y ⊂ Rn, with Y : compact,
minx∈X
maxy∈Y
x>y = maxy∈Y
minx∈X
x>y
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polar
Given X ⊂ Rn, its polar X is
X =
p : p>x ≤ 1 ∀x ∈ X.
Examples
For x0 ∈ Rn, x0 6= 0, x0 is a closed halfspace.
For X = x : ‖x‖2 ≤ c, X = x : ‖x‖2 ≤ 1c .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polar
Given X ⊂ Rn, its polar X is
X =
p : p>x ≤ 1 ∀x ∈ X.
Examples
For x0 ∈ Rn, x0 6= 0, x0 is a closed halfspace.
For X = x : ‖x‖2 ≤ c, X = x : ‖x‖2 ≤ 1c .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polar
Given X ⊂ Rn, its polar X is
X =
p : p>x ≤ 1 ∀x ∈ X.
Examples
For x0 ∈ Rn, x0 6= 0, x0 is a closed halfspace.
For X = x : ‖x‖2 ≤ c, X = x : ‖x‖2 ≤ 1c .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polar
Properties
Given X ⊂ Rn, X is closed and convex with 0 ∈ X .
If X ⊂ Y ⊂ Rn, then X ⊃ Y .
Let X : convex, closed, with 0 ∈ X . Then
(X ) = X .0 ∈ int(X ) iff X : bounded.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Convex functions
Epigraph
The epigraph epi(f ) of function f : S −→ R is defined as
epi(f ) = (x , t) ∈ Rn × R : x ∈ S , t ≥ f (x) .
Let S ⊂ Rn : convex. f : S −→ R is convex if epi(f ) is convex.Equivalently, f is convex iff
f ((1− λ)x + λy) ≤ (1− λ)f (x) + λf (y).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Convex functions
Epigraph
The epigraph epi(f ) of function f : S −→ R is defined as
epi(f ) = (x , t) ∈ Rn × R : x ∈ S , t ≥ f (x) .
Let S ⊂ Rn : convex. f : S −→ R is convex if epi(f ) is convex.Equivalently, f is convex iff
f ((1− λ)x + λy) ≤ (1− λ)f (x) + λf (y).
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Algebra of convex functions
Any C2 in Rn with PSD Hessian in Rn is convex.
Given f : convex on the convex set S ⊂ Rn,
λf + µ, with λ ≥ 0, µ ∈ R : convexf (Ax + b) : convex, with A ∈ Rm×n, b ∈ Rm
Given f1, f2, convex on the convex S ⊂ Rn, the followingfunctions are also convex:
f1 + f2.maxf1, f2.
Let f : S ⊂ Rn −→ R, convex on the convex S , y g : R −→ R :non-decreasing. Then g f : convex on S .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Algebra of convex functions
Any C2 in Rn with PSD Hessian in Rn is convex.
Given f : convex on the convex set S ⊂ Rn,
λf + µ, with λ ≥ 0, µ ∈ R : convexf (Ax + b) : convex, with A ∈ Rm×n, b ∈ Rm
Given f1, f2, convex on the convex S ⊂ Rn, the followingfunctions are also convex:
f1 + f2.maxf1, f2.
Let f : S ⊂ Rn −→ R, convex on the convex S , y g : R −→ R :non-decreasing. Then g f : convex on S .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Gauges
Let B ⊂ Rn, convex, compact, with 0 ∈ int(B). Define the gaugeγB with unit ball B as
γB(x) = min t ≥ 0 : x ∈ tB .
Properties
γB(x) ≥ 0 ∀x ∈ Rn, γB(x) = 0 iff x = 0.
γB(λx) = λγB(x) ∀x ∈ Rn, λ ∈ R+.
γB(x + y) ≤ γB(x) + γB(y) ∀x , y ∈ Rn.
γB : convex in Rn.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Gauges
Let B ⊂ Rn, convex, compact, with 0 ∈ int(B). Define the gaugeγB with unit ball B as
γB(x) = min t ≥ 0 : x ∈ tB .
Properties
γB(x) ≥ 0 ∀x ∈ Rn, γB(x) = 0 iff x = 0.
γB(λx) = λγB(x) ∀x ∈ Rn, λ ∈ R+.
γB(x + y) ≤ γB(x) + γB(y) ∀x , y ∈ Rn.
γB : convex in Rn.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Gauges
Let B ⊂ Rn, convex, compact, with 0 ∈ int(B). Define the gaugeγB with unit ball B as
γB(x) = min t ≥ 0 : x ∈ tB .
Properties
γB(x) ≥ 0 ∀x ∈ Rn, γB(x) = 0 iff x = 0.
γB(λx) = λγB(x) ∀x ∈ Rn, λ ∈ R+.
γB(x + y) ≤ γB(x) + γB(y) ∀x , y ∈ Rn.
γB : convex in Rn.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Gauges
γ :
γ(x) ≥ 0∀x ∈ Rn
γ(x) = 0 iff x = 0γ(λx) = λγ(x)∀x ∈ Rn, λ ∈ R+
γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn
convex compactsets, with 0 :interior point
γ x : γ(x) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Gauges
γ :
γ(x) ≥ 0∀x ∈ Rn
γ(x) = 0 iff x = 0γ(λx) = λγ(x)∀x ∈ Rn, λ ∈ R+
γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn
convex compactsets, with 0 :interior point
γ x : γ(x) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Gauges
γ :
γ(x) ≥ 0∀x ∈ Rn
γ(x) = 0 iff x = 0γ(λx) = λγ(x)∀x ∈ Rn, λ ∈ R+
γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn
convex compactsets, with 0 :interior point
γ x : γ(x) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Symmetric gauges: norms
B : symmetric w.r.t. 0 iff γB : norm in Rn.
γ :
γ(x) ≥ 0∀x ∈ Rn
γ(x) = 0 iff x = 0γ(λx) = |λ|γ(x)∀x ∈ Rn
γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn
compact convexsets , symmetricw.r.t. 0 with 0 in
the interior
γ x : γ(x) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Symmetric gauges: norms
B : symmetric w.r.t. 0 iff γB : norm in Rn.
γ :
γ(x) ≥ 0∀x ∈ Rn
γ(x) = 0 iff x = 0γ(λx) = |λ|γ(x)∀x ∈ Rn
γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn
compact convexsets , symmetricw.r.t. 0 with 0 in
the interior
γ x : γ(x) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Symmetric gauges: norms
B : symmetric w.r.t. 0 iff γB : norm in Rn.
γ :
γ(x) ≥ 0∀x ∈ Rn
γ(x) = 0 iff x = 0γ(λx) = |λ|γ(x)∀x ∈ Rn
γ(x + y) ≤ γ(x) + γ(y)∀x , y ∈ Rn
compact convexsets , symmetricw.r.t. 0 with 0 in
the interior
γ x : γ(x) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Norms. Examples
Euclidean (`2): γ(x) =√
x>x
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Norms. Examples
Manhattan (`1): γ(x) =∑
i |xi |
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Norms. Examples
Chebyshev (`∞): γ(x) = maxi|xi |
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Norms. Examples
That’s not a gauge!!!
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Norms. Examples
Composite norm
Let γ1, . . . , γk : norms in Rn1 , . . . ,Rnk . Let τ : norm in Rk ,no-decreasing in Rk
+. Then,
function ν, defined as
ν(s1, s2, . . . , sk) = τ(γ1(s1), . . . , γk(sk)) :
norm in Rn1 × . . .× Rnk .
for n1 = n2 = . . . = nk = n, function
ν(x) = τ(γ1(x), . . . , γk(x)) :
norm in Rn.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Norms. Examples
Norm of sorted components
Let c : Rn −→ Rn s.t., for each x ∈ Rn, c(x) is the sorting of(|x1|, . . . , |xn|) with c1(x) ≥ c2(x) ≥ . . . ≥ cn(x). Function
x 7−→ γ(x) =∑
j
ωjcj(x)
is a norm if ω1 ≥ . . . ≥ ωn ≥ 0, ω1 > 0.
Particular cases:
ω1 ω2 . . . ωn γ1 0 . . . 0 `∞1 1 . . . 1 `1λ µ . . . µ `1,∞
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Norms. Examples
Norm of sorted components
Let c : Rn −→ Rn s.t., for each x ∈ Rn, c(x) is the sorting of(|x1|, . . . , |xn|) with c1(x) ≥ c2(x) ≥ . . . ≥ cn(x). Function
x 7−→ γ(x) =∑
j
ωjcj(x)
is a norm if ω1 ≥ . . . ≥ ωn ≥ 0, ω1 > 0.
Particular cases:
ω1 ω2 . . . ωn γ1 0 . . . 0 `∞1 1 . . . 1 `1λ µ . . . µ `1,∞
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Skewed gauge
Given p ∈ Rn, ‖p‖2 < 1, define γ as
γ(x) = ‖x‖2 + p>x .
p = (0, 0)
p = (0.3, 0) p = (0.5, 0)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Skewed gauge
Given p ∈ Rn, ‖p‖2 < 1, define γ as
γ(x) = ‖x‖2 + p>x .
p = (0, 0) p = (0.3, 0)
p = (0.5, 0)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Skewed gauge
Given p ∈ Rn, ‖p‖2 < 1, define γ as
γ(x) = ‖x‖2 + p>x .
p = (0, 0) p = (0.3, 0) p = (0.5, 0)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Skewed gauge
M. Cera, J.A. Mesa, F.A. Ortega, F. Plastria
”Locating a central hunter on the plane”
JOTA 136 (2008) 155–166.
P. Chaudhuri
”On a geometric notion on quantiles for multivariate data”
Journal of the American Statistical Association 91 (1996) 862–872.
F. Plastria
”On destination optimality in asymmetric distance Fermat-Weberproblems”
Annals of Operations Research 40 (1992) 355–369.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Skewed gauge
Skewed gauge in Rn :
Given p ∈ Rn, ‖p‖2 < 1, define γ :
γ(x) = ‖x‖2 + p>x .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Skewed gauge
Skewed gauge in R :
Given p ∈ R, |p| < 1, define γ :
γ(x) = |x |+ px =
(1 + p)x , si x > 0,−(1− p)x , si x ≤ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Skewed gauge
Skewed gauge in R :
Given p ∈ R, |p| < 1, define γ :
γ(x) = |x |+ px =
(1 + p)x , si x > 0,−(1− p)x , si x ≤ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polyhedral gauge
Let E ⊂ Rn : finite, with 0 ∈ conv(E ), affine(E ) = Rn (or,equivalently, 0 ∈ int(conv(E )). Then, the gauge γ,
γ(x) = maxe∈E
e>x = maxe∈conv(E)
e>x
is a polyhedral gauge.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polyhedral gauge
Let E ⊂ Rn : finite, with 0 ∈ conv(E ), affine(E ) = Rn (or,equivalently, 0 ∈ int(conv(E )). Then, the gauge γ,
γ(x) = maxe∈E
e>x = maxe∈conv(E)
e>x
is a polyhedral gauge.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polyhedral gauge
Let E ⊂ Rn : finite, and let γ(x) = maxe∈E e>x . Then γ is a polyhedralgauge iff it holds:
the following LP has optimal value 0
max∑
e∈E λe
s.t. 0 =∑
e∈E λeeλe ≥ 0 ∀e ∈ E .
∀i = 1, 2, . . . , n, δ ∈ −1, 1, the following LP has strictly positiveoptimal value
min zs.t. z ≥ e>x ∀e ∈ E
xi = δx ∈ Rn.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polyhedral gauge
Let E ⊂ Rn : finite, and let γ(x) = maxe∈E e>x . Then γ is a polyhedralgauge iff it holds:
the following LP has optimal value 0
max∑
e∈E λe
s.t. 0 =∑
e∈E λeeλe ≥ 0 ∀e ∈ E .
∀i = 1, 2, . . . , n, δ ∈ −1, 1, the following LP has strictly positiveoptimal value
min zs.t. z ≥ e>x ∀e ∈ E
xi = δx ∈ Rn.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Polyhedral gauge
Let E ⊂ Rn : finite, and let γ(x) = maxe∈E e>x . Then γ is a polyhedralgauge iff it holds:
the following LP has optimal value 0
max∑
e∈E λe
s.t. 0 =∑
e∈E λeeλe ≥ 0 ∀e ∈ E .
∀i = 1, 2, . . . , n, δ ∈ −1, 1, the following LP has strictly positiveoptimal value
min zs.t. z ≥ e>x ∀e ∈ E
xi = δx ∈ Rn.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Dual of a gauge
Dual gauge
Let γB : gauge, with unit ball B. Its dual γB is
γB(x) = maxp∈B
p>x
Dual of a polyhedral gauge
γ(x) = maxe∈E e>x :
γ(x) = maxγ(u)≤1
u>x
= maxu>x : e>u ≤ 1∀e ∈ E= min
∑e∈E
λe :∑e∈E
λee = x , λe ≥ 0 ∀e ∈ E
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Dual of a gauge
Dual gauge
Let γB : gauge, with unit ball B. Its dual γB is
γB(x) = maxp∈B
p>x
Dual of a polyhedral gauge
γ(x) = maxe∈E e>x :
γ(x) = maxγ(u)≤1
u>x
= maxu>x : e>u ≤ 1∀e ∈ E= min
∑e∈E
λe :∑e∈E
λee = x , λe ≥ 0 ∀e ∈ E
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Cauchy-Schwarz inequality
γB is a gauge with unit ball B :
γB = γB
((γB))
= γB = γB
γB(x) = ((γB))
(x) = maxu∈B
u>x = maxu 6=0
u>x
γB(u)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Cauchy-Schwarz inequality
γB is a gauge with unit ball B :
γB = γB
((γB))
= γB = γB
γB(x) = ((γB))
(x) = maxu∈B
u>x = maxu 6=0
u>x
γB(u)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Cauchy-Schwarz inequality
γB is a gauge with unit ball B :
γB = γB
((γB))
= γB = γB
γB(x) = ((γB))
(x) = maxu∈B
u>x = maxu 6=0
u>x
γB(u)
Cauchy-Schwarz
γB(x)γB(u) ≥ x>u ∀x , u
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Cauchy-Schwarz inequality
γB is a gauge with unit ball B :
γB = γB
((γB))
= γB = γB
γB(x) = ((γB))
(x) = maxu∈B
u>x = maxu 6=0
u>x
γB(u)
Cauchy-Schwarz for Euclidean norm . . .
‖x‖2‖u‖2 ≥ x>u ∀x , u
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Convex functions, continuity and differentiability
Let f : convex in the open convex set S ⊂ Rn.
f is continuous in S .
The set of points of non-differentiability of f has Lebesguemeasure 0.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Subdiferentiability
Let f : convex in the convex set S ⊂ Rn, and x0 ∈ S : punt ofdifferentiability of f . Then
f (x) ≥ f (x0) +∇f (x0)>(x − x0) ∀x ∈ S .
Let f : convex in the convex set S ⊂ Rn. Given x0 ∈ S , thesubdifferential ∂f (x0) of f at x0 is defined as
∂f (x0) =
p : f (x) ≥ f (x0) + p>(x − x0) ∀x.
p ∈ ∂f (x0) iff the affine function x 7−→ f (x0) + p>(x − x0) is alower bound of f .
∂f (x) is convex, closed and non-empty for all x ∈ int(S)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Subdiferentiability
Let f : convex in the convex set S ⊂ Rn, and x0 ∈ S : punt ofdifferentiability of f . Then
f (x) ≥ f (x0) +∇f (x0)>(x − x0) ∀x ∈ S .
Let f : convex in the convex set S ⊂ Rn. Given x0 ∈ S , thesubdifferential ∂f (x0) of f at x0 is defined as
∂f (x0) =
p : f (x) ≥ f (x0) + p>(x − x0) ∀x.
p ∈ ∂f (x0) iff the affine function x 7−→ f (x0) + p>(x − x0) is alower bound of f .
∂f (x) is convex, closed and non-empty for all x ∈ int(S)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Subdiferentiability
Let f : convex in the convex set S ⊂ Rn, and x0 ∈ S : punt ofdifferentiability of f . Then
f (x) ≥ f (x0) +∇f (x0)>(x − x0) ∀x ∈ S .
Let f : convex in the convex set S ⊂ Rn. Given x0 ∈ S , thesubdifferential ∂f (x0) of f at x0 is defined as
∂f (x0) =
p : f (x) ≥ f (x0) + p>(x − x0) ∀x.
p ∈ ∂f (x0) iff the affine function x 7−→ f (x0) + p>(x − x0) is alower bound of f .
∂f (x) is convex, closed and non-empty for all x ∈ int(S)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Subdiferentials and directional derivatives
Let f : convex in the open convex S ⊂ Rn. For x0 ∈ S , d ∈ Rn,d 6= 0, ∃∇f (x0; d), the directional derivative of f at x0 in thedirection d . Moreover, it holds:
∇f (x0; d) = maxp∈∂f (x0)
p>d
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Subdiferential calculus
If f is convex and differentiable at x0, then
∂f (x0) = ∇f (x0)
Let f1, f2, . . . , fk : convex in the open convex S ⊂ Rn. For x ∈ S ,
∂ (f1(x) + f2(x) + . . .+ fk(x)) = ∂f1(x)+∂f2(x)+. . .+∂fk(x).
∂max1≤i≤k fi (x) = conv(⋃
i∈A(x) ∂fi (x), conA(x) = j : fj(x) = max1≤i≤k fi (x)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Subdiferential calculus
If f is convex and differentiable at x0, then
∂f (x0) = ∇f (x0)
Let f1, f2, . . . , fk : convex in the open convex S ⊂ Rn. For x ∈ S ,
∂ (f1(x) + f2(x) + . . .+ fk(x)) = ∂f1(x)+∂f2(x)+. . .+∂fk(x).
∂max1≤i≤k fi (x) = conv(⋃
i∈A(x) ∂fi (x), conA(x) = j : fj(x) = max1≤i≤k fi (x)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Example
Let E ⊂ Rn, finite, aee∈E , f (x) = maxe∈E e>(x − ae).
∂f (x) = conv(
e : f (x) = e>(x − ae))
Let f (t) = |t − a| = maxt − a, a− t. Then
∂f (t) =
−1, si t < aconv(−1, 1) = [−1, 1], si t = a1, si t > a
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Example
Let E ⊂ Rn, finite, aee∈E , f (x) = maxe∈E e>(x − ae).
∂f (x) = conv(
e : f (x) = e>(x − ae))
Let f (t) = |t − a| = maxt − a, a− t. Then
∂f (t) =
−1, si t < aconv(−1, 1) = [−1, 1], si t = a1, si t > a
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Composition with affine functions
Let f convex in Rm, and let A ∈ Rm×n, b ∈ Rm. Let g be theconvex function in Rn g(x) = f (Ax + b). Then
∂g(x) = A>∂f (y)|y=Ax+b
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
Subdifferential of a gauge
∂γ(x) =
u ∈ B : u>x = max
u∈Bu>x
∂γ(0) = B
For x 6= 0, ∂γ(x) is an exposed face of B :
p ∈ ∂γ(x) iff γ(p) = 1, p>x = γ(x).
γ(x) = maxp∈∂γ(x)
p>x
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Examples
∂‖x‖2 =
1‖x‖2 x , si x 6= 0
y : ‖y‖2 ≤ 1, si x = 0
Let p ∈ Rn, ‖p‖2 < 1, γ(x) = ‖x‖2 + p>x . Then
∂γ(x) =
1‖x‖2 x + p, si x 6= 0
y + p : ‖y‖2 ≤ 1, si x = 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Subdifferentials of monotonic norms
Let γ : norm, monotonic in Rn+. Given x ∈ Rn
+,
1 If x ∈ Rn++, then ∂γ(x) ⊂ Rn
+.
2 If x ∈ Rn+, then ∂γ(x) ∩ Rn
+ 6= ∅.3 If p ∈ Rn
+, then γ(p) = maxγ(x)≤1,x∈Rn+
p>x .
4 γ is monotonic in Rn+.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
A few examplesA few ideas on convexityGauges
Dual of a composite gauge
Let γ1, . . . , γk : gauges in Rn1 , . . . ,Rnk y τ : gauge in Rk ,non-decreasing in Rk
+. The composite gauge,ν(s1, s2, . . . , sk) = τ(γ1(s1), . . . , γk(sk)), has as dual
ν(s1, . . . , sk) = τ(γ1(s1), . . . , γk(sk))
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
inf γ(Cx + c) + d>xs. t. x ∈ K
(P)
γ : gauge in IRm
C : IRm×n matrix
c ∈ IRm, d ∈ IRn
K = M + E ⊆ IRn nonempty asymptotically conical set witha. c. r. (M,E ).
d = 0 : gauge- or homogeneous programming, e.g. Freund,Math Prog, 1987. Glassey, Math Prog, 1976, Gwinner, JOTA,1985, . . .
K = IRn, Kaplan & Yang, Math Prog, 1997.
General: Carrizosa & Fliege, Math Prog, 2002.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
inf γ(Cx + c) + d>xs. t. x ∈ K
(P)
γ : gauge in IRm
C : IRm×n matrix
c ∈ IRm, d ∈ IRn
K = M + E ⊆ IRn nonempty asymptotically conical set witha. c. r. (M,E ).
d = 0 : gauge- or homogeneous programming, e.g. Freund,Math Prog, 1987. Glassey, Math Prog, 1976, Gwinner, JOTA,1985, . . .
K = IRn, Kaplan & Yang, Math Prog, 1997.
General: Carrizosa & Fliege, Math Prog, 2002.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
S ⊆ IRn,S 6= ∅ : asymptotically conical set (a.c.s.) if has the form
S = M + E ,
with
M : compact convex
E : closed convex cone
(M,E ) : asymptotically conical representation (a. c. r.) of S .
If (M,E ) : a. c. r. of K , then K∞ = E
Let (M1,E1), (M2,E2) : a. c. r. of K1,K2 ⊆ IRn. Then,
(M1 ×M2,E1 × E2) : a. c. r. of K1 × K2
(M1 + M2,E1 + E2) : a. c. r. of K1 + K2
Let A(x) = Ax + b. Then, (AM + b,AE ) : a.c.r. of A(K )
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
S ⊆ IRn,S 6= ∅ : asymptotically conical set (a.c.s.) if has the form
S = M + E ,
with
M : compact convex
E : closed convex cone
(M,E ) : asymptotically conical representation (a. c. r.) of S .
If (M,E ) : a. c. r. of K , then K∞ = E
Let (M1,E1), (M2,E2) : a. c. r. of K1,K2 ⊆ IRn. Then,
(M1 ×M2,E1 × E2) : a. c. r. of K1 × K2
(M1 + M2,E1 + E2) : a. c. r. of K1 + K2
Let A(x) = Ax + b. Then, (AM + b,AE ) : a.c.r. of A(K )
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
A.c.s.:
compact setspolyhedraaffine spacesclosed convex cones. . .
Class of a.c.s. not closed under intersections
The inverse image of an a.c.s. under an affine mapping: notnecessarily a.c.s.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Duality
(P)︷ ︸︸ ︷infx∈K
(γ(Cx + c) + d>x
)=
(D)︷ ︸︸ ︷max
γ(u)≤1
(u>c + inf
x∈Kx>(C>u + d
))
S∗ := x ∈ IRn | x>s ≥ 0 for all s ∈ Sδ∗S(x) := sup
x>y | y ∈ S
inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E ∗
γ(u) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Duality
(P)︷ ︸︸ ︷infx∈K
(γ(Cx + c) + d>x
)=
(D)︷ ︸︸ ︷max
γ(u)≤1
(u>c + inf
x∈Kx>(C>u + d
))
S∗ := x ∈ IRn | x>s ≥ 0 for all s ∈ Sδ∗S(x) := sup
x>y | y ∈ S
inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E ∗
γ(u) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Duality
(P)︷ ︸︸ ︷infx∈K
(γ(Cx + c) + d>x
)=
(D)︷ ︸︸ ︷max
γ(u)≤1
(u>c + inf
x∈Kx>(C>u + d
))
S∗ := x ∈ IRn | x>s ≥ 0 for all s ∈ Sδ∗S(x) := sup
x>y | y ∈ S
inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E ∗
γ(u) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
The unconstrained case: K = IRn
inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E∗
γ(u) ≤ 1
M := 0E = IRn
max u>cs.t. C>u + d = 0
γ(u) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: distance to a cone
inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E∗
γ(u) ≤ 1
S : closed convex cone
x0 ∈ IRn
‖ · ‖ norm with unit ball B
infx∈S ‖x − x0‖
γ : ‖ · ‖C = I , c = −x0, d = 0
M = 0, E = S
infx∈S‖x − x0‖ = δ∗S∗∩B(−x0)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: distance to a cone
inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E∗
γ(u) ≤ 1
S : closed convex cone
x0 ∈ IRn
‖ · ‖ norm with unit ball B
infx∈S ‖x − x0‖
γ : ‖ · ‖C = I , c = −x0, d = 0
M = 0, E = S
infx∈S‖x − x0‖ = δ∗S∗∩B(−x0)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: distance to a hyperplane
inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E∗
γ(u) ≤ 1
H = x ∈ IRn : ω>(x − p) = 0
‖ · ‖ gauge
infx∈H ‖x − x0‖
γ : ‖ · ‖
C = I , c = −x0, d = 0
M = p, E = x : ω>x = 0
infx∈H‖x − x0‖ = max
ω>(p − x0)
‖ω‖,ω>(x0 − p)
‖ − ω‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: distance to a hyperplane
inf γ(Cx + c) + d>x = max u>c − δ∗M(−C>u − d)s.t. x ∈ M + E s.t. C>u + d ∈ E∗
γ(u) ≤ 1
H = x ∈ IRn : ω>(x − p) = 0
‖ · ‖ gauge
infx∈H ‖x − x0‖
γ : ‖ · ‖
C = I , c = −x0, d = 0
M = p, E = x : ω>x = 0
infx∈H‖x − x0‖ = max
ω>(p − x0)
‖ω‖,ω>(x0 − p)
‖ − ω‖
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: single facility location
minx∈IRn
γ(γ1(x−a1), . . . , γm(x−am)) γ(u1, . . . , um) := γ(γ1(u1), . . . , γm(um)
)
minx∈IRn
γ
In×n
In×n
...In×n
x +
−a1
−a2
...−am
max u>cs.t. C>u + d = 0
γ(u) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: single facility location
minx∈IRn
γ(γ1(x−a1), . . . , γm(x−am)) γ(u1, . . . , um) := γ(γ1(u1), . . . , γm(um)
)
minx∈IRn
γ
In×n
In×n
...In×n
x +
−a1
−a2
...−am
max u>cs.t. C>u + d = 0
γ(u) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: single facility location
minx∈IRn
γ(γ1(x−a1), . . . , γm(x−am)) γ(u1, . . . , um) := γ(γ1(u1), . . . , γm(um)
)
minx∈IRn
γ
In×n
In×n
...In×n
x +
−a1
−a2
...−am
max u>cs.t. C>u + d = 0
γ(u) ≤ 1
max −∑m
j=1 u>j aj
s.t.∑m
j=1 uj = 0n×1
γ(u1, . . . , um) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: single facility location
minx∈IRn
γ(γ1(x−a1), . . . , γm(x−am)) γ(u1, . . . , um) := γ(γ1(u1), . . . , γm(um)
)
minx∈IRn
γ
In×n
In×n
...In×n
x +
−a1
−a2
...−am
max u>cs.t. C>u + d = 0
γ(u) ≤ 1
max −∑m
j=1 u>j aj
s.t.∑m
j=1 uj = 0n×1
γ(γ1 (u1), . . . , γm(um)) ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
The linearly-constrained case: K = x : Ax ≥ b
infx∈Ax≥b
(γ(Cx + c) + d>x
)= maxγ(u)≤1
(c>u + inf
Ax≥bx>(C>u + d
))
max c>u + b>vs.t. −C>u + A>v = d
γ(u) ≤ 1v ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
The linearly-constrained case: K = x : Ax ≥ b
infx∈Ax≥b
(γ(Cx + c) + d>x
)= maxγ(u)≤1
(c>u + inf
Ax≥bx>(C>u + d
))
max c>u + b>vs.t. −C>u + A>v = d
γ(u) ≤ 1v ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: Support Vector Machines
min γ(Cx + c) + d>xs.t. Ax ≥ b
max c>u + b>vs.t. −C>u + A>v = d
γ(u) ≤ 1v ≥ 0
min ν
((In×n, 01×n)
(ωβ
))
s.t.
y1x>1 y1
y2x>2 y2
......
ymx>m ym
(ωβ
)≥
11...1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: Support Vector Machines
min γ(Cx + c) + d>xs.t. Ax ≥ b
max c>u + b>vs.t. −C>u + A>v = d
γ(u) ≤ 1v ≥ 0
min ν
((In×n, 01×n)
(ωβ
))
s.t.
y1x>1 y1
y2x>2 y2
......
ymx>m ym
(ωβ
)≥
11...1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: Support Vector Machines
min γ(Cx + c) + d>xs.t. Ax ≥ b
max c>u + b>vs.t. −C>u + A>v = d
γ(u) ≤ 1v ≥ 0
min ν
((In×n, 01×n)
(ωβ
))
s.t.
y1x>1 y1
y2x>2 y2
......
ymx>m ym
(ωβ
)≥
11...1
max
∑mj=1 vj
s.t.∑m
j=1 vjyj = 0∑mj=1 vjyjxj = u
ν(u) ≤ 1v ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: Support Vector Machines
min γ(Cx + c) + d>xs.t. Ax ≥ b
max c>u + b>vs.t. −C>u + A>v = d
γ(u) ≤ 1v ≥ 0
min ν
((In×n, 01×n)
(ωβ
))
s.t.
y1x>1 y1
y2x>2 y2
......
ymx>m ym
(ωβ
)≥
11...1
max
∑mj=1 vj
s.t.∑m
j=1 vjyj = 0
ν(∑m
j=1 vjyjxj) ≤ 1
v ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
Application: Support Vector Machines
min γ(Cx + c) + d>xs.t. Ax ≥ b
max c>u + b>vs.t. −C>u + A>v = d
γ(u) ≤ 1v ≥ 0
min ν
((In×n, 01×n)
(ωβ
))
s.t.
y1x>1 y1
y2x>2 y2
......
ymx>m ym
(ωβ
)≥
11...1
min ν(a+ − a−)s.t. a+ ∈ conv (xi : i ∈ I +)
a− ∈ conv (xi : i ∈ I−)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
SVM dual and closest pairs
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
SVM dual and closest pairs
a+*
a-*
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Problem statementDuality
SVM dual and closest pairs
1/2 a+* +1/2 a-
*
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
The set of optimal solutions of a convex problem
Let f : convex in the closed convex set S ⊂ Rn. The set of optimalsolutions to minx∈S f (x) is closed and convex, though it may beempty.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Optimality conditions. Unconstrained convex problems
minx∈Rn
f (x) x∗ : optimal solution iff 0 ∈ ∂f (x∗)
Under differentiability: ∂f (x∗) = ∇f (x∗)
x∗ : optimal solution iff 0 ∈ ∇f (x∗)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Optimality conditions. Unconstrained convex problems
minx∈Rn
f (x) x∗ : optimal solution iff 0 ∈ ∂f (x∗)
Under differentiability: ∂f (x∗) = ∇f (x∗)
x∗ : optimal solution iff 0 ∈ ∇f (x∗)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂|x − a|
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1].
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂|x − a|
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1].
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂|x − a|
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1].
for x 6∈ A :
0 =∑
a∈A,a<x
ωa −∑
a∈A,a>x
ωa.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂|x − a|
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1].
for x 6∈ A : ∑a∈A,a<x
ωa =∑
a∈A,a>x
ωa =1
2.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂|x − a|
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1].
for x = a0 ∈ A :
−∑
a∈A,a<a0
ωa +∑
a∈A,a>a0
ωa ∈ ωa0 · [−1, 1]
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂|x − a|
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1].
for x = a0 ∈ A :
−ωa0 ≤ −∑
a∈A,a<a0
ωa +∑
a∈A,a>a0
ωa ≤ ωa0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂|x − a|
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1].
for x = a0 ∈ A : ∑a∈A,a≥a0
ωa ≥∑
a∈A,a<a0ωa∑
a∈A,a≤a0ωa ≥
∑a∈A,a>a0
ωa
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂|x − a|
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1].
for x = a0 ∈ A : ∑a∈A,a≥a0
ωa ≥ 12∑
a∈A,a≤a0ωa ≥ 1
2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|Optimal solution: median(s) ofA.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|Optimal solution: median(s) ofA.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line
minx∈R
f (x) :=∑a∈A
ωa|x − a|Optimal solution: median(s) ofA.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line ( skewedgauge)
minx∈R
f (x) :=∑a∈A
ωa (|x − a|+ p(x − a))
w.l.o.g.,∑
a∈A ωa = 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line ( skewedgauge)
minx∈R
f (x) :=∑a∈A
ωa (|x − a|+ p(x − a))
w.l.o.g.,∑
a∈A ωa = 1
Optimality conditions:
0 ∈ ∂f (x)=
∑a∈A ωa∂ (|x − a|+ p(x − a))
=∑
a∈A,a<x ωa −∑
a∈A,a>x ωa +∑
a∈A,a=x ωa[−1, 1] + p.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line ( skewedgauge)
minx∈R
f (x) :=∑a∈A
ωa (|x − a|+ p(x − a))
w.l.o.g.,∑
a∈A ωa = 1
for x 6∈ A : ∑a∈A,a<x ωa = 1−p
2∑a∈A,a>x ωa = 1+p
2 .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in the line ( skewedgauge)
minx∈R
f (x) :=∑a∈A
ωa (|x − a|+ p(x − a))
w.l.o.g.,∑
a∈A ωa = 1
for x = a0 ∈ A : ∑a∈A,a≤a0
ωa ≥ 1−p2∑
a∈A,a≥a0ωa ≥ 1+p
2 .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Euclidean Fermat-Weber in Rn
minx∈Rn
f (x) :=∑a∈A
ωa‖x − a‖2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Euclidean Fermat-Weber in Rn
minx∈Rn
f (x) :=∑a∈A
ωa‖x − a‖2
For x 6∈ A,
0 ∈ ∂f (x) = ∇f (x)Optimality at x :
∑a∈A
ωa‖x−a‖2 (x − a) = 0
x =∑
a∈A Ωa(x)a, with Ωa(x) = ωa/‖x−a‖2∑b∈A ωb/‖x−b‖2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Euclidean Fermat-Weber in Rn
minx∈Rn
f (x) :=∑a∈A
ωa‖x − a‖2
For x = a0 ∈ A,
0 ∈ ∂f (x) =∑
a 6=a0
ωa‖x−a‖2 (x − a) +ωa0B (B : unit ball of `2)
Optimality at x = a0: ‖∑
a∈A1ωa0
ωa‖x−a‖2 (x − a)‖2 ≤ 1.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Euclidean Fermat-Weber in Rn
minx∈Rn
f (x) :=∑a∈A
ωa‖x − a‖2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
minx∈R2
‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
minx∈R2
‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
minx∈R2
‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
For x 6∈ a, b, c
0 =x − a
‖x − a‖+
x − b
‖x − b‖+
x − c
‖x − c‖0 = pa + pb + pc
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
For x 6∈ a, b, c
0 =x − a
‖x − a‖+
x − b
‖x − b‖+
x − c
‖x − c‖0 = pa + pb + pc
−1 = p>a pb + p>a pc
−1 = p>a pb + p>b pc
−1 = p>a pc + p>b pc
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
For x 6∈ a, b, c
0 =x − a
‖x − a‖+
x − b
‖x − b‖+
x − c
‖x − c‖0 = pa + pb + pc
p>a pb = p>a pc = p>b pc = −1
2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
minx∈R2
‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2
For x ∈ a, b, c, x = a say
0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
minx∈R2
‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2
For x ∈ a, b, c, x = a say
0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)
∥∥∥∥ a− b
‖a− b‖+
a− c
‖a− c‖
∥∥∥∥ ≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
minx∈R2
‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2
For x ∈ a, b, c, x = a say
0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)
∥∥∥∥ a− b
‖a− b‖+
a− c
‖a− c‖
∥∥∥∥2
≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
minx∈R2
‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2
For x ∈ a, b, c, x = a say
0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)
∥∥∥∥ a− b
‖a− b‖
∥∥∥∥2
+
∥∥∥∥ a− c
‖a− c‖
∥∥∥∥2
+ 2(a− b)>(a− c)
‖a− b‖‖a− c‖≤ 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat problem
minx∈R2
‖x − a‖2 + ‖x − b‖2 + ‖x − c‖2
For x ∈ a, b, c, x = a say
0 ∈ ∂ (‖x − a‖+ ‖x − b‖+ ‖x − c‖)
(a− b)>(a− c)
‖a− b‖‖a− c‖≤ −1
2
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat-Weber problem in Rn. Different norms yielddifferent solutions
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat-Weber problem in Rn. Different norms yielddifferent solutions
γ : `2, . . .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Fermat-Weber problem in Rn. Different norms yielddifferent solutions
γ : `1, . . .
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Center problem in Rn
Let A ⊂ Rn, finite. Let γ : norm
minx∈Rn
f (x) := maxa∈A
γ(x − a)
Optimality at x∗ :
0 ∈ conv (∂γ(x∗ − a) : f (x∗) = γ(x∗ − a))
Caratheodory:
If x∗ : optimal, then ∃A∗ ⊂ A, with cardinality |A∗| ≤ n + 1 s.t.x∗ : optimal for
minx∈Rn
f (x) := maxa∈A∗
γ(x − a)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Center problem in Rn
Let A ⊂ Rn, finite. Let γ : norm
minx∈Rn
f (x) := maxa∈A
γ(x − a)
Optimality at x∗ :
0 ∈ conv (∂γ(x∗ − a) : f (x∗) = γ(x∗ − a))
Caratheodory:
If x∗ : optimal, then ∃A∗ ⊂ A, with cardinality |A∗| ≤ n + 1 s.t.x∗ : optimal for
minx∈Rn
f (x) := maxa∈A∗
γ(x − a)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Center problem in Rn
Let A ⊂ Rn, finite. Let γ : norm
minx∈Rn
f (x) := maxa∈A
γ(x − a)
Optimality at x∗ :
0 ∈ conv (∂γ(x∗ − a) : f (x∗) = γ(x∗ − a))
Caratheodory:
If x∗ : optimal, then ∃A∗ ⊂ A, with cardinality |A∗| ≤ n + 1 s.t.x∗ : optimal for
minx∈Rn
f (x) := maxa∈A∗
γ(x − a)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Optimality conditions. Constrained convex problems
minx∈S
f (x)
x∗ : optimal solution iff 0 ∈ ∂f (x∗) + NS(x∗)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in a segment
minx∈S
f (x) :=∑a∈A
ωa|x − a|
S = [s1, s2], s1 < s2, s1, s2 6∈ A.∑
a∈A ωa = 1
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in a segment
minx∈S
f (x) :=∑a∈A
ωa|x − a|
S = [s1, s2], s1 < s2, s1, s2 6∈ A.∑
a∈A ωa = 1
NS(x) =
R−, si x = s1
0, si s1 < x < s2
R+, si x = s2.
Optimality conditions in x ∈ (s1, s2) : x : median of A
Optimality at xs1 : 0 ∈∑
a<x ωa −∑
a>x ωa + R−.Optimality at xs2 : 0 ∈
∑a<x ωa −
∑a>x ωa + R+.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in a segment
minx∈S
f (x) :=∑a∈A
ωa|x − a|
S = [s1, s2], s1 < s2, s1, s2 6∈ A.∑
a∈A ωa = 1
NS(x) =
R−, si x = s1
0, si s1 < x < s2
R+, si x = s2.
Optimality conditions in x ∈ (s1, s2) : x : median of A
Optimality at xs1 : 0 ∈∑
a<x ωa −∑
a>x ωa + R−.Optimality at xs2 : 0 ∈
∑a<x ωa −
∑a>x ωa + R+.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Example. Fermat-Weber problem in a segment
minx∈S
f (x) :=∑a∈A
ωa|x − a|
S = [s1, s2], s1 < s2, s1, s2 6∈ A.∑
a∈A ωa = 1
NS(x) =
R−, if x = s1
0, if s1 < x < s2
R+, if x = s2.
Optimality at x ∈ (s1, s2) : x : median of A
Optimality at xs1 :∑
a>x ωa ≤ 12
Optimality at xs2 :∑
a<x ωa ≤ 12
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Probabilities approximately proportional to a vector
min∑N
i=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ≥ 0
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Probabilities approximately proportional to a vector
min∑N
i=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ∈ R
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Probabilities approximately proportional to a vector
min∑N
i=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ∈ R
(πi
ci− k)
1ci
+ αi − βi − ϑ = 0 ∀i = 1, 2, . . . ,N1N
∑Ni=1
πi
ci− k = 0∑N
i=1 πi = nπi ≤ 1 ∀i = 1, 2, . . . ,Nπi ≥ 0 ∀i = 1, 2, . . . ,N
αi (1− πi ) = 0 ∀i = 1, 2, . . . ,Nβiπi = 0 ∀i = 1, 2, . . . ,Nαi , βi ≥ 0 ∀i = 1, 2, . . . ,N
k , ϑ ∈ REmilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Probabilities approximately proportional to a vector
min∑N
i=1
(πi
ci− k)2
s.t. 0 ≤ πi ≤ 1 ∀i = 1, 2, . . . ,N∑Ni=1 πi = n0
k ∈ R
From this, an O(N log(N))-time algorithm yields an optimal solution.
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Hard-margin SVM
min 12ω>ω
s.t. yi (ω>xi + β) ≥ 1∀i ∈ I
ω ∈ Rn, β ∈ R
0 ∈ ∂f (ω, β) + NS(ω, β)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Hard-margin SVM
min 12ω>ω
s.t. yi (ω>xi + β) ≥ 1∀i ∈ I
ω ∈ Rn, β ∈ R
0 = ω −∑i∈I
λiyixi
0 =∑i∈I
λiyi
0 ≤ λi ∀i ∈ I
0 = λi
(yi (ω
>xi + β)− 1)∀i ∈ I
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Hard-margin SVM
I +(λ) =
i ∈ I + : λi > 0
I−(λ) =
i ∈ I− : λi > 0
I (λ) = I +(λ) ∪ I−(λ)
xi : i ∈ I (λ) : support vectors at λ
Let (ω∗, β∗) : optimal.
∃ λ ∈ RI multiplier at (ω∗, β∗), with |I (λ)| ≤ n + 1
For λ ∈ RI multiplier in (ω∗, β∗),
I +(λ) 6= ∅, I−(λ) 6= ∅.(ω∗, β∗) is also optimal to
min ω>ωs.t. yi (ω
>xi + β) ≥ 1 ∀i ∈ I (λ)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)
IntroductionGauge problems
Optimality conditions and examples
Unconstrained problemsUnconstrained problems
Hard-margin SVM
I +(λ) =
i ∈ I + : λi > 0
I−(λ) =
i ∈ I− : λi > 0
I (λ) = I +(λ) ∪ I−(λ)
xi : i ∈ I (λ) : support vectors at λ
Let (ω∗, β∗) : optimal.
∃ λ ∈ RI multiplier at (ω∗, β∗), with |I (λ)| ≤ n + 1
For λ ∈ RI multiplier in (ω∗, β∗),
I +(λ) 6= ∅, I−(λ) 6= ∅.(ω∗, β∗) is also optimal to
min ω>ωs.t. yi (ω
>xi + β) ≥ 1 ∀i ∈ I (λ)
Emilio Carrizosa, [email protected] MATHEMATICAL PROGRAMMING (I)