Modern Methods in Decision Making (2016)...2016/06/10 · Modern Methods in Decision Making (2016)...
Transcript of Modern Methods in Decision Making (2016)...2016/06/10 · Modern Methods in Decision Making (2016)...
Modern Methods in Decision Making (2016)
Assignment 2 - Solutions
Problem 4 in Problem Sheet 6. VC Theory.
(i) This is a simple consequence of the VC inequality, once you notice the relation R(y
n
)�inf
y2F R(y) 2 sup
y2F | ˆRn
(y)�R(y)| derived on page 16 of the lecture notes.
(ii) First, note that Jensen’s inequality ensures that (EZ)
2 E(Z2). Thus
E(Z2) =
Z 1
0
P(Z
2> s)ds =
Z
u
0
P(Z
2> s)ds+
Z 1
u
P(Z
2> s)ds
u+ c
Z 1
u
e
�2nsds
= u+
c
2n
e
�2nu.
The minimum is obtained for u = log c/(2n).
(iii) The relation established in (ii) is valid for all t > 0. Put t = ✏/16, for some ✏ > 0. Thenthe condition becomes
P(Z > t) = P(Z > ✏/16) = P(16Z > ✏) c e
�2nt2= c e
�n✏
2/128
.
This relation is satisfied for the random variable 16Z = |R(y
n
) � inf
y2F R(y)|, withc = 8S(F , n). Applying the lemma derived in (ii) yields
ER(y
n
)� inf
y2FR(y) = 16EZ 16
r
log(8eS(F , n))
2n
,
as required.
Problem 4 Part I in Problem Sheet 8. AdaBoost.See hand-written notes at the end
Problem 5 in Problem Sheet 9. Convex Optimization.
(i) The Lagrangian is
L(p,�, ⌫) =
X
i
p
i
log p
i
+ �
t
(Ap� b) + ⌫(1
t
p� 1) ,
and the Lagrange dual function is
g(�, ⌫) = inf
p
L(p,�, ⌫) = �b
t
�� ⌫ + inf
p
(
X
i
p
i
log p
i
+ (A
t
�+ 1
t
⌫)
t
p
)
.
Put (p) =P
i
p
i
log p
i
. The Lagrange dual function can be rewritten
g(�, ⌫) = �b
t
�� ⌫ �
⇤(�A
t
�� 1
t
⌫) ,
1
Modern Methods in Decision Making (2016)
where
⇤(y) = sup
x
(y
t
x� (x)) .
Note that in the special case (x) = x log x, we obtain ⇤(y) = e
y�1. Thus
g(�, ⌫) = �b
t
�� ⌫ �n
X
i=1
exp(�a
t
i
�� ⌫ � 1) ,
where a
i
denotes the i-th column of A. The dual problem is
maximize � b
t
�� ⌫ � e
�⌫�1n
X
i=1
e
�a
ti�
subject to A↵ + b� � = 0
� ⌫ 0 ,
The expression of the dual problem can be further simplified. For a fixed �,
@g(�, ⌫)
@⌫
= �1 + e
�⌫�1n
X
i=1
e
�a
ti�
= 0 ,
yields
⌫ = log
n
X
e
�(ati�+1)o
.
The function to maximise in the dual problem simplifies to
g(�) = �b
t
�� log
n
X
i=1
e
�a
ti�
!
.
(ii) Optimal gap is zero if there exists a p � 0 such that Ap � b and 1
t
p = 1.
2