Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random...
Transcript of Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random...
![Page 1: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/1.jpg)
Independent random variables
Will MonroeJuly 24, 2017
with materials byMehran Sahamiand Chris Piech
![Page 2: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/2.jpg)
Announcements: Midterm
Tomorrow!
Tuesday, July 25, 7:00-9:00pm
Building 320-105(main quad, Geology Corner)
One page of notes (front & back)
No books/computers/calculators
![Page 3: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/3.jpg)
Review: Joint distributions
A joint distribution combines multiple random variables. Its PDF or PMF gives the probability or relative likelihood of both random variables taking on specific values.
pX ,Y (a ,b)=P(X=a ,Y=b)
![Page 4: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/4.jpg)
Review: Joint PMF
A joint probability mass function gives the probability of more than one discrete random variable each taking on a specific value (an AND of the 2+ values).
pX ,Y (a ,b)=P(X=a ,Y=b)
01
2
0.050.100.05
0.200.100.10
0.100.100.20
0 1 2Y
X
![Page 5: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/5.jpg)
Review: Joint PDF
A joint probability density function gives the relative likelihood of more than one continuous random variable each taking on a specific value.
P(a1<X≤a2,b1<Y≤b2)=
∫a1
a2
dx∫b1
b2
dy f X ,Y (x , y)
![Page 6: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/6.jpg)
Review: Joint CDF
F X ,Y (x , y)=P(X≤x ,Y≤ y)
x
y
to 0 asx → -∞,y → -∞,
to 1 asx → +∞,y → +∞,
plot by Academo
![Page 7: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/7.jpg)
Probabilities from joint CDFsP(a1<X≤a2,b1<Y≤b2)=F X ,Y (a2,b2)
−FX ,Y (a1,b2)
−FX ,Y (a2,b1)
+F X ,Y (a1,b1)
a1
a2
b1
b2
![Page 8: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/8.jpg)
Review: Marginalization
Marginal probabilities give the distribution of a subset of the variables (often, just one) of a joint distribution.
Sum/integrate over the variables you don’t care about.
pX (a)=∑y
pX ,Y (a , y)
f X (a)=∫−∞
∞
dy f X ,Y (a , y)
![Page 9: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/9.jpg)
Review: Non-negative RVexpectation lemma
You can integrate y times the PMF, or you can integrate 1 minus the CDF!
E [Y ]=∫0
∞
dy P(Y > y)
=∫0
∞
dy (1−FY ( y))
FY ( y)
y
![Page 10: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/10.jpg)
Non-negative RV expectation lemma:Rearranging terms
E [X ]=0 P (X=0)+1 P(X=1)+2 P(X=2)+3 P(X=3)+⋯
=0 P(X=0)+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯
[Addendum]
1
2
3...
=0 P(X=0)+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯+1 P(X=1)+2 P(X=2)+3 P (X=3)+⋯
=0 P(X=0)+1 P(X≥1)+1 P(X=1)+2 P(X≥2)+1 P(X=1)+2 P(X=2)+3 P (X≥3)+⋯
=∑i=1
∞
P(X≥i)
![Page 11: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/11.jpg)
Non-negative RV expectation lemma:Graphically
E [X ]=∑
[Addendum]
0 P(X=0) 1 P(X=1) 2 P(X=2) 3 P(X=3) 4 P(X=4)0
0.05
0.1
0.15
0.2
0.25
0.3
P(X ≥ 1)P(X ≥ 2)
P(X ≥ 3)P(X ≥ 4)
x
p_
X(x
)
![Page 12: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/12.jpg)
Review: Multinomial random variable
An multinomial random variable records the number of times each outcome occurs, when an experiment with multiple outcomes (e.g. die roll) is run multiple times.
X1 ,…, Xm∼MN (n , p1, p2,…, pm)
P(X1=c1 , X2=c2 ,…, Xm=cm)
=( nc1 , c2 ,…, cm
) p1c1 p2
c2… pmcmvector!
![Page 13: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/13.jpg)
A question from last class
“Are X and Y independent?”
pX ,Y (a ,b)=P(X=a ,Y=b)
01
2
0.050.100.05
0.200.100.10
0.100.100.20
0 1 2Y
X
P(X=0,Y=0)=P(X=0)P(Y=0)
![Page 14: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/14.jpg)
Independence ofdiscrete random variables
Two random variables are independent if knowing the value of one tells you nothing about the value of the other (for all values!).
X⊥Y iff ∀ x , y :
P(X=x ,Y= y)=P(X=x)P(Y= y)- or -
pX ,Y (x , y)=pX (x) pY ( y)
![Page 15: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/15.jpg)
Coin flips
P(X=x ,Y= y)=(nx) px(1−p)n−x
(my ) py(1−p)m− y
=P(X=x)P(Y= y)
n flips m flipsX: number of headsin first n flips
Y: number of headsin next m flips
∴X⊥Y
![Page 16: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/16.jpg)
Coin flips
n flips m flipsX: number of headsin first n flips
X⟂Z
Z: total number of heads in n + m flips
Z=0→X=0
![Page 17: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/17.jpg)
Web server hits
Your web server gets N requests in a day. N ~ Poi(λ).
Each request comes independently from human (prob. p) or bot (1 – p).
X: # requests from humans in dayY: # requests from bots in day
Knowing N:
P(X=i ,Y= j)=P (X=i ,Y= j|N=i+ j)P (N=i+ j)
X∼Bin (N , p)Y∼Bin (N ,1−p)
+P (X=i ,Y= j|N≠i+ j)P(N≠i+ j)
=(i+ ji ) p
i(1−p) j =e−λ λ
i+ j
(i+ j)!
![Page 18: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/18.jpg)
Web server hits
P(X=i ,Y= j)=(i+ ji ) p
i(1−p) je−λ λ
i+ j
(i+ j)!
=(i+ j)!i ! j !
pi(1−p) j e−λ λ
iλ
j
(i+ j)!
=e−λpiλ
i
i !(1−p) jλ j
j !
=e−λ p (λ p)i
i !e−λ(1− p) [λ(1−p)] j
j !
=P (X=i)P(Y= j)
X∼Poi (λ p)Y∼Poi (λ(1−p))
∴X⊥Y
![Page 19: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/19.jpg)
Independence ofcontinuous random variables
Two random variables are independent if knowing the value of one tells you nothing about the value of the other (for all values!).
X⊥Y iff ∀ x , y :
f X ,Y (x , y)=f X (x) f Y ( y)- or -
F X ,Y (x , y)=FX (x)FY ( y)- or -
f X ,Y (x , y)=g(x)h( y)
![Page 20: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/20.jpg)
Density functions and independence
f X ,Y (x , y)=6 e−3 x e−2 y for 0<{x , y}<∞X⊥Y : yes!
X⊥Y : yes!
X⊥Y : no!
f X ,Y (x , y)=4 x y for 0<{x , y}<1
f X ,Y (x , y)=4 x y for 0<x<1− y<1
g(x) h(y)
g(x) h(y)
https://bit.ly/1a2ki4G → https://b.socrative.com/login/student/Room: CS109SUMMER17
![Page 21: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/21.jpg)
The Joy of Meetings
P(X+10<Y )+P(Y +10<X )=2 P(X+10<Y )(symmetry)
2 people set up a meeting for 12pm.
Each arrives independently, uniformly between 12:00 and 12:30.
X ~ Uni(0, 30): mins after 12 for person 1Y ~ Uni(0, 30): mins after 12 for person 2
P(first to arrive waits > 10 minutes for the other) = ?
=2 ∬x , y : x+10< y
dx dy f X ,Y (x , y)
(symmetry)
![Page 22: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/22.jpg)
The Joy of Meetings
P(X+10<Y )+P(Y +10<X )=2 P(X+10<Y )
=2 ∬x , y : x+10< y
dx dy f X ,Y (x , y)
=2 ∫y=10
30
dy ∫x=0
y−10
dx ( 130 )
2
=2 ∬x , y : x+10< y
dx dy f X (x) f Y ( y)
=2
302 ∫y=10
30
dy ( y−10)
(symmetry)
(independence)
=2
302 [ y2
2−10 y ]
y=10
30
=2
302 [( 302
2−300)−( 102
2−100)]=4
9
![Page 23: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/23.jpg)
Independence ofcontinuous random variables
Two random variables are independent if knowing the value of one tells you nothing about the value of the other (for all values!).
X⊥Y iff ∀ x , y :
f X ,Y (x , y)=f X (x) f Y ( y)- or -
F X ,Y (x , y)=FX (x)FY ( y)- or -
f X ,Y (x , y)=g(x)h( y)
![Page 24: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/24.jpg)
Setting records
Let X₁, X₂, … be a sequence ofindependent and identically distributed (I.I.D.) continuous random variables.
“record value”: an Xₙ that beats all previous Xᵢ ⇒ Xₙ = max(X₁, …, Xₙ)
Aᵢ: event that Xᵢ is a “record value”
An+1⊥An ?
https://bit.ly/1a2ki4G → https://b.socrative.com/login/student/Room: CS109SUMMER17
![Page 25: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/25.jpg)
Independence is symmetricX⊥Y ⇔ Y ⊥X
Let X₁, X₂, … be a sequence ofindependent and identically distributed (I.I.D.) continuous random variables.
“record value”: an Xₙ that beats all previous Xᵢ ⇒ Xₙ = max(X₁, …, Xₙ)
Aᵢ: event that Xᵢ is a “record value”
E⊥F ⇔ F⊥E
An+1⊥An ?An⊥ An+1 ?https://bit.ly/1a2ki4G → https://b.socrative.com/login/student/
Room: CS109SUMMER17
yes!
P(An An+1)
=1n⋅
1n+1
=P (An)P(An+1)
![Page 26: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/26.jpg)
Break time!
![Page 27: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/27.jpg)
Sum of independent binomials
X∼Bin (n , p) Y∼Bin (m, p)
n flips m flipsX: number of headsin first n flips
Y: number of headsin next m flips
X+Y∼Bin (n+m, p)
More generally:
X i∼Bin (ni , p) ⇒ ∑i=1
N
X i∼Bin (∑i=1
N
ni , p)all X i independent
![Page 28: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/28.jpg)
Sum of independent Poissons
X∼Poi (λ1) Y∼Poi (λ2)
λ₁ chips/cookieX: number of chipsin first cookie
Y: number of chipsin second cookie
X+Y∼Poi(λ1+λ2)
More generally:
X i∼Poi(λi) ⇒ ∑i=1
N
X i∼Poi(∑i=1
N
λ i)
λ₂ chips/cookie
all X i independent
![Page 29: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/29.jpg)
Convolution
A convolution is the distribution of the sum of two independent random variables.
f X+Y (a)=∫−∞
∞
dy f X (a− y) f Y ( y)
![Page 30: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/30.jpg)
Dance Dance Convolution
X, Y: independent discrete random variables
P(X+Y=a)=∑y
P(X+Y=a ,Y= y)(law of total probability)
=∑y
P(X=a− y)P(Y= y)
X, Y: independent continuous random variables
f X+Y (a)=∫−∞
∞
dy f X (a− y) f Y ( y)
![Page 31: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/31.jpg)
Blurring a photo
images: Stig Nygaard (left), Daniel Paxton (right)
![Page 32: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/32.jpg)
=∫0
1
dy f X (a− y) f Y ( y)
Sum of independent uniforms
f X+Y (a)=∫−∞
∞
dy f X (a− y) f Y ( y)
X∼Uni (0 ,1) Y∼Uni (0,1)
0 10
1
0 10
1
1
Case 1: if 0 ≤ a ≤ 1, then we need 0 ≤ y ≤ a (for a – y to be in [0, 1])Case 2: if 1 ≤ a ≤ 2, then we need a – 1 ≤ y ≤ 1
={ ∫0
ady⋅1=a 0≤a≤1
∫a−1
1dy⋅1=2−a 1≤a≤2
0 otherwise 0 1 20
1
![Page 33: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/33.jpg)
Sum of independent normals
More generally:
X∼N (μ1 ,σ12) Y∼N (μ2 ,σ2
2)
X+Y∼N (μ1+μ2 ,σ12+σ2
2)
X i∼N (μi ,σi2) ⇒ ∑
i=1
N
X i∼N (∑i=1
N
μi ,∑i=1
N
σ i2)
all X i independent
![Page 34: Will Monroe July 24, 2017 Mehran Sahami and Chris Piech · 2017. 7. 24. · Independent random variables Will Monroe July 24, 2017 with materials by Mehran Sahami and Chris Piech](https://reader035.fdocuments.us/reader035/viewer/2022063020/5fe1749ae411107a63023155/html5/thumbnails/34.jpg)
Virus infections
M∼Bin (50 ,0.1)≈X∼N (5,4.5)
150 computers in a dorm:
50 Macs (each independently infected with probability 0.1)
100 PCs (each independently infected with probability 0.4)
What is P(≥ 40 machines infected)?
M: # infected Macs
P: # infected PCs P∼Bin (100 ,0.4)≈Y∼N (40,24)
P(M+P≥40)≈P(X+Y≥39.5)W=X+Y∼N (5+40, 4.5+24)=N (45,28.5)
P(W≥39.5)=P (W−45√28.5
≥39.5−45√28.5 )≈1−Φ(−1.03)≈0.8485