Normal Distribution, Continuous Analogs,Derived Distributions

EECS 126 Spring 2020

February 6, 2020

Agenda

Announcements

Review

Normal (Gaussian) Distribution

Analogs to Discrete Probability / RVs

Derived Distributions

Announcements

I Lab 1 and Lab 0 Self-Grades are due Friday (2/7).

I HW3 will be released today and is due next Wednesday(2/12).

I These slides will be posted later.

Questions

Let R be equal to the distance from the origin of a point randomlysampled on a unit ball. What is the

I CDF of R?

I PDF of R?

I Expectation of R?

Answers

Let R be the distance from the origin of a point randomly sampledon a unit ball. What is the

I CDF of R?FR(r) = P(R ≤ r) = 3

4π ·43πr

3 = r3.

I PDF of R?fR(r) = d

dr r3 = 3r2.

I Expectation of R?∫ 10 r · 3r2 = 3

4 .

Normal Distribution

The Normal distribution is seen abundantly in nature (e.g. examscores). This can be explained by the Central Limit Theorem(CLT), which we will go over later in the course.

The Normal distribution is parameterized by its mean µ andvariance σ2 (denoted N (µ, σ2)). Its PDF and CDF are

fX (x) =1√

2πσ2e−(x−µ)

2/2σ2

FX (x) = Φ(x), (cannot be expressed in elementary functions)

Proof that it Integrates to 1

We will show this when µ = 0 and σ2 = 1. Idea is to show(∫∞−∞ fX (x) dx

)2= 1.

(∫ ∞−∞

fX (x) dx)2

=(∫ ∞−∞

fX (x) dx)(∫ ∞

−∞fX (y) dy

)=

∫ ∞−∞

∫ ∞−∞

fX (x)fX (y) dy dx

=

∫ ∞−∞

∫ ∞−∞

( 1√2π

e−x2

2

)( 1√2π

e−y2

2

)dy dx

=

∫ ∞−∞

∫ ∞−∞

1

2πe−

x2+y2

2 dy dx

Proof that it Integrates to 1 (cont.)

Key is now to use polar integration: x2 + y2 = r2 anddy dx = r dr dθ.∫ ∞

−∞

∫ ∞−∞

1

2πe−

x2+y2

2 dy dx =

∫ 2π

0

∫ ∞0

1

2πe−

r2

2 r dr dθ

=

∫ ∞0

e−r2

2 r dr

Now letting u = r2

2 so that du = r dr∫ ∞0

e−r2

2 r dr =

∫ ∞0

e−u du =[−e−u

]u=∞u=0

=[−e−∞ − (−e−0)

]= 0− (−1) = 1

Properties of the Normal DistributionI The sum of two independent Normals is Normal. If

X ∼ N(µ1, σ21), Y ∼ N(µ2, σ

22), and Z = X + Y , then

Z ∼ N(µ1 + µ2, σ21 + σ22)

I The sum of two dependent Normals isn’t always Normal.Consider the following example.

X = N(0, 1)

Y =

{X w.p. 1

2

−X w.p. 12

They are both Normal but X + Y is not Normal.

I A Normal multiplied by a constant is Normal. IfX ∼ N(µ, σ2) and Y = aX , then

Y ∼ N(a · µ, a2 · σ2)

Scaling to the Standard Normal

I The properties on the previous slide allow us to convert anyNormal into the standard Normal.

I If X ∼ N(µ, σ2), then

Z =X − µσ

is distributed with Z ∼ N(0, 1).

I Intuition: I got SDs on midterm 1.

Question

Suppose male height is distributed as N (70, 5) and female heightis distributed as N (64, 4). What’s the probability that a randomlychosen male is taller than a randomly chosen female? You mayexpress your answer in terms of Φ, the standard normal CDF.

Answer

Express X ∼ N (70, 5) and Y ∼ N (64, 4). We want to findP(X > Y ).

P(X > Y ) = P(X + (−Y ) > 0)

= P(N (70, 5) + (−N (64, 4)) > 0)

= P(N (70, 5) +N (−1 · 64, (−1)2 · 4)) > 0)

= P(N (6, 9) > 0)

= P(N (0, 9) > −6)

= P(N (0, 1) >−6

3)

= P(N (0, 1) < 2)

So the answer is Φ(2) ≈ 0.977.

Joint PDFs

We know that multiple discrete RVs have a joint PMF. Similarly,multiple continuous RVs have a joint PDF.

I Discrete PMF: pX ,Y (x , y)

P((X ,Y ) ∈ A) =∑

(x ,y)∈A

pX ,Y (x , y)

I Continuous PDF: fX ,Y (x , y)

P((X ,Y ) ∈ A) =

∫AfX ,Y (x , y)dxdy

I Still needs to be non-negative.

I Still needs to integrate to 1.

Joint CDFs

I Both discrete and continuous RVs have the same definitionfor CDFs and joint CDFs:

FX (x) = P(X ≤ x)

FX ,Y (x , y) = P(X ≤ x ,Y ≤ y)

I Continuous RVs:

fX (x) =d

dxFX (x)

fX ,Y (x , y) =∂2

∂x∂yFX ,Y (x , y)

Marginal Probability Density

I Discrete

pX (x) =∑y∈Y

pX ,Y (x , y)

I Continuous

fX (x) =

∫ ∞−∞

fX ,Y (x , y) dy

I fX (x) is still a density, not a probability.

Conditional Probability Density

I Discrete

pX |Y (x | y) =pX ,Y (x , y)

pY (y)

I Continuous

fX |Y (x | y) =fX ,Y (x , y)

fY (y)

I By definition, Multiplication Rule still holds.

Independence

Similar to discrete, 3 equivalent definitions.

I For all x and y ,

fX ,Y (x , y) = fX (x)fY (y)

I For all x and y ,

fX |Y (x | y) = fX (x)

I For all x and y ,

fY |X (y | x) = fY (y)

Bayes RuleI Discrete (simple form)

pX |Y (x | y) =pY |X (y | x)pX (x)

pY (y)

I Discrete (extended form)

pX |Y (x | y) =pY |X (y | x)pX (x)∑

x ′∈X pY |X (y | x ′)pX (x ′)

I Continuous (simple form)

fX |Y (x | y) =fY |X (y | x)fX (x)

fY (y)

I Continuous (extended form)

fX |Y (x | y) =fY |X (y | x)fX (x)∫∞

−∞ fY |X (y | t)fX (t) dt

Conditional Expectation

I Discrete

E [Y | X = x ] =∑y∈Y

y · pY |X (y | x)

I Continuous

E [Y | X = x ] =

∫ ∞−∞

y · fY |X (y | x) dy

Combining Discrete and Continuous RVs

I You can also have discrete and continuous RVs defined jointly.

I Ex. let X be the outcome of a dice roll and Y be Exp(X ).

∀x ∈ {1, · · · , 6}, pX (x) =1

6fY |X (y | x) = xe−xy

Change of Variables / Derived Distributions

I Let X be a discrete RV with P(X = 0) = 12 , P(X = 1) = 1

2 ,and Y = 2X . Write down the distribution of Y .

I Change of variables in PMF:

P(Y = y) = P(2X = y) = P(X =y

2)

I In general, if X is a discrete RV, Y = f (X ),

P(Y = y) = P(f (X ) = y) = P(X ∈ f −1(y))

Change of Variables / Derived DistributionsI Let X ∼ U[0, 1], and Y = 2X . Then is it true that

fY (y) = P(Y = y) = P(2X = y) = P(X =y

2) = fX (

y

2)

I No, this won’t integrate to 1.

∀x ∈ [0, 1], fX (x) = 1∫ 2

0fY (y)dy =

∫ 2

0fX (

y

2)dy = 2

I You have to use the CDF.

FY (y) = P(Y ≤ y) = P(2X ≤ y) = P(X ≤ y

2) = FX (

y

2)

I Taking the derivative on both sides,

fY (y) =d

dyFX (

y

2) =

1

2· fX (

y

2).

Change of Variables / Derived Distributions

I Let X ∼ U[0, 1], and Y = 2X . Then is it true that

fY (y) = P(Y = y) = P(2X = y) = P(X =y

2) = fX (

y

2)

I Note: PDF does not satisfy fY (y) = P(Y = y).

fY (y)dy = P(y ≤ Y ≤ y + dy)

= P(y ≤ 2X ≤ y + dy)

= P(y

2≤ X ≤ y

2+

dy

2)

= fX (y

2) · dy

2

This implies fY (y) = 12 · fX ( y2 ).

Change of Variables / Derived Distributions

I In general, if X is continuous RV, Y = g(X ). Then we have

FY (y) = P(Y ≤ y) = P(g(X ) ≤ y).

I When g is monotone increasing,

P(g(X ) ≤ y) = P(X ≤ g−1(y)) = FX (g−1(y)) (1)

I When g is monotone decreasing,

P(g(X ) ≤ y) = P(X ≥ g−1(y)) = 1− FX (g−1(y)) (2)

References

Introduction to probability. DP Bertsekas, JN Tsitsiklis - 2002