Exam #2 Review - Western Michigan Universitybazuinb/ECE3800/Exam2_Review.pdf · Exam #2 Review What...

Notes and figures are based on or taken from materials in the course textbook: Charles Boncelet, Probability, Statistics, and Random Signals, Oxford University Press, February 2016.

B.J. Bazuin, Spring 2020 1 of 48 ECE 3800

Exam #2 Review

What is on an exam? Read through the homework and class examples…

4-5 multipart questions. Points assigned based on complexity. (4Q, 120 pts. Sp 2018, 4Q, 120 pts. Sp 2019, 4Q, 125 pts. Fa 2019)

Multiple Discrete & Continuous Random Variables Binomial Probabilities Continuous Probability Expected Value Operator Moments and Variance Failure rate related questions It will not contain statistics or confidence interval related questions that have not been covered yet this semester. Our textbook and notes with a focus on Chap. 5-8.

Previous homework problem solutions as examples – Dr. Severance’s Skill Examples

Skills #1-5

Exam and homework like problems:

HW#5

HW#6

HW#7

Read through previous exams … 2018 and 2019 exams are relevant!

And now for a quick chapter review … the important information without the rest!



The Chapter Content: 5-8

5 Multiple Discrete random Variables 5.1 Multiple Random Variables and PMFs 5.2 Independence 5.3 Moments and Expected Values

5.3.1 Expected Values for Two Random Variables 5.3.2 Moments for Two Random Variables

5.4 Example: Two Discrete Random Variables 5.4.1 Marginal PMFs and Expected Values 5.4.2 Independence 5.4.3 Joint CDF 5.4.4 Transformations With One Output 5.4.5 Transformations With Several Outputs 5.4.6 Discussion

5.5 Sums of Independent Random Variables 5.6 Sample Probabilities, Mean, and Variance 5.7 Histograms 5.8 Entropy and Data Compression

5.8.1 Entropy and Information Theory 5.8.2 Variable Length Coding 5.8.3 Encoding Binary Sequences 5.8.4 Maximum Entropy

6 Binomial Probability 6.1 Basics of the Binomial Distribution 6.2 Computing Binomial Probabilities 6.3 Moments of the Binomial Distribution 6.4 Sums of Independent Binomial Random Variables

6.5 Distributions Related to the Binomial 6.5.1 Connections Between Binomial and Hypergeometric Probabilities 6.5.2 Multinomial Probabilities 6.5.3 The Negative Binomial Distribution 6.5.4 The Poisson Distribution

6.6 Parameter Estimation for Binomial and Multinomial Distributions 6.7 Alohanet 6.8 Error Control Codes

6.8.1 Repetition-by-Three Code 6.8.2 General Linear Block Codes 6.8.3 Conclusions



7 A Continuous random Variable 7.1 A Continuous Random Variable and Its Density, Distribution Function, and Expected

Values 7.2 Example Calculations for a Single Random Variable 7.3 Selected Continuous Distributions

7.3.1 The Uniform Distribution 7.3.2 The Exponential Distribution

7.4 Conditional Probabilities for a Continuous Random Variable 7.5 Discrete PMFs and Delta Functions 7.6 Quantization 7.7 A Final Word

8 Multiple Continuous Random Variables 8.1 Joint Densities and Distribution Functions 8.2 Expected Values and Moments 8.3 Independence 8.4 Conditional Probabilities for Multiple Random Variables 8.5 Extended Example: Two Continuous Random Variables 8.6 Sums of Independent Random Variables 8.7 Random Sums 8.8 General Transformations and the Jacobian 8.9 Parameter Estimation for the Exponential Distribution 8.10 Comparison of Discrete and Continuous Distributions

Necessary considerations:

Joint density functions of multiple R.V. (discrete or continuous) Compute marginal densities, means, 2nd moments, variances, covariance, and correlation coefficient.

Continuous R.V. The pdf, CDF, probability computations and conditional probability. Properties of the exponential pdf.

One-to-one mapping/translation of one R.V. into another R.V. Where

𝑌 𝑚 ∙ 𝑋 𝑏 and 𝑓 𝑦| |

∙ 𝑓

Multiple independent random variables sum or difference: 𝑍 𝑎 ∙ 𝑋 𝑏 ∙ 𝑌



Chapter 5: Multiple Discrete Random Variables

The joint probability mass function 𝑝 𝑘, 𝑙 𝑃𝑟 𝑋 𝑥 ∩ 𝑌 𝑦

Properties of the joint pmf

1. 𝑝 𝑘, 𝑙 𝑃𝑟 𝑋 𝑥 ∩ 𝑌 𝑦 0 (all probabilities are positive)

2. The summation of the pmf for all k is equal to 1.

𝑝 𝑘, 𝑙 1.0

The marginal pmf of the individual random variables can be computed

𝑝 𝑘, 𝑙 𝑝 𝑙

𝑝 𝑘, 𝑙 𝑝 𝑘

The joint Cumulative Distribution Function exists

𝐹 𝑢, 𝑣 𝑃𝑟 𝑋 𝑢 ∩ 𝑌 𝑣

Properties 0 𝐹 𝑢, 𝑣 1, 𝑓𝑜𝑟 ∞ 𝑢 ∞ 𝑎𝑛𝑑 ∞ 𝑣 ∞

𝐹 𝑢 𝑃𝑟 𝑋 𝑢 𝑃𝑟 𝑋 𝑢 ∩ 𝑌 ∞ 𝐹 𝑢, ∞

𝐹 𝑢𝑣 𝑃𝑟 𝑌 𝑣 𝑃𝑟 𝑋 ∞ ∩ 𝑌 𝑣 𝐹 ∞, 𝑣

Calculating the probability of an “area”

𝑃𝑟 𝑎 𝑋 𝑏 ∩ 𝑐 𝑌 𝑑 𝐹 𝑏, 𝑑 𝐹 𝑏, 𝑐 𝐹 𝑎, 𝑑 𝐹 𝑎, 𝑐

Conditional Probability

Let 𝐴 𝑋 𝑥 and 𝐵 𝑌 𝑦

𝑃𝑟 𝑋 𝑥 |𝑌 𝑦 𝑃𝑟 𝐴|𝐵𝑃𝑟 𝐴, 𝐵

𝑃𝑟 𝐵

𝑃𝑟 𝑋 𝑥 |𝑌 𝑦 𝑃𝑟 𝐴|𝐵𝑃𝑟 𝑋 𝑥 ∩ 𝑌 𝑦

𝑃𝑟 𝑌 𝑦



Independence

If X and Y are independent 𝑝 𝑘, 𝑙 𝑃𝑟 𝑋 𝑥 ∩ 𝑌 𝑦 𝑃𝑟 𝑋 𝑥 ∙ 𝑃𝑟 𝑌 𝑦 𝑝 𝑘 ∙ 𝑝 𝑙

and 𝐹 𝑢, 𝑣 𝑃𝑟 𝑋 𝑢 ∩ 𝑌 𝑣 𝑃𝑟 𝑋 𝑢 ∙ 𝑃𝑟 𝑌 𝑣 𝐹 𝑢 ∙ 𝐹 𝑣

For three or more random variables,

1) Each pair is independent

2) The joint pmf of all three factors for all outcomes is independent

Useful terminology and concept: Independent and Identically Distributed (IID)

For this case, the R.V. are independent and all have the same pmf!

Moments and Expected Values

𝐸 𝑔 𝑋, 𝑌 𝑔 𝑥 , 𝑦 ∙ 𝑝 𝑘, 𝑙

Property Additive

𝐸 𝑔 𝑋, 𝑌 𝑔 𝑋, 𝑌 𝑔 𝑥 , 𝑦 𝑔 𝑥 , 𝑦 ∙ 𝑝 𝑘, 𝑙

𝐸 𝑔 𝑋, 𝑌 𝑔 𝑋, 𝑌 𝑔 𝑥 , 𝑦 ∙ 𝑝 𝑘, 𝑙 𝑔 𝑥 , 𝑦 ∙ 𝑝 𝑘, 𝑙

𝐸 𝑔 𝑋, 𝑌 𝑔 𝑋, 𝑌 𝐸 𝑔 𝑋, 𝑌 𝐸 𝑔 𝑋, 𝑌

Multiplicative if and only if X and Y independent

𝐸 𝑋 ∙ 𝑌 𝑋 ∙ 𝑌 ∙ 𝑝 𝑘, 𝑙 𝑋 ∙ 𝑌 ∙ 𝑝 𝑘 ∙ 𝑝 𝑙

𝐸 𝑋 ∙ 𝑌 𝑌 ∙ 𝑝 𝑙 ∙ 𝑋 ∙ 𝑝 𝑘 𝑌 ∙ 𝑝 𝑙 ∙ 𝐸 𝑋

𝐸 𝑋 ∙ 𝑌 𝐸 𝑋 ∙ 𝑌 ∙ 𝑝 𝑙 𝐸 𝑋 ∙ 𝐸 𝑌



and in general

𝐸 𝑔 𝑋 ∙ 𝑔 𝑌 𝐸 𝑔 𝑋 ∙ 𝐸 𝑔 𝑌

Not that if X and Y are not independent, they may be said to be correlated and

𝑟 𝐸 𝑋 ∙ 𝑌 𝐸 𝑋 ∙ 𝐸 𝑌

Covariance and Correlation Coefficients

This gives rise to another term Covariance 𝜎 𝐶𝑜𝑣 𝑋, 𝑌 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇

also 𝐶𝑜𝑣 𝑋, 𝑌 𝐸 𝑋 ∙ 𝑌 𝜇 ∙ 𝑌 𝑋 ∙ 𝜇 𝜇 ∙ 𝜇

𝐶𝑜𝑣 𝑋, 𝑌 𝐸 𝑋 ∙ 𝑌 𝐸 𝜇 ∙ 𝑌 𝐸 𝑋 ∙ 𝜇 𝐸 𝜇 ∙ 𝜇

𝐶𝑜𝑣 𝑋, 𝑌 𝐸 𝑋 ∙ 𝑌 𝜇 ∙ 𝜇 𝜇 ∙ 𝜇 𝜇 ∙ 𝜇

𝐶𝑜𝑣 𝑋, 𝑌 𝐸 𝑋 ∙ 𝑌 𝜇 ∙ 𝜇

or

𝜎 𝐶𝑜𝑣 𝑋 ∙ 𝑌 𝑟 𝜇 ∙ 𝜇

Note that if X and Y are independent 𝑟 𝜇 ∙ 𝜇

and 𝜎 𝐶𝑜𝑣 𝑋 ∙ 𝑌 0

As more notation 𝑟 𝐸 𝑋

𝐶𝑜𝑣 𝑋 ∙ 𝑋 𝜎 𝜎

Correlation Coefficient

Letting 𝑍 𝑋 𝑌

The mean value

𝜇 𝐸 𝑍 𝐸 𝑋 𝑌 𝐸 𝑋 𝐸 𝑌 𝜇 𝜇

The variance



𝜎 𝐸 𝑍 𝜇 𝐸 𝑋 𝜇 𝑌 𝜇

𝜎 𝐸 𝑋 𝜇 2 ∙ 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇 𝐸 𝑌 𝜇

𝜎 𝜎 2 ∙ 𝐶𝑜𝑣 𝑋, 𝑌 𝜎

𝜎 𝜎 2 ∙ 𝜎 𝜎

Thinking in terms of a product function we define

𝜎 𝜎 2 ∙ 𝜌 ∙ 𝜎 ∙ 𝜎 𝜎

With the correlation coefficient defined as

𝜌𝜎

𝜎 ∙ 𝜎

or

𝜌𝐸 𝑋 ∙ 𝑌 𝜇 ∙ 𝜇

𝜎 ∙ 𝜎

As a result of the “normalized scaling” we expect

1 𝜌𝜎

𝜎 ∙ 𝜎1

A special note

Independent random variables are uncorrelated.

𝐸 𝑋 ∙ 𝑌 𝐸 𝑋 ∙ 𝐸 𝑌 𝜇 ∙ 𝜇


𝜎 ∙ 𝜎

𝜌𝜇 ∙ 𝜇 𝜇 ∙ 𝜇

𝜎 ∙ 𝜎0

However, uncorrelated random variables are not necessarily independent!



5.5 Sums of Independent Random Variables


The mean value 𝜇 𝐸 𝑍 𝐸 𝑋 𝑌 𝐸 𝑋 𝐸 𝑌 𝜇 𝜇

The variance 𝜎 𝐸 𝑍 𝜇 𝐸 𝑋 𝜇 𝑌 𝜇



𝜎 𝜎 2 ∙ 0 𝜎

𝜎 𝜎 𝜎

Letting 𝑆 𝑋 𝑋 𝑋

The mean value 𝜇 𝐸 𝑆 𝐸 𝑋 𝑋 𝑋 𝐸 𝑋 𝐸 𝑋 𝐸 𝑋 𝜇 𝜇 𝜇

The variance

Letting 𝑆 𝑋 𝑋 𝑋 𝑍 𝑋

𝜎 𝜎 𝜎

𝜎 𝜎 𝜎 𝜎


The pmf 𝑝 𝑛 𝑃𝑟 𝑋 𝑌 𝑛

from total probability

𝑝 𝑛 𝑃𝑟 𝑋 𝑌 𝑛|𝑌 𝑙 ∙ 𝑃𝑟 𝑌 𝑙

But this is equivalent to



𝑝 𝑛 𝑃𝑟 𝑋 𝑛 𝑙|𝑌 𝑙 ∙ 𝑃𝑟 𝑌 𝑙

With independence a joint probability is the product of probabilities; therefore,

𝑝 𝑛 𝑃𝑟 𝑋 𝑛 𝑙 ∙ 𝑃𝑟 𝑌 𝑙

Resulting in

𝑝 𝑛 𝑝 𝑛 𝑙 ∙ 𝑝 𝑙

This is the discrete convolution of the two pmf functions! 𝑝 𝑝 ∗ 𝑝

and 𝑝 𝑝 ∗ 𝑝 ∗ 𝑝

Textbook Moment Generating Function of two ind. R.V.

From Laplace … convolution in the time domain is multiplication in the Laplace domain.

𝑍 𝑋 𝑌

𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑍

𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 𝑌

𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 ∙ 𝑒𝑥𝑝 𝑠 ∙ 𝑌

𝑀 𝑠 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑋 ∙ 𝐸 𝑒𝑥𝑝 𝑠 ∙ 𝑌

𝑀 𝑠 𝑀 𝑠 ∙ 𝑀 𝑠

For the sum of independent R.V. , the MGF is the product of the MGF!



Sampling Theory – The Sample Mean

Definitions

Population: the collection of data being studied N is the size of the population

Sample: a random sample is the part of the population selected all members of the population must be equally likely to be selected! n is the size of the sample

Sample Mean: the average of the numerical values that make of the sample

Population: N

Sample set: 𝑆 ∈ 𝑥 , 𝑥 , 𝑥 , ⋯ , 𝑥

Sample Mean �̅� ∙ ∑ 𝑥

To generalize, describe the statistical properties of arbitrary random samples rather than those of any particular sample.

Sample Mean 𝑋 ∙ ∑ 𝑋 , where iX are random variables with a pdf.

Notice that for a pdf, the true mean, X , can be compute while for a sample data set the above

sample mean, is computed.

As may be noted, the sample mean is a combination of random variables and, therefore, can also be considered a random variable. As a result, the hoped for result can be derived as:

𝐸 𝑋 𝜇 𝐸1𝑁

∙ 𝑋1𝑁

∙ 𝐸 𝑋1𝑁

∙ 𝑋 𝑋 𝜇

If and when this is true, the estimate is said to be an unbiased estimate.

Though the sample mean may be unbiased, the sample mean may still not provide a good estimate.

X̂



Variance of the sample mean – (the mean itself, not the value of X)

You would expect the sample mean to have some variance about the “probabilistic” or actual mean; therefore, it is also desirable to know something about the fluctuations around the mean. As a result, computation of the variance of the sample mean is desired.

For N>>n or N infinity (or even a known pdf), using the collected samples … based on the prior definition of variance, a statistical estimate of the 2nd moment and the square of the mean.

22

1

ˆ1ˆ XEXn

EXVarn

ii

211

2

1ˆ XXXn

EXVarn

jj

n

ii

21 1

2

1ˆ XXXn

EXVarn

i

n

jji

21 1

2

1ˆ XXXEn

XVarn

i

n

j

ji

For iX independent (measurements should be independent of each other)

jiforXXEXEXE

jiforXXEXXE

ji

ii

ji

,ˆ

,

22

22

As a result we can define two summation where i=j and i<>j,

21 ,1

2

1ˆ XXXEXXEn

XVarn

i

n

ijj

jiii

2222

2

1ˆ XXEnnXEnn

XVar ii

22

2

221ˆ XX

n

nnX

nXVar

nn

XXX

n

nX

nXVar

2222

221ˆ

where 2 is the true variance (probabilistic) of the random variable, X.



Therefore, as n approaches infinity, this variance in the sample mean estimate goes to zero! It is referred to as a “consistent” estimate. Thus a larger sample size leads to a better estimate of the population mean.

Note: this variance is developed based on “sampling with replacement”.

Sampling Theory – The Sample Variance

When dealing with probability, both the mean and variance provide valuable information about the “DC” and “AC” operating conditions (about what value is expected) and the variance (in terms of power or squared value) about the operating point.

Therefore, we are also interested in the sample variance as compared to the true data variance.

The sample variance of the population (stdevp) is defined as:

n

i

i XXn

S

1

22 ˆ1

and continuing until (shown in the coming pages)

22 1

n

nSE

where is the true variance of the random variable.

Note: the sample variance is not equal to the true variance; it is a biased estimate!

To create an unbiased estimator, scale by the biasing factor to compute (stdev):

n

ii

n

iix XX

nXX

nn

nSE

n

nSE

1

2

1

2222 ˆ

1

1ˆ1

11

~

This is equation 5.12 in the textbook!



Statistical Mean and Variance Summary

For taking samples and estimating the mean and variance …

The Estimate Variance of Estimate

Mean

n

iiX X

nX

1

1ˆ̂

An unbiased estimate

XEXE ˆ

XX ˆ

n

XVar X2

ˆ

Variance (biased)

n

i

i XXn

S

1

22 ˆ1

A biased estimate

22 1Xn

nSE

2

442

1

~

n

nSVar X

44 XXE

Variance (unbiased) 222

1

~XSE

n

nSE

An unbiased estimate

22~XXESE

222 ˆ~

XXSE

n

SVar X4

42

44 XXE



Homework Problem 5.30:

Prove the Cauchy-Schwarz inequality:

𝑥 ∙ 𝑦 𝑥 ∙ 𝑦

where the x’s and y’s are arbitrary numbers.

Hint: Start with the following inequality (why is this true?):

0 𝑥 𝑎 ∙ 𝑦

Find the value of a that minimizes the right hand side above, substitute that value into the same inequality, and rearrange the terms into the Cauchy-Schwarz inequality at the top.



𝑥 ∙ 𝑦 𝑥 ∙ 𝑦

or

0 𝑥 ∙ 𝑦 𝑥 ∙ 𝑦

You may have heard the phrase,” The square of the sum of the product is less than or equal to the product of the sums of the squares!



Basics of the Binomial Distribution

The binomial distribution arises from the summation of Independent and Identically Distributed (IID) Bernoulli R.V.

𝑝 𝑘1 𝑝 𝑞, 𝑋 0, 𝑘 0

𝑝, 𝑋 1, 𝑘 1

Then, 𝑆 𝑋 𝑋 𝑋 ⋯ 𝑋

𝑝 𝑆 𝑘 𝑃𝑟 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒𝑠 𝑤𝑖𝑡ℎ 𝑘 1 𝑠 𝑎𝑛𝑑 𝑛 𝑘 0′𝑠. " "

As described previously, the number of sequences with exactly k ones in N trials is based on the combinatorial computation of the binomial coefficient.

𝑝 𝑆 𝑘 𝑝 𝑘𝑛𝑘 ∙ 𝑝 ∙ 𝑞

The total summation (which must equal one) can be performed as

𝑝 𝑆 𝑘𝑛𝑘 ∙ 𝑝 ∙ 𝑞

From previous examples it was demonstrated that

𝑝 𝑆 𝑘𝑛𝑘 ∙ 𝑝 ∙ 𝑞 𝑝 𝑞

𝑝 𝑆 𝑘 𝑝 𝑞 1 1

Determine the expected value

n

xX nPxXE

0

n

x

xnx ppx

nxXE

0

1

pnXE

Determine the 2nd moment

0

22

xX nPxXE

pnpnnXE 22 1



Determine the variance

222 XEXEXE

222 1 pnpnpnnXE

pnpnpnpnpnpnXE 2222222

qpnppnXE 12

Additive relationship of binomial

From the combinatorial analysis operations shown in Chapter 3 …. 𝑎 𝑏 𝑎 𝑏 ∙ 𝑎 𝑏

with

𝑎 𝑏𝑛𝑘 ∙ 𝑎 ∙ 𝑏

The incorporation of “another element” or coin can be derived from the previous number of elements and the next being one or the other of the possible trial outcomes. This derivation can use conditional probability for the “next coin” …

Defining

𝑏 𝑛. 𝑘, 𝑝 𝑝 𝑆 𝑘 𝑝 𝑘𝑛𝑘 ∙ 𝑝 ∙ 𝑞

𝑃𝑟 𝑆 𝑘 𝑃𝑟 𝑆 𝑘|𝑋 1 ∙ 𝑃𝑟 𝑋 1 𝑃𝑟 𝑆 𝑘|𝑋 0 ∙ 𝑃𝑟 𝑋 0

which can also be considered from the “previous” trial as

𝑃𝑟 𝑆 𝑘 𝑃𝑟 𝑆 𝑘 1|𝑋 1 ∙ 𝑝 𝑃𝑟 𝑆 𝑘|𝑋 0 ∙ 𝑞

but the previous result is not based on the nth result, so that

𝑃𝑟 𝑆 𝑘 𝑃𝑟 𝑆 𝑘 1 ∙ 𝑝 𝑃𝑟 𝑆 𝑘 ∙ 𝑞

or using the b notation

𝑏 𝑛. 𝑘, 𝑝 𝑏 𝑛 1. 𝑘 1, 𝑝 ∙ 𝑝 𝑏 𝑛 1. 𝑘, 𝑝 ∙ 𝑞

This is related to the “Matlab” homework problem for forming Pascal’s triangle! HW 3.5



6.4 Sums of Independent Binomial R.V.

The sum of binomial R.V. give1n equivalent values of p is binomial.

Basis for N trials which is a sum of the two R.V. trials

𝑁 𝑁 𝑁

As the sum of independent random variables, the MGF can be used

𝑀 𝑠 𝑀 𝑠 ∙ 𝑀 𝑠

but the MGF is

𝑀 𝑠 𝑝 ∙ 𝑒𝑥𝑝 𝑠 𝑞

For the sum this is equivalent to

𝑀 𝑠 𝑀 𝑠 ∙ 𝑀 𝑠 𝑝 ∙ 𝑒𝑥𝑝 𝑠 𝑞 ∙ 𝑝 ∙ 𝑒𝑥𝑝 𝑠 𝑞

𝑀 𝑠 𝑀 𝑠 ∙ 𝑀 𝑠 𝑝 ∙ 𝑒𝑥𝑝 𝑠 𝑞

Which is the desired result for a single binomial R.V. with 𝑁 𝑁 𝑁



6.5.3 The Negative Binomial Distribution

The binomial distribution is based on “In n independent flips, how many heads can one expect?”

The negative binomial distribution is based on “to get k heads, how many independent flips, n, are needed?”

To “build up to the kth head on the nth flip, the k-1 head must have occurred in n-1 flips. Therefore, the prior number is a binomial probability in n-1 and k-1 and the final flip is based on

𝑃𝑟 𝑁 𝑛, 𝑘 𝑝 ∙ 𝑛 1𝑘 1

∙ 𝑝 ∙ 𝑞 , 𝑓𝑜𝑟 𝑛 𝑘

𝑃𝑟 𝑁 𝑛 𝑛 1𝑘 1

∙ 𝑝 ∙ 𝑞 , 𝑓𝑜𝑟 𝑛 𝑘

As a note, the negative binomial can be thought of as the sum of k geometric random variables (where the geometric RV is thought of as a sequence of tails followed by a head and one negative binomial is the summation of k geometric R.V. )

If this relation holds, the mean and variance can be computed from the mean and variance of the geometric sequence!

Geometric R.V.

𝐸 𝑋 𝜇1𝑝

𝑉𝑎𝑟 𝑋 𝜎1 𝑝

𝑝

Sum of IID

𝑍 𝑋

𝐸 𝑍 𝜇 𝑘 ∙1𝑝

𝑉𝑎𝑟 𝑍 𝜎 𝜎 𝑘 ∙ 𝜎 𝑘 ∙1 𝑝

𝑝



6.6 Binomial Estimation

In experimental cases when success and failure or positive and negative results can occur based on assumed to be independent trials, the statistic can provide parameters to model (and simulate) the underlying probabilistic experiments.

Binomial equation estimation:

The mean and the variance of the binomial, if estimated provide p!

𝐸 𝑆 𝜇 𝑛 ∙ 𝑝

𝑉𝑎𝑟 𝑆 𝜎 𝑛 ∙ 𝑝 ∙ 𝑝 1 𝑛 ∙ 𝑝 ∙ 𝑞

For an experiment with n trials, we observe that k successes occurred.

Let �̂�

Under our assumption

𝐸 �̂�𝑘𝑛

𝐸 𝑆𝑛

𝑛 ∙ 𝑝𝑛

𝑝

This is an unbiased estimate.

The variance of the estimate can be computed as

𝑉𝑎𝑟 �̂� 𝑉𝑎𝑟𝑆𝑛

𝑉𝑎𝑟 𝑆𝑛

𝑛 ∙ 𝑝 ∙ 𝑞𝑛

𝑝 ∙ 𝑞𝑛

As the variance in the estimate goes to zero as n increases, it is also a consist estimator.

Unbiased: the expected value of the estimator equals the value being estimated.

Consistent: the variance in the estimator goes to zero as n goes to infinity.

For many discrete probability models, the estimation of the mean or mean and variance provide all the information required to define the pmf to be used in the model.



Chapter 7: CONTINUOUS RANDOM VARIABLES

7.1 Basic Properties

A random variable is continuous if Pr[X=x] = 0.

Strange statement, but for continuous random variables, there are an infinite number of points and any value over infinity is zero!

The continuous random variable has a probability over an interval, Pr[a<X≤b]!

Note that in performing and experiment or trial, the result takes on a specific value.

For continuous probability …

It may be best to start with the continuous Cumulative Distribution Function.

Cumulative Distribution Function (CDF):The probability of the event that the observed random variable X is less than or equal to the allowed value x.

xXxFX Pr

The defined function can be discrete or continuous along the x-axis. Constraints on the cumulative distribution function are:

xforxFX ,10

0XF and 1XF

XF is non-decreasing as x increases

1221Pr xFxFxXx XX

The final point defines the probability over a region! Alternately

𝑃𝑟 𝑎 𝑋 𝑏 𝐹 𝑏 𝐹 𝑎

A big difference between continuous and discrete CDFs is:



For continuous R.V. all the following are equal

𝐹 𝑏 𝐹 𝑎 𝑃𝑟 𝑎 𝑋 𝑏 𝑃𝑟 𝑎 𝑋 𝑏

𝐹 𝑏 𝐹 𝑎 𝑃𝑟 𝑎 𝑋 𝑏 𝑃𝑟 𝑎 𝑋 𝑏

For a and b integers, they may not be equal for discrete R.V.! The discrete CDF has step discontinuities at the values taken on by the R.V. There where an integer is included or excluded is significant!

Therefore, we usually use: 𝑃𝑟 𝑎 𝑋 𝑏 𝐹 𝑏 𝐹 𝑎

This is correct and appropriate for both continuous and discrete R.V. (This is also a trick question for homework or exams.)

Properties of the CDF, xFX

i. 0XF and 1XF

ii. XF is non-decreasing as x increases. For 2121 xFxFxx XX For continuous R.V. we describe “monotonically” increasing.

iii. xFX is continuous from the right, that is,

0,lim0

xFxF XX

Derivable concepts

xforxFX ,10

1221 xFxFxXxP XX

Opened and close interval derivations: (note these are needed for discrete probability)

11221 xXPxFxFxXxP XX

21221 xXPxFxFxXxP XX

211221 xXPxXPxFxFxXxP XX



Waiting for a bus

A bus arrives at random in the interval T,0 . For the random variable related to the bus arrival, the bus is equally likely of coming at any time during the interval (uniformly distributed). Then

tT

TtT

t

t

tFX

,1

0,

0,0

Figure 2.3-2 Cumulative distribution function of the uniform random variable X of Example 2.3-2.

Notes and figures are based on or taken from materials in the course textbook: Probability, Statistics and Random Processes for Engineers, 4th ed., Henry Stark and John W. Woods, Pearson Education, Inc., 2012.

With the pdf

𝑓 𝑥

0, 𝑡 01𝑇

, 0 𝑥 𝑇

0, 𝑇 𝑥



Probability Density Function (the pdf)

The derivative of the cumulative distribution function

dx

xdFxFxFxf XXX

X

0lim

Assumption … the CDF is continuous and differentiable.

Properties of the pdf, if it exists, include

1. xforxf X ,0

2. 1

XXX FFdxxf

3. xXPduufxFx

XX

4. 1221

2

1

Pr xFxFdxxfxXx XX

x

x

X

From: http://en.wikipedia.org/wiki/Probability_density_function

In mathematics, a probability density function (pdf) serves to represent a probability distribution in terms of integrals. A probability density function is everywhere non-negative and its integral from −∞ to +∞ is equal to 1. If a probability distribution has density f(x), then intuitively the infinitesimal interval [x, x + dx] has probability f(x) dx.

An interpretation is

xFdxxFdxxXxPdxxf XXX

This helps with discrete functions or mixed continuous and discrete ….

Observe that if xf X exists, meaning that it is bounded and has at most a finite number of

discontinuities, then xFX is continuous and therefore 0 xXP (except at the discontinuities).

The pdf to CDF relationships are based on calculus I and II … integration and differentiation.

𝑃𝑟 𝑎 𝑋 𝑏 𝐹 𝑏 𝐹 𝑎 𝑓 𝑥 ∙ 𝑑𝑥



Expected Value of a Continuous Random Variable

Mean Value: the expected mean value of measurements of a process involving a random variable.

This is commonly called the expectation operator or expected value of … and is mathematically described as:

dxxfxXEX X

For laboratory experiments, the expected value of a voltage measurement can be thought of as the DC voltage.

General concept of an expected value

In general, the expected value of a function is:

𝐸 𝑔 𝑋 𝑔 𝑥 ∙ 𝑓 𝑥 ∙ 𝑑𝑥

For continuous R.V. we integrate instead of sum.

Estimating a parameter:

If we know the expected value, you have a simple estimate of future expected outcomes.

XEXx ˆ

Or for xgy

XgEyEy ˆ

Moments

The moments of a random variable are defined as the expected value of the powers of the measured output or …

dxxfxXEX Xnnn

Therefore, the mean or average is sometimes called the first moment.



Expected Mean Squared Value or Second Moment

The mean square value or second moment is

dxxfxXEX X222

The second moment is related to the average “energy” or “power” in a signal, where the energy and power are defined as

dttxdttxE

T

TT

x

22

2

2lim

2

2

21lim

T

TT

x dttxT

P

Central Moments and Variance

The central moments are the moments of the difference between a random variable and its mean.

dxxfXxXXEXX X

nnn

The second central moment is referred to as the variance of the random variable …

dxxfXxXXEXX X

2222

Note that the variance may also be computed as: 222 XX



First Look describing “named” random variables

Note: there are documents describing specific discrete and continuous pmf, pdf and PDF or CDF on the password web site.

Notes and figures are based on or taken from materials in the course textbook: Probability, Statistics and Random Processes for Engineers, 4th ed., Henry Stark and John W. Woods, Pearson Education, Inc., 2012

Notes and figures are based on or taken from materials in the course textbook: Probability, Statistics and Random Processes for Engineers, 4th ed., Henry Stark and John W. Woods, Pearson Education, Inc., 2012



Functions of random variables

In engineering analysis, many times one random variable is a function of a second random variable, for example,

random power derived from a random voltage 2XY circular area derived from a random measurement of the diameter 2XY DC voltage measurement in the presence of R.V. noise XaY linear relationships bXmY

Think of XgY

… what can be described for the probability density functions (pdf) of Y and X?

Since the PDF is the integral of the pdf, we should have:

dyyfdxxf YX

For XgY a monotonically increasing function of X, the new pdf should be related to the previous pdf as something like:

dy

dxxfyf XY

For XgY a monotonically decreasing function of X, the new pdf must be increasing. As a result the new pdf should be related to the previous pdf as:

dy

dxxfyf XY

The generalization of this concept becomes:

Letting bXmY

For m positive

m

byXybXmyY

m

byFyF XY

m

byf

mm

by

dy

d

m

byf

dy

dx

m

byxfyf XXXY

1



What if m is negative? (not the inequality sign changes when multiplying by a negative)

m

byXybXmyY

For the probabilities,

m

byX

m

byXybXmyY Pr1PrPrPr

For a continuous distribution (notice that there may have to be corrections for discrete).

m

byX

m

byXybXmyY Pr1PrPrPr

Using the CDFs

m

byFyF XY 1

Taking a derivative to find the density functions and recognizing that m is negative.

m

byf

mm

byf

mm

by

dy

d

m

byfyf XXXY

11

Therefore, for all linear cases …

m

byf

myf XY

1

But it also works for other cases when there is a one-to-one relationship between X and Y!



General Formula of Determining the pdf of Y = g(X)

dyyfdyyYyP Y

For xgy has a finite number of real roots (multiple solutions), then the disjoint events (related to each of the roots) have the form

ygxi1 are related to the events

iiii dxxXxE or iiiii dxxXdxxE

If we then accumulate all of them we can find the probability

i

iiXY dxxfdyyfdyyYyP

By “engineering manipulation” (divide by dy)

i

iiX

i

iiXY dy

dxxf

dy

dxxfyf

Which can also be perform in terms of the functional derivative xgxgdx

d

dx

dy'

i i

iX

i iiXY xg

xf

dx

dyxfyf

'

1

, for ygxi1

A figure of the multiple solutions and dx segments follows.

Figure 3.2-15 The event {y < Y ≤ y + dy} is the union of two disjoint events on the probability

space of X.



7.4 Conditional Probabilities

The concepts of conditional probability apply to continuous probability as well.

Using the Cumulative Distribution Function (CDF), define X as a continuous R.V. and let A define some a-priori information about X.

𝐹 𝑥|𝐴 𝑃𝑟 𝑋 𝑥|𝐴𝑃𝑟 𝑋 𝑥 ∩ 𝐴

𝑃𝑟 𝐴

and taking the derivative

𝑓 𝑥|𝐴𝑑

𝑑𝑥𝐹 𝑥|𝐴

𝑑𝑑𝑥 𝑃𝑟 𝑋 𝑥 ∩ 𝐴

𝑃𝑟 𝐴

The event A that conditions the probability has several possibilities:

1. Every A may be an event that can be expressed in terms of the random variable X. Which means we have a simple conditional probability.

2. Every A may be an event that depends upon some other random variable, which may be either continuous or discrete. A joint probability or independent Prob.

3. Every A may be an event that depends upon both the random variable X and some other random variable. Even more complicated.

For our purposes (#1 above) envision that 𝐴 𝑋 𝑥

The conditional probability could then be rewritten as

𝐹 𝑥|𝑋 𝑥 𝑃𝑟 𝑋 𝑥|𝑋 𝑥𝑃𝑟 𝑋 𝑥 ∩ 𝑋 𝑥

𝑃𝑟 𝑋 𝑥

If 𝑥 𝑥, then 𝑋 𝑥 ∩ 𝑋 𝑥 ∅ and the two events or regions do not overlap. This results in 𝑃𝑟 𝑋 𝑥 ∩ 𝑋 𝑥 0.

If 𝑥 𝑥, then there will be an overlapping region where

𝑃𝑟 𝑋 𝑥 ∩ 𝑋 𝑥 𝑃𝑟 𝑥 𝑋 𝑥 𝐹 𝑥 𝐹 𝑥

Combining the results

𝑃𝑟 𝑋 𝑥 ∩ 𝑋 𝑥0, 𝑥 𝑥

𝐹 𝑥 𝐹 𝑥 , 𝑥 𝑥



Making the conditional probability

𝐹 𝑥|𝑋 𝑥 𝑃𝑟 𝑋 𝑥|𝑋 𝑥𝑃𝑟 𝑋 𝑥 ∩ 𝑋 𝑥

𝑃𝑟 𝑋 𝑥

𝐹 𝑥|𝑋 𝑥0, 𝑥 𝑥

𝐹 𝑥 𝐹 𝑥1 𝐹 𝑥

, 𝑥 𝑥

The conditional density becomes

𝑓 𝑥|𝑋 𝑥0, 𝑥 𝑥

𝑓 𝑥1 𝐹 𝑥

, 𝑥 𝑥

Properties of the new pdf must include

1. xforMxf ,0|

2. 1|

dxMxf

3. duMufMxF

x

||

4. dxMxfxXx

x

x

2

1

|Pr 21



7.5 Discrete pmfs and the delta function

Engineering’s Dirac Delta Function.

The delta function is a useful signal representation for a “distribution” the integral properties that allows a discreet point in a continuous field to have “mass/area/volume”. That is

𝑓 𝑥 ∙ 𝛿 𝑥 𝑥 ∙ 𝑑𝑥 𝑓 𝑥

The delta function may also be approximated by a number of “area” based functions.

For example, a rectangular pulse of width Δ and height of 1/Δ. The “area” is then equal to 1 even as we let Δ→∞.

Another example is a Gaussian density function where the variance goes to zero.

Using delta functions a pmf can be described as

𝑓 𝑥 𝑝 0 ∙ 𝛿 𝑥 𝑥 𝑝 1 ∙ 𝛿 𝑥 𝑥 ⋯ 𝑝 𝑛 ∙ 𝛿 𝑥 𝑥

𝑓 𝑥 𝑝 𝑘 ∙ 𝛿 𝑥 𝑥

For an expected value computation, the “continuous” pmf becomes …

𝐸 𝑔 𝑋 𝑔 𝑥 ∙ 𝑓 𝑥 ∙ 𝑑𝑥

𝐸 𝑔 𝑋 𝑔 𝑥 ∙ 𝑝 𝑘 ∙ 𝛿 𝑥 𝑥 ∙ 𝑑𝑥

𝐸 𝑔 𝑋 𝑝 𝑘 ∙ 𝑔 𝑥 ∙ 𝛿 𝑥 𝑥 ∙ 𝑑𝑥

𝐸 𝑔 𝑋 𝑝 𝑘 ∙ 𝑔 𝑥 ∙ 𝛿 𝑥 𝑥 ∙ 𝑑𝑥

𝐸 𝑔 𝑋 𝑝 𝑘 ∙ 𝑔 𝑥

This is equivalent to the discrete pmf expected value computations.



Chapter 8: MULTIPLE CONTINUOUS RANDOM VARIABLES

8.1 Joint Densities and Distribution Functions

Cumulative Distribution Function:The probability of the event that the observed random variable X is less than or equal to the allowed value x and that the observed random variable Y is less than or equal to the allowed value y.

𝐹 𝑥, 𝑦 𝑃𝑟 ∞ 𝑋 𝑥 ∩ ∞ 𝑌 𝑦 𝑃𝑟 𝑋 𝑥 ∩ 𝑌 𝑦

yYxXyxFXY ,Pr,

The defined function can be discrete or continuous along the x- and y-axis. Constraints on the cumulative distribution function are:

1. yandxforyxFXY ,1,0

2. 0,,, XYXYXY FxFyF

3. 1, XYF

4. yxFXY , is non-decreasing as either x or y increases

5. xFxF XXY , and yFyF YXY ,

Analogies:

a 2-dimensional probability moving from scalars to vectors (2 or more elements) Calc 3 as compared to Calc 1 & 2?

2-D probability computations: yYyxXxP 121 ,

Think in terms of unions and intersections of 2-D boxes in the x,y plane …

Then by inspection we can arrive at …

11211222121 ,,,,, yxFyxFyxFyxFyYyxXxP XYXYXYXY



Joint Probability Density Function (pdf)

The derivative of the cumulative distribution function is the density function

yx

xFyxf X

2

,

Properties of the pdf include

1. yandxforyxf ,0,

2. 1,

dydxyxf

Note: the “volume” of the 2-D density function is one.

3.

y x

dvduvufyxF ,,

4. dyyxfxf X

, and dxyxfyfY

,

5. 2

1

2

1

,,Pr 2121

y

y

x

x

dydxyxfyYyxXx

Extension to n dimensions

Similar concepts extend this to 3, 4, or n dimensions.

The error associated with sending a rocket into three dimensional space must be modeled and defined in at least three dimension … maybe more …



Expected Values and Moments

The expected values is computed as may be expected …

𝐸 𝑔 𝑋, 𝑌 𝑔 𝑥 , 𝑦 ∙ 𝑝 𝑘, 𝑙

𝐸 𝑔 𝑋, 𝑌 𝑔 𝑥, 𝑦 ∙ 𝑓 𝑥, 𝑦 ∙ 𝑑𝑥 ∙ 𝑑𝑦

All expected values may be computed using the Joint pdf. There are correlation and covariance relationships to be included.

Correlation and Covariance between Random Variables

The definition of correlation is given as

𝑟 𝐸 𝑋 ∙ 𝑌 𝑋 ∙ 𝑌 ∙ 𝑝 𝑘, 𝑙

𝑟 𝐸 𝑋 ∙ 𝑌 𝑥 ∙ 𝑦 ∙ 𝑓 𝑥, 𝑦 ∙ 𝑑𝑥 ∙ 𝑑𝑦

But most of the time, we are not interested in products of mean values but what results when they are removed prior to the computation. Developing values where the random variable means have been extracted, is defined as computing the covariance

𝜎 𝐶𝑜𝑣 𝑋, 𝑌 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇 𝑟 𝜇 ∙ 𝜇

𝜎 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇 𝑋 𝜇 ∙ 𝑌 𝜇 ∙ 𝑝 𝑘, 𝑙

𝜎 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇 𝑋 𝜇 ∙ 𝑌 𝜇 ∙ 𝑓 𝑥, 𝑦 ∙ 𝑑𝑥 ∙ 𝑑𝑦

This gives rise to another factor, when the random variable means and variances are used to normalize the factors or correlation/covariance computation. For example, the following definition – correlation coefficient – is based on the normalized covariance

𝜌 𝐸𝑋 𝜇

𝜎∙

𝑌 𝜇𝜎

𝑋 𝜇𝜎

∙𝑌 𝜇

𝜎∙ 𝑓 𝑥, 𝑦 ∙ 𝑑𝑥 ∙ 𝑑𝑦

𝜌𝑟 𝜇 ∙ 𝜇

𝜎 ∙ 𝜎



Reminder from discrete multiple R.V. that operates the same way for continuous!!


The mean value

𝜇 𝐸 𝑍 𝐸 𝑋 𝑌 𝐸 𝑋 𝐸 𝑌 𝜇 𝜇

The variance

𝜎 𝐸 𝑍 𝜇 𝐸 𝑋 𝜇 𝑌 𝜇



𝜎 𝜎 2 ∙ 𝜎 𝜎

Thinking in terms of a product function we define

𝜎 𝜎 2 ∙ 𝜌 ∙ 𝜎 ∙ 𝜎 𝜎

Therefore, either the covariance or correlation coefficient can be used.



8.3 Independence of multiple R.V.

When the multiple variables are independent, there is significant simplification in the above computations!

For independent R.V.

𝑓 𝑥, 𝑦 𝑓 𝑥 ∙ 𝑓 𝑦

𝐹 𝑥, 𝑦 𝐹 𝑥 ∙ 𝐹 𝑦

We would expect 𝑔 𝑥, 𝑦 𝑔 𝑥 ∙ 𝑔 𝑦

and

𝐸 𝑔 𝑋, 𝑌 𝐸 𝑔 𝑥 ∙ 𝑔 𝑦 𝑔 𝑥 ∙ 𝑔 𝑦 ∙ 𝑓 𝑥 ∙ 𝑓 𝑦 ∙ 𝑑𝑥 ∙ 𝑑𝑦

𝐸 𝑔 𝑋, 𝑌 𝐸 𝑔 𝑥 ∙ 𝑔 𝑦 𝑔 𝑥 ∙ 𝑓 𝑥 ∙ 𝑑𝑥 ∙ 𝑔 𝑦 ∙ 𝑓 𝑦 ∙ 𝑦

𝐸 𝑔 𝑋, 𝑌 𝐸 𝑔 𝑥 ∙ 𝑔 𝑦 𝐸 𝑔 𝑥 ∙ 𝐸 𝑔 𝑦

As well as

𝑟 𝐸 𝑋 ∙ 𝑌 𝐸 𝑋 ∙ 𝐸 𝑌 𝜇 ∙ 𝜇

𝜎 𝐶𝑜𝑣 𝑋, 𝑌 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇 𝐸 𝑋 𝜇 ∙ 𝐸 𝑌 𝜇

𝜎 𝐶𝑜𝑣 𝑋, 𝑌 0 ∙ 0 0

or

𝜎 𝑟 𝜇 ∙ 𝜇 𝜇 ∙ 𝜇 𝜇 ∙ 𝜇 0

and


𝜎 ∙ 𝜎

𝜌𝜇 ∙ 𝜇 𝜇 ∙ 𝜇

𝜎 ∙ 𝜎0

Independence can greatly simplify a multiple random variable problem.



8.4 Conditional Probability (Again, with multiple R.V.)

Multiple random variables easily exist when a single entity or even person is described in terms of multiple parameters. We may expect the parameters to be related for the entity or person.

In addition, when parameters are related, knowing something about one can influence the second parameter being considered. The text offers the example of a person’s height and weight. The medical community has often published “ideal” height and weight charts for people to aspire to …

Note that the following is a different approach than your textbook …

Using the Cumulative Distribution Function (CDF), define

yF

yxF

M

MxXyYxF

YX

,

Pr

|Pr|

Considering a smaller region in Y

12

1221

,,|

yFyF

yxFyxFyYyxF

YYX

Leading to (for the Y interval going to zero engineers proof below),

yf

yxfyYxf

YX

,|

and

xf

yxfxXyf

XY

,|

These are different from the probability of a continuous distribution taking on a single value in X and Y…

0 xXFX or 0 yYFY



Joint Density to marginal density computations.

The joint density total probability concepts can define the x and y marginal densities.

dyyxfxf X

, and dxyxfyfY

,

Then from the conditional density relationship with the joint density

xfxXyfyfyYxfyxf XY ||,

We can replace the joint density functions in the total probability equations to define the pdf densities of x and y based on the conditional densities as

dxxfxXyfdxyxfyf XY

|,

or

dyyfyYxfdyyxfxf YX

|,

Bayes Theorem To derive the multiple variables Bayes Theorem, we return to

xfxXyfyfyYxfyxf XY ||,

Equating the right two elements result in ….

yf

xfxXyfyYxf

Y

X

||

or

xf

yfyYxfxXyf

X

Y

||

Important Note in review: the joint probability density function completely specifies: both marginal density functions and both conditional density functions.



Reviewing Independent Random Variables

Property

yfxfyxf YX ,

and

yFxFyxF YX ,

Then

xFyF

yFxF

yF

yxFyYxF X

Y

YX

Y

XYXY

,|

yFxF

yFxF

xF

yxFxXyF Y

X

YX

X

XYXY

,|

and xfyYxf XXY |

yfxXyf YXY |

Independence simplifies required computations, so pay attention to problems statements!

Useful results of independence for joint density functions

yfxfxfxXyfyfyYxfyxf YXXY ||,

Therefore

yfxfxfxXyf YXX |

yfxXyf Y|

and

xfyYxf X|

Note: if they are independent, knowing one does not help with the other!

It only matters if x and y are correlated in some way … and not independent.



8.6 Sums of Independent Random Variables (Again)

The general solution for the sum of independent random variables can be approached as a conditional probability. (Note: not applicable in the previous problem, not independent).

Letting S=X+Y.

𝐹 𝑠 𝑃𝑟 𝑆 𝑠 𝑃𝑟 𝑋 𝑌 𝑠

Using a concept from total probability

𝐹 𝑠 𝑃𝑟 𝑋 𝑌 𝑠|𝑌 𝑦 ∙ 𝑓 𝑦 ∙ 𝑑𝑦

For the conditioned value of y

𝐹 𝑠 𝑃𝑟 𝑋 𝑦 𝑠|𝑌 𝑦 ∙ 𝑓 𝑦 ∙ 𝑑𝑦 𝑃𝑟 𝑋 𝑠 𝑦|𝑌 𝑦 ∙ 𝑓 𝑦 ∙ 𝑑𝑦

Assuming X and Y are actually independent

𝐹 𝑠 𝑃𝑟 𝑋 𝑠 𝑦 ∙ 𝑓 𝑦 ∙ 𝑑𝑦

But the probability is a proper CDF now …

𝐹 𝑠 𝐹 𝑠 𝑦 ∙ 𝑓 𝑦 ∙ 𝑑𝑦

Form the pdf taking a derivative in s

𝑑𝑑𝑠

𝐹 𝑠 𝑓 𝑠𝑑

𝑑𝑠𝐹 𝑠 𝑦 ∙ 𝑓 𝑦 ∙ 𝑑𝑦

Which in performing a partial with respect to s

𝑓 𝑠 𝑓 𝑠 𝑦 ∙ 𝑓 𝑦 ∙ 𝑑𝑦

The pdf of the independent R.V. sum is the convolution of the two pdf functions.



8.8 General Transformations and the Jacobian

The textbook solution sequence states:

1. Let Y=g(X) be the transformation. It is desired to compute the density of Y.

2. Rather than computing the density function directly, it may be better to compute (or consider the computation of) the CDF function:

𝐹 𝑦 𝑃𝑟 𝑌 𝑦 𝑃𝑟 𝑔 𝑋 𝑦

3. Now comes the crucial step: invert the transformation to get a probability on X and not g(X). X 𝑔 𝑌 𝑖𝑛𝑣_𝑜𝑓_𝑔 𝑌

4. Write the probability in terms of the distribution function of X as a function of y.

5. Differentiate with respect to y to get the density function of Y.

The simple examples involve linear one-to-one mappings.

Linear mapping 𝑌 𝑎 ∙ 𝑋 𝑏

The CDF of Y 𝐹 𝑦 𝑃𝑟 𝑌 𝑦 𝑃𝑟 𝑎 ∙ 𝑋 𝑏 𝑦

Creating the probability in X

𝐹 𝑦 𝑃𝑟 𝑋𝑦 𝑏

𝑎, 0 𝑎

𝐹 𝑦 𝑃𝑟 𝑋𝑦 𝑏

𝑎, 𝑎 0

Express this in terms of the CDF of X.

𝐹 𝑦 𝐹𝑦 𝑏

𝑎, 0 𝑎

𝐹 𝑦 1 𝐹𝑦 𝑏

𝑎, 𝑎 0

Continuing on the next page … differentiating for the pdf



Differentiating for the pdf

𝑓 𝑦1𝑎

∙ 𝑓𝑦 𝑏

𝑎, 0 𝑎

𝑓 𝑦1𝑎

∙ 𝑓𝑦 𝑏

𝑎, 𝑎 0

or

𝑓 𝑦1

|𝑎|∙ 𝑓

𝑦 𝑏𝑎

This is diagramed in the textbook as

Introducing the Jacobian

For the linear system transformation, the 1/a is the Jacobian of the transformation. As described in notes from the previous chapter when looking at the summation

𝑓 𝑥 ∙ 𝑑𝑥 𝑓 𝑦 ∙ 𝑑𝑦

we are interested in

𝑓 𝑦 ∙ 𝑑𝑦 𝑓 𝑥 ∙𝑑𝑥𝑑𝑦

∙ 𝑑𝑦 𝑓 𝑥 ∙ |𝐽 𝑥; 𝑦 | ∙ 𝑑𝑦

Where J(x;y) represents the Jacobian … a partial derivative function of the variables defined.

𝑋 𝑌 𝑏 𝑎⁄ and 𝑑𝑥 𝑑𝑦⁄ 1 𝑎⁄



Applying this technique to the linear transformation

Pr 𝑢 𝑋 𝑣 𝑓 𝑥 ∙ 𝑑𝑥 𝑓𝑦 𝑏

𝑎∙

1𝑎

∙ 𝑑𝑦∙

∙, 0 𝑎

Pr 𝑢 𝑋 𝑣 𝑓 𝑦 ∙1𝑎

∙ 𝑑𝑦∙

∙, 0 𝑎

for a negative, the integration bounds change

Pr 𝑢 𝑋 𝑣 𝑓 𝑥 ∙ 𝑑𝑥 𝑓𝑦 𝑏

𝑎∙

1𝑎

∙ 𝑑𝑦∙

∙

Pr 𝑢 𝑋 𝑣 𝑓 𝑦 ∙1𝑎

∙ 𝑑𝑦∙

∙, 𝑎 0



Probability Density Function of a Function of Two Random Variables


Functions of random variables create new random variable. As before, expect that the resulting probability distribution and density function are different.

Assume that there is a function that combines two random variables and that the functions inverse exists.

YXZ ,1 and YXW ,2

and the inverse WZX ,1 and WZY ,2

The original pdf is yxf , with the derived pdf in the transform space of wzg , .

Then it can be proven that: 21212121 ,Pr,Pr yYyxXxwWwzZz

or equivalently

2

1

2

1

2

1

2

1

,,

y

y

x

x

w

w

z

z

dydxyxfdwdzwzg

Empirically, since the density function must integrate to one for infinite bounds, the “transformed” portion of one density must have the same “volume” as the original density function.

Using an advanced calculus theorem to perform a transformation of coordinates. [ Using the determinant of the Jacobian)

Jwzwzf

w

y

z

yw

x

z

x

wzwzfwzg

,,,,,,, 2121

And the integrals become.

2

1

2

1

2

1

2

1

,,,, 21

w

w

z

z

w

w

z

z

dwdzJwzwzfdwdzwzg



For a linear problems that may seem familiar


Given: yxf YX ,,

We can generate YXV and YXW

Therefore 2

WVX

and

2

WVY

Forming the Jacobian and taking the determinant

2

1

2

12

1

2

1

w

y

v

yw

x

v

x

J and 2

1

2

1

2

1

2

1

2

1det

v

y

w

x

w

y

v

xJ

Finally we arrive at

2

,22

1, ,,

wvwvfwvf YXWV

OK, but what about just V or W

V= X+Y

dwwvfvf WVV ,,

dwwvwv

fvf YXV 2,

22

1,

Use a change in variable:

2

wvz

and

2

1

dw

dz

dzzvzfvf YXV ,,

For X, and Y independent

dzzvfzfvf YXV

Convolution

W=X-Y

dvwvfwf WVW ,,

dvwvwv

fwf YXW 2,

22

1,

Use a change in variable:

2

wvz

and

2

1

dv

dz

dzzwzfwf YXW ,,

For X, and Y independent

dzzfwzfwf YXW

Correlation



Example 8.6 on p. 212

R √X Y and Θ tan

The Inverse Transformation becomes.

X R ∙ cos Θ and Y R ∙ sin Θ

Thus Jacobian matrix

J

𝜕𝑥𝜕𝑟

𝜕𝑥𝜕𝜃

𝜕𝑦𝜕𝑟

𝜕𝑦𝜕𝜃

cos Θ r ∙ sin Θsin Θ r ∙ cos Θ

The determinant is

|J|

𝜕𝑥𝜕𝑟

𝜕𝑥𝜕𝜃

𝜕𝑦𝜕𝑟

𝜕𝑦𝜕𝜃

r ∙ cos Θ r ∙ sin Θ 𝑟

Therefore

𝑓 , 𝑟, Θ 𝑓 , r ∙ cos Θ , r ∙ sin Θ ∙ |J| 𝑟 ∙ 𝑓 , r ∙ cos Θ , r ∙ sin Θ

Exam #2 Review - Western Michigan Universitybazuinb/ECE3800/Exam2_Review.pdf · Exam #2 Review What...

Documents

Transcript of Exam #2 Review - Western Michigan Universitybazuinb/ECE3800/Exam2_Review.pdf · Exam #2 Review What...