Exam #3 Review - Western Michigan Universitybazuinb/ECE3800/Exam3_Review.pdf · B.J. Bazuin, Fall...

B.J. Bazuin, Spring 2020 1 of 35 ECE 3800

Exam #3 Review

What is on an exam? Read through the homework and class examples… 4 multipart questions. Points assigned based on complexity. (4Q, 100-120 pts. Sp. 2019/Fa. 2019)

Skills #5 … 21.3, 24.2, 24.6, 25.1

Skills #6 … 21.3, 24.2, 24.6, 25.1

Skills #7 … This exam is likely to be four problems, 2 old questions (exam 2 material) and 2 new questions. Old Questions (similar to 2, 3 or 4 from exam2): 1) probability density function – derive constant, means, variances, defined probability, conditional probability. 2) Functions of other random variables Y=f(X) – determine the pdf, defined prob., conditional prob. 3) Functions of two random variables Z=aX±bY – determine the pdf, mean, variance New Questions: 1) Autocorrelation/Crosscorrelation in time and probability.

Independent R.V. Is it WSS. 2) Autocorrelation to Power Spectral Density (forward and inverse)

Given an autocorrelation, determine the mean, variance, 2nd moment, and power spectral density.

You will be given a random sequence or process. Determine the mean. Determine the autocorrelation. Perform an auto- and/or cross-correlation. Answer related questions.

Determine the PSD from an autocorrelation function. Have a Fourier Transform Table available!

Notes: No filtering in the frequency domain questions. No Chapter 10 materials … confidence intervals. (There was in 2016, 2018, and 2019Sp.) And now for a quick chapter review … the important information without the rest!


Text Elements

7 A Continuous random Variable 7.1 A Continuous Random Variable and Its Density, Distribution Function, and Expected

Values 7.2 Example Calculations for a Single Random Variable 7.3 Selected Continuous Distributions

7.3.1 The Uniform Distribution 7.3.2 The Exponential Distribution

7.4 Conditional Probabilities for a Continuous Random Variable 7.5 Discrete PMFs and Delta Functions 7.6 Quantization 7.7 A Final Word

8 Multiple Continuous Random Variables 8.1 Joint Densities and Distribution Functions 8.2 Expected Values and Moments 8.3 Independence 8.4 Conditional Probabilities for Multiple Random Variables 8.5 Extended Example: Two Continuous Random Variables 8.6 Sums of Independent Random Variables 8.7 Random Sums 8.8 General Transformations and the Jacobian 8.9 Parameter Estimation for the Exponential Distribution 8.10 Comparison of Discrete and Continuous Distributions

9 The Gaussian and Related Distributions 9.1 The Gaussian Distribution and Density 9.2 Quantile Function 9.3 Moments of the Gaussian Distribution 9.4 The Central Limit Theorem 9.5 Related Distributions

9.5.1 The Laplace Distribution 9.5.2 The Rayleigh Distribution 9.5.3 The Chi-Squared and F Distributions

9.6 Multiple Gaussian Random Variables 9.6.1 Independent Gaussian Random Variables 9.6.2 Transformation to Polar Coordinates 9.6.3 Two Correlated Gaussian Random Variables

9.7 Example: Digital Communications Using QAM 9.7.1 Background 9.7.2 Discrete Time Model 9.7.3 Monte Carlo Exercise 9.7.4 QAM Recap


10 Elements of Statistics (Final exam only 10.1 A Simple Election Poll 10.2 Estimating the Mean and Variance 10.3 Recursive Calculation of the Sample Mean 10.4 Exponential Weighting 10.5 Order Statistics and Robust Estimates 10.6 Estimating the Distribution Function 10.7 PMF and Density Estimates 10.8 Confidence Intervals 10.9 Significance Tests and p-Values 10.10 Introduction to Estimation Theory 10.11 Minimum Mean Squared Error Estimation 10.12 Bayesian Estimation

13 Random Signals and Noise 13.1 Introduction to Random Signals 13.2 A Simple Random Process 13.3 Fourier Transforms 13.4 WSS Random Processes 13.5 WSS Signals and Linear Filters 13.6 Noise

13.6.1 Probabilistic Properties of Noise 13.6.2 Spectral Properties of Noise

13.7 Example: Amplitude Modulation 13.8 Example: Discrete Time Wiener Filter 13.9 The Sampling Theorem for WSS Random Processes

13.9.1 Discussion 13.9.2 Example: Figure 13.4 13.9.3 Proof of the Random Sampling Theorem


See Exam 2 Review for related materials

Joint density function – derive marginal densities, means, variances, correlation, identify if independent and/or correlated.

The defined function can be discrete or continuous along the x- and y-axis. Constraints on the cumulative distribution function are:

1. yandxforyxFXY ,1,0

2. 0,,, XYXYXY FxFyF

3. 1, XYF

4. yxFXY , is non-decreasing as either x or y increases

5. xFxF XXY , and yFyF YXY ,

Properties of the pdf include

1. yandxforyxf ,0,

2. 1,

dydxyxf

Note: the “volume” of the 2-D density function is one.

3.

y x

dvduvufyxF ,,

4. dyyxfxf X

, and dxyxfyfY

,

5. 2

1

2

1

,,Pr 2121

y

y

x

x

dydxyxfyYyxXx

The definition of correlation is given as

𝑟 𝐸 𝑋 ∙ 𝑌 𝑋 ∙ 𝑌 ∙ 𝑝 𝑘, 𝑙

𝑟 𝐸 𝑋 ∙ 𝑌 𝑥 ∙ 𝑦 ∙ 𝑓 𝑥, 𝑦 ∙ 𝑑𝑥 ∙ 𝑑𝑦

But most of the time, we are not interested in products of mean values but what results when they are removed prior to the computation. Developing values where the random variable means have been extracted, is defined as computing the covariance


𝜎 𝐶𝑜𝑣 𝑋,𝑌 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇 𝑟 𝜇 ∙ 𝜇

𝜎 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇 𝑋 𝜇 ∙ 𝑌 𝜇 ∙ 𝑝 𝑘, 𝑙

𝜎 𝐸 𝑋 𝜇 ∙ 𝑌 𝜇 𝑋 𝜇 ∙ 𝑌 𝜇 ∙ 𝑓 𝑥,𝑦 ∙ 𝑑𝑥 ∙ 𝑑𝑦

This gives rise to another factor, when the random variable means and variances are used to normalize the factors or correlation/covariance computation. For example, the following definition – correlation coefficient – is based on the normalized covariance

𝜌 𝐸𝑋 𝜇𝜎

∙𝑌 𝜇𝜎

𝑋 𝜇𝜎

∙𝑌 𝜇𝜎

∙ 𝑓 𝑥,𝑦 ∙ 𝑑𝑥 ∙ 𝑑𝑦

𝜌𝑟 𝜇 ∙ 𝜇

𝜎 ∙ 𝜎𝜎

𝜎 ∙ 𝜎

For independent R.V.

𝑓 𝑥,𝑦 𝑓 𝑥 ∙ 𝑓 𝑦

𝐹 𝑥, 𝑦 𝐹 𝑥 ∙ 𝐹 𝑦

As well as

𝑟 𝐸 𝑋 ∙ 𝑌 𝐸 𝑋 ∙ 𝐸 𝑌 𝜇 ∙ 𝜇

𝜎 𝐶𝑜𝑣 𝑋,𝑌 0 ∙ 0 0

and

𝜌𝜇 ∙ 𝜇 𝜇 ∙ 𝜇

𝜎 ∙ 𝜎0

Independence can greatly simplify a multiple random variable problem.


Functions of other random variables Y=f(X) – determine the pdf, defined prob., conditional prob.

For all linear cases …

m

byf

myf XY

1

But it also works for other cases when there is a one-to-one relationship between X and Y!

If xgy has a finite number of real roots (multiple solutions), then the disjoint events (related to each of the roots) have the form

ygxi1 are related to the events

iiii dxxXxE or iiiii dxxXdxxE

If we then accumulate all of them we can find the probability

i

iiXY dxxfdyyfdyyYyP

By “engineering manipulation” (divide by dy)

i

iiX

i

iiXY dy

dxxf

dy

dxxfyf

Which can also be perform in terms of the functional derivative xgxgdx

d

dx

dy'

i i

iX

i iiXY xg

xf

dx

dyxfyf

'

1

, for ygxi1

Remember that the final result is a function of y, so substitute the correct values in ygxi1 !


Functions of two random variables Z=aX±bY – determine the pdf, mean , variance

Letting 𝑍 𝑎 ∙ 𝑋 𝑏 ∙ 𝑌

The mean value

𝜇 𝐸 𝑍 𝐸 𝑎 ∙ 𝑋 𝑏 ∙ 𝑌 𝑎 ∙ 𝐸 𝑋 𝑏 ∙ 𝐸 𝑌 𝑎 ∙ 𝜇 𝑏 ∙ 𝜇

The variance

𝜎 𝐸 𝑍 𝜇 𝐸 𝑎 ∙ 𝑋 𝑎 ∙ 𝜇 𝑏 ∙ 𝑌 𝑏 ∙ 𝜇

𝜎 𝐸 𝑎 ∙ 𝑋 𝜇 2 ∙ 𝐸 𝑎 ∙ 𝑏 ∙ 𝑋 𝜇 ∙ 𝑌 𝜇 𝐸 𝑏 ∙ 𝑌 𝜇

𝜎 𝑎 ∙ 𝜎 2 ∙ 𝑎 ∙ 𝑏 ∙ 𝐶𝑜𝑣 𝑋,𝑌 𝑏 ∙ 𝜎

𝜎 𝑎 ∙ 𝜎 2 ∙ 𝑎 ∙ 𝑏 ∙ 𝜎 𝑏 ∙ 𝜎

Thinking in terms of a product function we define

𝜎 𝑎 ∙ 𝜎 2 ∙ 𝑎 ∙ 𝑏 ∙ 𝜌 ∙ 𝜎 ∙ 𝜎 𝑏 ∙ 𝜎

Therefore, either the covariance or correlation coefficient can be used.

For X and Y independent

𝜎 𝑎 ∙ 𝜎 𝑏 ∙ 𝜎

For computing the pdf of Z, either the Jacobian method can be used or if X, and Y are intendant

Letting 𝑍 𝑋 𝑌

The pdf can be computed as the convolution

𝑓 𝑧 𝑓 𝑧 𝑦 ∙ 𝑓 𝑦 ∙ 𝑑𝑦

or

𝑓 𝑧 𝑓 𝑥 ∙ 𝑓 𝑧 𝑥 ∙ 𝑑𝑥


Chap. 9: Gaussian Distribution and Density

The Gaussian or Normal probability density function is defined as:

𝑓 𝑥1

√2𝜋 ∙ 𝜎∙ 𝑒𝑥𝑝

𝑥 𝜇2 ∙ 𝜎

, ∞ 𝑥 ∞

where μ is the mean and σ is the variance

The Gaussian Cumulative Distribution Function (CDF)

𝐹 𝑥1


𝑣 𝜇2 ∙ 𝜎

∙ 𝑑𝑣

The CDF can not be represented in a closed form solution!

Normal Distribution – Gaussian with zero mean and unit variance.

The Normal probability density function is defined as:

xfor

xxN ,

2exp

2

1 2

The Normal Cumulative Distribution Function (CDF)

dvv

xx

v

N

2exp

2

1 2

Note the relationship between the Gaussian and Gaussian-Normal is

x

xF XX

see the MATLAB: GaussianDemo.m

-8 -6 -4 -2 0 2 4 6 80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Gaussian PDF and pdf


Important notes on the Gaussian curve:

The pdf

1. There is only one maximum and it occurs at the mean value.

2. The density function is symmetric about the mean value.

3. The width of the density function is directly proportional to the standard deviation, . The width of 2 occurs at the points where the height is 0.607 of the maximum value. These are also the points of the maximum slope. Also note that:

683.0Pr X

955.022Pr X

4. The maximum value of the density function is inversely proportional to the standard deviation, .

2

1Xf

5. Since the density function has an area of unity, it can be used as a representation of the impulse or delta function by letting approach zero. That is

2

2

0 2exp

2

1lim

xx


Specific Values for the Standard Normal CDF

𝜑 𝑥1

√2𝜋∙ 𝑒𝑥𝑝

𝑥2

, ∞ 𝑥 ∞

where μ = 0 is the mean and the variance σ = 1.

Φ 𝑥1


𝑣2

∙ 𝑑𝑣


Gaussian to Normal is a linear scaling

Letting the linear relationship be defined as

𝑍𝑋 𝜇𝜎

The inverse mapping 𝑋 𝑍 ∙ 𝜎 𝜇

the Jocobian or derivative becomes 𝑑𝑥𝑑𝑧

𝜎

Therefore

𝑓 𝑧 𝑓 𝑥 ∙𝑑𝑥𝑑𝑧

Then for the normalized form the R.V.

𝑓 𝑥1


𝑥 𝜇2 ∙ 𝜎

, ∞ 𝑥 ∞

𝑓 𝑧1


𝑧 ∙ 𝜎 𝜇 𝜇2 ∙ 𝜎

∙ 𝜎

𝑓 𝑧1


𝑧 ∙ 𝜎2 ∙ 𝜎

𝑓 𝑧1


𝑧2

𝜑 𝑦

In addition, we would expect

𝐹 𝑥 Φ 𝑧

𝐹 𝑥 Φ𝑥 𝜇𝜎


Two-sided Gaussian Probability

Pr μ σ x μ σ 0.6827

Pr μ 2σ x μ 2σ 0.9545

Pr μ 3σ x μ 3σ 0.9973

One-Sided Gaussian Probability

𝑃𝑟 𝑥 0 0.5

𝑃𝑟 𝑥 𝜇 𝜎 0.8413

𝑃𝑟 𝑥 𝜇 2𝜎 0.9772

𝑃𝑟 𝑥 𝜇 3𝜎 0.9987

Three will be multiple problems and examples where either a two-sided or on-sided Gaussian probability is required. There are differences in the solutions derived if the wrong one is selected!


Equivalent Gaussian probability representations

𝑓 𝑥1


𝑥 𝜇2 ∙ 𝜎

, ∞ 𝑥 ∞

𝜑 𝑧1


𝑧2

Manipulations

Pr a X b 𝐹 𝑏 𝐹 𝑎

Pr a X b Pr a 𝜇 X 𝜇 b 𝜇 , 𝑠ℎ𝑖𝑓𝑡𝑖𝑛𝑔 𝑚𝑒𝑎𝑛

Pr a X b Pra 𝜇𝜎

X 𝜇𝜎

b 𝜇𝜎

, 𝑙𝑖𝑛𝑒𝑎𝑟 𝑠𝑐𝑎𝑙𝑖𝑛𝑔

𝑍𝑋 𝜇𝜎

Pr a X b Pra 𝜇𝜎

Zb 𝜇𝜎

, 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛

Using normalized probability

Pr a X b Φb 𝜇𝜎

Φa 𝜇𝜎

The normalization of the Gaussian is often implemented using “Z”.

The computation with the standard normalization has been referred to as a z-score.

Equivalent Probabilities Pr Z b Φ b

Pr Z a 1 Φ a

Pr a Z b Φ b Φ a

Also note Φ z 1 Φ z


Other relationships with normalized Gaussian

Pr a Z a Φ 𝑎 Φ 𝑎 Φ 𝑎 1 Φ 𝑎

Pr a Z a 2 ∙ Φ 𝑎 1

or in general

Pr a Z b Φ 𝑏 Φ 𝑎 Φ 𝑏 1 Φ 𝑎

Pr a Z b Φ 𝑏 Φ 𝑎 1

Performing Computations

The error function

𝑒𝑟𝑓 𝑧2

√𝜋∙ 𝑒𝑥𝑝 𝑦 ∙ 𝑑𝑦

Φ z12

12∙ 𝑒𝑟𝑓

𝑧

√2

𝑍𝑋 𝜇𝜎

𝐹 𝑥12

12∙ 𝑒𝑟𝑓

𝑥 𝜇

√2 ∙ 𝜎

For multiple bounds

22

1

2

1

22

1

2

11

aerf

berfFbFbXaP XXX

𝑃𝑟 𝑎 𝑋 𝑏 𝐹 𝑏 𝐹 𝑎12∙ 𝑒𝑟𝑓

𝑏 𝜇

√2 ∙ 𝜎

12∙ 𝑒𝑟𝑓

𝑎 𝜇

√2 ∙ 𝜎

This definition is valid for MATLAB and EXCEL and WIKIPEDIA. There are other sources that do not define it this way, so check before use!

Φ z12

12∙ 𝑒𝑟𝑓

𝑧

√2


The complementary error function

𝑒𝑟𝑓𝑐 𝑧 1 𝑒𝑟𝑓 𝑧2

√𝜋∙ 𝑒𝑥𝑝 𝑦 ∙ 𝑑𝑦

Φ z 1 Φ 𝑧 112

12∙ 𝑒𝑟𝑓

𝑧

√2

Φ z 1 Φ 𝑧12

12∙ 𝑒𝑟𝑓

𝑧

√2

12∙ 𝑒𝑟𝑓𝑐

𝑧

√2

There are also inverse functions for erf and erfc!

z Φ 𝑃𝑟 √2 ∙ erfinv 2 ∙ 𝑃𝑟 1

The Q function in communications is “the tail of the Gaussian

Q z 1 Φ 𝑧12

12∙ 𝑒𝑟𝑓

𝑧

√2

12∙ 𝑒𝑟𝑓𝑐

𝑧

√2


9.4 Central Limit Theorem

https://en.wikipedia.org/wiki/Central_limit_theorem

“In probability theory, the central limit theorem (CLT) establishes that, in most situations, when independent random variables are added, their properly normalized sum tends toward a normal distribution (informally a "bell curve") even if the original variables themselves are not normally distributed. The theorem is a key concept in probability theory because it implies that probabilistic and statistical methods that work for normal distributions can be applicable to many problems involving other types of distributions.”

The convolution of pdf of summed R.V. begins to look Gaussian after a large number of R.V. are summed.

Sums of IID R.V.

S 𝑋

If n is known, the expected value of the sum should be expected

𝐸 S E 𝑋 𝐸 𝑋 𝜇 n ∙ 𝜇

𝑉𝑎𝑟 S E 𝑋 𝜇 𝑉𝑎𝑟 𝑋 𝜎 n ∙ 𝜎

If we normalize the summed random variance

YS 𝐸 S

𝑉𝑎𝑟 S

Then

𝐸 Y ES 𝐸 S

𝑉𝑎𝑟 S0

𝑉𝑎𝑟 Y VarS 𝐸 S

𝑉𝑎𝑟 S1.0

Based on the Central Limit Theorem, Y will be a Normal R.V. as n becomes very large.


13.1 Introduction to Random Signals

A random process is a collection of time functions and an associated probability description.

When a continuous or discrete or mixed process in time/space can be describe mathematically as a function containing one or more random variables.

A sinusoidal waveform with a random amplitude. A sinusoidal waveform with a random phase. A sequence of digital symbols, each taking on a random value for a defined time period

(e.g. amplitude, phase, frequency). A random walk (2-D or 3-D movement of a particle)

The entire collection of possible time functions is an ensemble, designated as tx , where one

particular member of the ensemble, designated as tx , is a sample function of the ensemble. In general only one sample function of a random process can be observed!

Let X(t) be a random process.

If we take multiple time samples, 𝑡 , 𝑡 ,⋯ , 𝑡 , then each time sample is a random variable. 𝑋 𝑡 ,𝑋 𝑡 ,⋯ ,𝑋 𝑡

The random process might then have a nth order density function that could be described as 𝑓 𝑥 , 𝑥 ,⋯ , 𝑥 ; 𝑡 , 𝑡 ,⋯ , 𝑡

Nominally we might described the density function of the elements sampled from the random process as

𝑓 𝑥; 𝑡

The mean, 2nd moment and variance of X(t) would then be defined as

𝜇 𝑡 𝐸 𝑋 𝑡 𝑥 ∙ 𝑓 𝑥; 𝑡 ∙ 𝑑𝑥

𝐸 𝑋 𝑡 𝑥 ∙ 𝑓 𝑥; 𝑡 ∙ 𝑑𝑥

𝜎 𝑡 𝐸 𝑋 𝑡 𝐸 𝑋 𝑡 𝐸 𝑋 𝑡 𝜇 𝑡

Signal correlation

As we have multiple times at which the sample may be taken, we must be able to compare samples sets to themselves or different samples or sample offset sequences to each other.


Auto-correlation is defined as

𝑅 𝑡 , 𝑡 𝐸 𝑋 𝑡 ∙ 𝑋 𝑡

Auto-covariance is defined as

𝐶 𝑡 , 𝑡 𝐸 𝑋 𝑡 𝜇 𝑡 ∙ 𝑋 𝑡 𝜇 𝑡

𝐶 𝑡 , 𝑡 𝐸 𝑋 𝑡 ∙ 𝑋 𝑡 𝑋 𝑡 ∙ 𝜇 𝑡 𝜇 𝑡 ∙ 𝑋 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡

𝐶 𝑡 , 𝑡 𝐸 𝑋 𝑡 ∙ 𝑋 𝑡 𝐸 𝑋 𝑡 ∙ 𝜇 𝑡 𝜇 𝑡 ∙ 𝐸 𝑋 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡

𝐶 𝑡 , 𝑡 𝐸 𝑋 𝑡 ∙ 𝑋 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡

𝐶 𝑡 , 𝑡 𝐸 𝑋 𝑡 ∙ 𝑋 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡

𝐶 𝑡 , 𝑡 𝑅 𝑡 , 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡

For real R.V. it can also be shown that

𝑅 𝑡 , 𝑡 𝑅 𝑡 , 𝑡

𝐶 𝑡 , 𝑡 𝐶 𝑡 , 𝑡

If two separate random process exist, we describe cross-correlation and cross covariance as

𝑅 𝑡 , 𝑡 𝐸 𝑋 𝑡 ∙ 𝑌 𝑡

𝐶 𝑡 , 𝑡 𝐸 𝑋 𝑡 𝜇 𝑡 ∙ 𝑌 𝑡 𝜇 𝑡

𝐶 𝑡 , 𝑡 𝐸 𝑋 𝑡 ∙ 𝑌 𝑡 𝑋 𝑡 ∙ 𝜇 𝑡 𝜇 𝑡 ∙ 𝑌 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡

𝐶 𝑡 , 𝑡 𝐸 𝑋 𝑡 ∙ 𝑌 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡

𝐶 𝑡 , 𝑡 𝑅 𝑡 , 𝑡 𝜇 𝑡 ∙ 𝜇 𝑡


Stationary vs. Nonstationary Random Processes

The probability density functions for random variables in time have been discussed, but what is the dependence of the density function on the value of time, t or n, when it is taken?

If all marginal and joint density functions of a process do not depend upon the choice of the time origin, the process is said to be stationary (that is it doesn’t change with time). All the mean values and moments are constants and not functions of time!

For nonstationary processes, the probability density functions change based on the time origin or in time. For these processes, the mean values and moments are functions of time.

In general, we always attempt to deal with stationary processes … or approximate stationary by assuming that the process probability distribution, means and moments do not change significantly during the period of interest.

Examples:

Resistor values (noise varies based on the local temperature)

Wind velocity (varies significantly from day to day)

Humidity (though it can change rapidly during showers)

The requirement that all marginal and joint density functions be independent of the choice of time origin is frequently more stringent (tighter) than is necessary for system analysis.

A more relaxed requirement is called stationary in the wide sense: where the mean value of any random variable is independent of the choice of time, t, and that the correlation of two random variables depends only upon the time difference between them. That is

XXtXE and

XXRXXttXXEtXtXE 00 1221 for 12 tt

You will typically deal with Wide-Sense Stationary Signals (WSS).

For WSS, the autocorrelation and autocovariance are a function of the difference in time and not the absolute times.

𝜏 𝑡 𝑡

𝑅 𝑡 , 𝑡 𝑅 𝑡, 𝑡 𝜏 𝑅 0, 𝜏 𝑅 𝜏


For WSS random processes 𝜇 𝑡 𝐸 𝑋 𝑡 𝜇

𝜎 𝑡 𝜎 𝐸 𝑋 𝑡 𝜇

𝑅 𝑡 , 𝑡 𝑅 𝑡, 𝑡 𝜏 𝑅 𝑡 𝑡 𝑅 𝜏

𝐶 𝑡 , 𝑡 𝐶 𝑡, 𝑡 𝜏 𝐶 𝑡 𝑡 𝐶 𝜏

𝐶 𝑡, 𝑡 𝜏 𝐶 𝜏 𝑅 𝜏 𝜇

Additional properties

1. For real random processes the auto-correlation and auto-covariance are symmetric about 0. 𝑅 𝜏 𝑅 𝜏

𝐶 𝜏 𝐶 𝜏

2. The zeroeth lag (t=0) of the auto-correlation is the 2nd moment or power. And it must be positive.

𝑅 0 𝜎 𝜇 0

𝐶 0 𝜎 0

3. The zeroeth lag (t=0) of the auto-correlation is a maximum for all time lags. 𝑅 0 |𝑅 𝜏 |

4. If X(t) is a zero mean WSS random process, the sum of the process and a constant will have a constant factor as part of the autocorrelation and can be described as.

𝑌 𝑡 𝑋 𝑡 𝑎

𝑅 𝜏 𝑅 𝜏 𝑎

The previous can be extended to, for a non-zero mean X(t)

𝑌 𝑡 𝑎 ∙ 𝑋 𝑡 𝑏

𝜇 𝑎 ∙ 𝜇 𝑏

𝑅 𝜏 𝑎 ∙ 𝑅 𝜏 2 ∙ a ∙ b ∙ 𝜇 𝑏

For two independent WSS random processes

𝑍 𝑡 𝑋 𝑡 𝑌 𝑡

𝜇 𝜇 𝜇

𝑅 𝑡, 𝑡 𝜏 𝐸 𝑋 𝑡 𝑌 𝑡 ∙ 𝑋 𝑡 𝜏 𝑌 𝑡 𝜏


𝑅 𝑡, 𝑡 𝜏 𝐸 𝑋 𝑡 ∙ 𝑋 𝑡 𝜏 𝐸 𝑋 𝑡 ∙ 𝑌 𝑡 𝜏 𝐸 𝑌 𝑡 ∙ 𝑋 𝑡 𝜏𝐸 𝑌 𝑡 ∙ 𝑌 𝑡 𝜏

𝑅 𝜏 𝑅 𝜏 𝜇 ∙ 𝜇 𝜇 ∙ 𝜇 𝑅 𝜏

𝑅 𝜏 𝑅 𝜏 2 ∙ 𝜇 ∙ 𝜇 𝑅 𝜏

Now what is the auto-covariance?

𝐶 𝜏 𝑅 𝜏 𝜇

𝐶 𝜏 𝑅 𝜏 2 ∙ 𝜇 ∙ 𝜇 𝑅 𝜏 𝜇 𝜇

𝐶 𝜏 𝑅 𝜏 2 ∙ 𝜇 ∙ 𝜇 𝑅 𝜏 𝜇 2 ∙ 𝜇 ∙ 𝜇 𝜇

𝐶 𝜏 𝑅 𝜏 𝜇 𝑅 𝜏 𝜇

𝐶 𝜏 𝐶 𝜏 𝐶 𝜏


Ergodic and Nonergodic Random Processes

Ergodicity deals with the problem of determining the statistics of an ensemble based on measurements from a sample function of the ensemble.

For ergodic processes, all the statistics can be determined from a single function of the process.

This may also be stated based on the time averages. For an ergodic process, the time averages (expected values) equal the ensemble averages (expected values). That is to say,

T

T

n

T

nn dttXT

dxxfxX2

1lim

Note that ergodicity cannot exist unless the process is stationary!

Ergodicity is the concept that ties time based computations with probabilistic based computations!

T

T

n

T

nn dttXT

dxxfxX2

1lim

The time autocorrelation

txtxdttxtx

T

T

TT

XX 2

1lim

Overall … WSS, ergodic processes are preferred as the starting conditions for engineering model, systems and simulations!

Notes and figures are based on or taken from materials in the course textbook: Probabilistic Methods of Signal and System Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9


A Process for Determining Stationarity and Ergodicity

a) Find the mean and the 2nd moment based on the probability

b) Find the time sample mean and time sample 2nd moment based on time averaging.

c) If the means or 2nd moments are functions of time … non-stationary

d) If the time average mean and moments are not equal to the probabilistic mean and moments or if it is not stationary, then it is non ergodic.

Example Computations for means and 2nd moment:

dxxfxX X and

dxxfxX X22

T

TT

dttxT

Xx2

1limˆ and

T

TT

dttxT

x 22

2

1lim


The Autocorrelation Function

For a sample function defined by samples in time of a random process, how alike are the different samples? Define: 11 tXX and 22 tXX The autocorrelation is defined as:

2121212121 ,, xxfxxdxdxXXEttRXX

The above function is valid for all processes, stationary and non-stationary. For WSS processes:

XXXX RtXtXEttR 21, If the process is ergodic, the time average is equivalent to the probabilistic expectation, or

txtxdttxtx

T

T

TT

XX 2

1lim

and XXXX R

Define: 𝑥 𝑋 𝑘 and 𝑥 𝑋 𝑙

l kx x

lkXlkKK lkxxpmfxxlXkXElkR ,;,, **

For WSS

kx x

kXkKK kxxpmfxxnXnkXEXkXEkR0

0,;,00 0*

0**

If the process is ergodic, the sample average is equivalent to the probabilistic expectation, or

N

NnN

KK nXknXN

k *

12

1lim

As a note for things you’ve been computing, the “zeroth lag of the autocorrelation” is

221

212

211111 0, XXXXXX xfxdxXEXXERttR

22

2

1lim0 txdttx

T

T

TT

XX


Properties of Autocorrelation Functions

1) 220 XXERXX The mean squared value of the random process can be obtained by observing the zeroth lag of the autocorrelation function. 2) XXXX RR or kRkR XXXX The autocorrelation function is an even function in time. Only positive (or negative) needs to be computed for an ergodic WSS random process. 3) 0XXXX RR or 0XXXX RkR

The autocorrelation function is a maximum at 0. For periodic functions, other values may equal the zeroth lag, but never be larger. 4) If X has a DC component, then Rxx has a constant factor.

tNXtX

NNXX RXR 2

Note that the mean value can be computed from the autocorrelation function constants! 5) If X has a periodic component, then Rxx will also have a periodic component of the same period. Think of:

20,cos twAtX where A and w are known constants and theta is a uniform random variable.

wA

tXtXERXX cos2

2

5b) For signals that are the sum of independent random variable, the autocorrelation is the sum of the individual autocorrelation functions.

tYtXtW

YXYYXXWW RRR 2

For non-zero mean functions, (let w, x, y be zero mean and W, X, Y have a mean) YXYYXXWW RRR 2

YXYyyXxxWwwWW RRRR 2222

222 2 YYXXyyxxWwwWW RRRR

22YXyyxxWwwWW RRRR

Then we have

22YXW

yyxxww RRR


6) If X is ergodic and zero mean and has no periodic component, then we expect 0lim

XXR

7) Autocorrelation functions can not have an arbitrary shape. One way of specifying shapes permissible is in terms of the Fourier transform of the autocorrelation function. That is, if

dtjwtRR XXXX exp

then the restriction states that wallforRXX 0

Additional concept: tNatX

NNXX RatNtNEaR 22


The Crosscorrelation Function

For a two sample function defined by samples in time of two random processes, how alike are the different samples?

Define: 11 tXX and 22 tYY The cross-correlation is defined as:

2121212121 ,, yxfyxdydxYXEttRXY

2121212121 ,, xyfxydxdyXYEttRYX

The above function is valid for all processes, jointly stationary and non-stationary. For jointly WSS processes:

XYXY RtYtXEttR 21,

YXYX RtXtYEttR 21,

Note: the order of the subscripts is important for cross-correlation!

If the processes are jointly ergodic, the time average is equivalent to the probabilistic expectation, or

tytxdttytx

T

T

TT

XY 2

1lim

txtydttxty

T

T

TT

YX 2

1lim

and

XYXY R

YXYX R


Properties of Crosscorrelation Functions

1) The properties of the zoreth lag have no particular significance and do not represent mean-square values. It is true that the “ordered” crosscorrelations must be equal at 0. .

00 YXXY RR or 00 YXXY

2) Crosscorrelation functions are not generally even functions. However, there is an antisymmetry to the ordered crosscorrelations:

YXXY RR For

tytxdttytx

T

T

TT

XY 2

1lim

Substitute t

yxdyxT

T

TT

XY

2

1lim

YX

T

TT

XY xydxyT2

1lim

3) The crosscorrelation does not necessarily have its maximum at the zeroth lag. This makes sense if you are correlating a signal with a timed delayed version of itself. The crosscorrelation should be a maximum when the lag equals the time delay!

It can be shown however that 00 XXXXXY RRR

As a note, the crosscorrelation may not achieve the maximum anywhere …

4) If X and Y are statistically independent, then the ordering is not important YXtYEtXEtYtXERXY

and YXXY RYXR

Definition: PSD

Let Rxx(t) be an autocorrelation function for a WSS random process. The power spectral density is defined as the Fourier transform of the autocorrealtion function.


diwRRwS XXXXXX exp

0 wS XX

The power spectral density necessarily contains no phase information!

The inverse exists in the form of the inverse transform

dwiwtwStR XXXX exp2

1

Properties:

1. Sxx(w) is purely real as Rxx(t) is conjugate symmetric

2. If X(t) is a real-valued WSS process, then Sxx(w) is an even function, as Rxx(t) is real and even.

3. Sxx(w)>= 0 for all w.

Find the psd of the following autocorrelation function … of the random telegraph.

0,exp forRXX

Find a good Fourier Transform Table … otherwise

dwjRwS XXXX exp

2222

22

ww

wS XX


White Noise

𝑅 𝜏𝑁2∙ 𝛿 𝑡

𝑆 𝑓 𝑆𝑁2

For typical applications, we are interested in Band-Limited White Noise where

𝑆 𝑓 𝑆𝑁2

, |𝑓| 𝐵

0, 𝐵 |𝑓|

For xt

xtxt

sinc

𝑅 𝜏 2 ∙ 𝐵 ∙ 𝑠𝑖𝑛𝑐 2 ∙ 𝐵 ∙ 𝜏

The equivalent noise power is then:

𝐸 𝑋 𝑅 0 𝑆 ∙ 𝑑𝑓 2 ∙ 𝐵 ∙ 𝑆 2 ∙ 𝐵 ∙𝑁2

𝑁 ∙ 𝐵


Example Section 13.7 Amplitude Modulation.

𝑋 𝑡 𝐴 ∙ 1 𝛽 ∙ 𝑚 𝑡 ∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝑡 𝜃 Where m(t) is a random message, typically WSS, zero mean and bounded by +/-1. A is a an amplitude, f is the center frequency and there is a random phase angle (uniform distribution around a circle). The R.V. are independent.

𝐸 𝑋 𝑡 𝐸 𝐴 ∙ 1 𝛽 ∙ 𝑚 𝑡 ∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝑡 𝜃

𝐸 𝑋 𝑡 𝐴 ∙ 𝐸 1 𝛽 ∙ 𝑚 𝑡 ∙ 𝐸 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝑡 𝜃

𝐸 𝑋 𝑡 𝐴 ∙ 1 0 ∙ 0 0

Autocorrelation

𝐸 𝑋 𝑡 ∙ 𝑋 𝑡 𝜏 𝑅 𝑡, 𝑡 𝜏

𝑅 𝑡, 𝑡 𝜏 𝐸 𝐴 ∙ 1 𝛽 ∙ 𝑚 𝑡 ∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝑡 𝜃

∙ 𝐴 ∙ 1 𝛽 ∙ 𝑚 𝑡 𝜏 ∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝑡 𝜏 𝜃

𝐸 𝑋 𝑡 ∙ 𝑋 𝑡 𝜏 𝐴 ∙ 𝐸1 𝛽 ∙ 𝑚 𝑡 𝛽 ∙ 𝑚 𝑡 𝜏 𝛽 ∙ 𝑚 𝑡 ∙ 𝑚 𝑡 𝜏∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝑡 𝜃 ∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝑡 𝜏 𝜃

𝑅 𝑡, 𝑡 𝜏 𝐴 ∙ 1 𝛽 ∙ 𝑅 𝜏 ∙12∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝜏

𝑅 𝜏𝐴2∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝜏

𝐴2∙ 𝛽 ∙ 𝑅 𝜏 ∙ 𝑐𝑜𝑠 2𝜋 ∙ 𝑓 ∙ 𝜏

Forming the PSD

𝑆 𝑓𝐴4∙ 𝛿 𝑓 𝑓 𝛿 𝑓 𝑓

𝐴4∙ 𝛽 ∙ 𝑆 𝑓 ∗ 𝛿 𝑓 𝑓 𝛿 𝑓 𝑓

Which after performing the convolution becomes

𝑆 𝑓𝐴4∙ 𝛿 𝑓 𝑓 𝛿 𝑓 𝑓

𝐴4∙ 𝛽 ∙ 𝑆 𝑓 𝑓 𝑆 𝑓 𝑓

Your textbook simplifies the problem a bit … dealing with only the message and not the additional carrier component.


Remember the generic autocorrelation example from chap 13 notes …

2211 2cos2sin tfCtfBAtX

where the phase angles are uniformly distributed R.V from 0 to 2π.

2211

2211

2cos2sin

2cos2sin

tfCtfBAtX

tfCtfBAtXE

tXtXERXX

1122

2211

22222

2222

11112

11112

2sin2cos

2cos2sin

2cos2cos

2cos2cos

2sin2sin

2sin2sin

tftfBC

tftfBC

tftfC

tfACtfAC

tftfB

tfABtfABA

ERXX

2211

22222

11112

2

22sin

2cos2cos

2sin2sin

tftfBC

tftfC

tftfB

A

ERXX

With practice, we can see that the above math becomes

2222

11122

222cos2

12cos

2

1

222cos2

12cos

2

1

tffEC

tffEBARXX

which lead to

2

2

1

22 2cos

22cos

2f

Cf

BARXX

Forming the PSD

And then taking the Fourier transform

22

2

11

22

2

1

2

1

22

1

2

1

2ffff

Cffff

BfAfS XX

22

2

11

22

44ffff

Cffff

BfAfS XX

We also know from the before


dffSdwwSX XXXX2

12

Therefore, the 2nd moment can be immediately computed as

dfffff

Cffff

BfAX 22

2

11

222

44

22

24

24

222

2222 CB

ACB

AX

We can also see that the mean value becomes

AtfCtfBAEX 2211 2cos2sin

So, the variance is

2222

222

2222 CB

ACB

A

A is a “DC” term whereas B and C are “AC” terms as would be expected from X(t).


Trig Identities

By the way, it is useful to have basic trig identities handy when dealing with this stuff …

bababa cos2

1cos

2

1sinsin

bababa cos2

1cos

2

1coscos

bababa sin2

1sin

2

1cossin

bababa sin2

1sin

2

1sincos

and aaa cossin22sin

1cos2sincos2cos 222 aaaa

as well as

aa 2cos12

1sin 2

aa 2cos12

1cos 2

Please review Homework #9 Skills #5, #6, and #7.

Exam #3 Review - Western Michigan Universitybazuinb/ECE3800/Exam3_Review.pdf · B.J. Bazuin, Fall...

Documents

Transcript of Exam #3 Review - Western Michigan Universitybazuinb/ECE3800/Exam3_Review.pdf · B.J. Bazuin, Fall...