Discrete Random Variables - Imperial College Londonnaheard/Comp245/discrete_random... · 2018. 1....

Discrete Random VariablesCOMP 245 STATISTICS

Dr N A Heard

Room 545 Huxley Building

Dr N A Heard (Room 545 Huxley Building) Discrete Random Variables 1 / 62

Definition

Suppose, as before, that for a random experiment we have identified asample space S and a probability measure P(E) defined on(measurable) subsets E ⊆ S.A random variable is a mapping from the sample space to the realnumbers. So if X is a random variable, X : S→ R. Each element of thesample space s ∈ S is assigned by X a (not necessarily unique)numerical value X(s).If we denote the unknown outcome of the random experiment as s∗,then the corresponding unknown outcome of the random variableX(s∗) will be generically referred to as X.The probability measure P already defined on S induces a probabilitydistribution on the random variable X in R:


For each x ∈ R, let Sx ⊆ S be the set containing just those elements of Swhich are mapped by X to numbers no greater than x. That is, let Sx bethe inverse image of (−∞, x] under the function X. Then noting theequaivalence

X(s∗) ≤ x ⇐⇒ s∗ ∈ Sx.

we see thatPX(X ≤ x) ≡ P(Sx).


The image of S under X is called the range of the random variable:

Defn.range(X) ≡ X(S) = {x ∈ R|∃s ∈ S s.t. X(s) = x}

So as S contains all the possible outcomes of the experiment, range(X)contains all the possible outcomes for the random variable X.


Example

Let our random experiment be tossing a fair coin, with sample space

{H, T} and probability measure P({H}) = P({T}) = 12

.

We can define a random variable X : {H, T} → R taking values, say,

X(T) = 0,X(H) = 1.

In this case, what does Sx look like for each x ∈ R?


Sx =

∅ if x < 0;{T} if 0 ≤ x < 1;{H, T} if x ≥ 1.

This defines a range of probabilities PX on the continuum R

PX(X ≤ x) = P(Sx) =

P(∅) = 0 if x < 0;P({T}) = 1

2 if 0 ≤ x < 1;P({H, T}) = 1 if x ≥ 1.


-

6

1

12

01 x

PX(X ≤ x)


cdf

The cumulative distribution function (cdf) of a random variable X,written FX(x) (or just F(x)) is the probability that X takes value lessthan or equal to x.

Defn.FX(x) = PX(X ≤ x)

For any random variable X, FX is right-continuous, meaning if adecreasing sequence of real numbers x1, x2, . . .→ x, thenFX(x1), FX(x2), . . .→ FX(x).


cdf Properties

For a given function FX(x), to check this is a valid cdf, we need tomake sure the following conditions hold.

1 0 ≤ FX(x) ≤ 1, ∀x ∈ R;2 Monotonicity: ∀x1, x2 ∈ R, x1 < x2 ⇒ FX(x1) ≤ FX(x2);3 FX(−∞) = 0, FX(∞) = 1.


For finite intervals (a, b] ⊆ R, it is easy to check that

PX(a < X ≤ b) = FX(b)− FX(a).

Unless there is any ambiguity, we generally suppress the subscriptof PX(·) in our notation and just write P(·) for the probabilitymeasure for the random variable.

I That is, we forget about the underlying sample space and just thinkabout the random variable and its probabilities.

I Often, it will be most convenient to work this way and consider therandom variable directly from the very start, with the range of Xbeing our sample space.


Simple Random Variables

We say a random variable is simple if it can take only a finite numberof possible values. That is,

Defn.X is simple ⇐⇒ range(X) is finite.

Suppose X is simple, and can take one of m values X = {x1, x2, . . . , xm}ordered so that x1 < x2 < . . . < xm. Each sample space element s ∈ S ismapped by X to one of these m values.Then in this case, we can partition the sample space S into m disjointsubsets {E1, E2, . . . , Em} so that s ∈ Ei ⇐⇒ X(s) = xi, i = 1, 2, . . . , m.


We can then write down the probability of the random variable Xtaking the particular value xi as

PX(X = xi) ≡ P(Ei).

It is also easy to check that

PX(X = xi) = FX(xi)− FX(xi−1),

where we can take x0 = −∞.


Examples of Simple Random Variables

Consider once again the experiment of rolling a single die.Then S = { , , , , , } and for any s ∈ S (e.g. s = ) we haveP({s}) = 1

6 .An obvious random variable we could define on S would beX : S→ R, s.t.

X( ) = 1,X( ) = 2,

...X( ) = 6.

Then e.g. PX(1 < X ≤ 5) = P({ , , , }) = 46 = 2

3and PX(X ∈ {2, 4, 6}) = P({ , , }) = 1

2 .


Alternatively, we could define a random variable Y : S→ R, s.t.

Y( ) = Y( ) = Y( ) = 0,Y( ) = Y( ) = Y( ) = 1.

Then clearly PY(Y = 0) = P({ , , }) = 12 and

PY(Y = 1) = P({ , , }) = 12 .


Comments

Note that under either random variable X or Y, we still got thesame probability of getting an even number of spots on the die.

Indeed, this would be the case for any random variable we maycare to define.

A random variable is simply a numeric relabelling of ourunderlying sample space, and all probabilities are derived fromthe associated underlying probability measure.


Discrete Random Variables

A simple random variable is a special case of a discrete random variable.We say a random variable is discrete if it can take only a countablenumber of possible values. That is,

Defn.X is discrete ⇐⇒ range(X) is countable.

Suppose X is discrete, and can take one of the countable set of valuesX = {x1, x2, . . .} ordered so that x1 < x2 < . . .. Each sample spaceelement s ∈ S is mapped by X to one of these values.Then in this case, we can partition the sample space S into a countablecollection of disjoint subsets {E1, E2, . . .} s.t.s ∈ Ei ⇐⇒ X(s) = xi, i = 1, 2, . . ..


As with simple random variables we can write down the probability ofthe discrete random variable X taking the particular value xi as

PX(X = xi) ≡ P(Ei).

We again havePX(X = xi) = FX(xi)− FX(xi−1),

For a discrete random variable X, FX is a monotonic increasing stepfunction with jumps only at points in X .


Example: Poisson(5) cdf

●

●

●

●

●

●

●

●

●

●

●● ● ● ● ● ● ● ● ● ●

0 5 10 15 20

0.0

0.2

0.4

0.6

0.8

1.0

x

F(x

)


Probability Mass Function

For a discrete random variable X and x ∈ R, we define the probabilitymass function (pmf), pX(x) (or just p(x)) as

Defn.pX(x) = PX(X = x)


pmf Properties

To check we have a valid pmf, we need to make sure the followingconditions hold.

If X can take values X = {x1, x2, . . .} then we must have:

1 0 ≤ pX(x) ≤ 1, ∀x ∈ R;2 ∑

x∈XpX(x) = 1.


Example: Poisson(5) pmf

0 5 10 15 20

0.00

0.05

0.10

0.15

x

p(x)


Knowing either the pmf or cdf of a discrete random variablecharacterises its probability distribution.

That is, from the pmf we can derive the cdf, and vice versa.

p(xi) = F(xi)− F(xi−1),

F(xi) =i

∑j=1

p(xj).


Links with Statistics

We can now see the first links between the numerical summaries andgraphical displays we saw in earlier lectures and probability theory:

We can often think of a set of data (x1, x2, . . . , xn) as n realisations of arandom variable X defined on an underlying population for the data.

Recall the normalised frequency counts we considered for a set ofdata, known as the empirical probability mass function. This canbe seen as an empirical estimate for the pmf of their underlyingpopulation.Also recall the empirical cumulative distribution function. Thistoo is an empirical estimate, but for the cdf of the underlyingpopulation.


E(X)

For a discrete random variable X we define the expectation of X,

Defn.

EX(X) = ∑x

xpX(x).

EX(X) (often just written E(X) or even µX) is also referred to as themean of X.

It gives a weighted average of the possible values of the randomvariable X, with the weights given by the probabilities of eachoutcome.


Examples

1 If X is a r.v. taking the integer value scored on a single roll of a fairdie, then

E(X) =6

∑x=1

xp(x)

= 1.16+ 2.

16+ 3.

16+ 4.

16+ 5.

16+ 6.

16=

216

= 3.5.

2 If now X is a score from a student answering a single multiplechoice question with four options, with 3 marks awarded for acorrect answer, -1 for a wrong answer and 0 for no answer, what isthe expected value if they answer at random?

E(X) = 3.P(Correct) + (−1).P(Incorrect) = 3.14− 1.

34= 0.


E{g(X)}

More generally, for a function of interest g : R→ R of the randomvariable X, first notice that the composition g(X),

g(X)(s) = (g ◦X)(s)

is also a random variable. It follows that

Defn.

EX{g(X)} = ∑x

g(x)pX(x) (1)


Linearity of Expectation

Consider the linear function g(X) = aX + b for constants a, b ∈ R. Wecan see from (1) that

EX(aX + b) = ∑x(ax + b)pX(x)

= a ∑x

xpX(x) + b ∑x

pX(x)

and since ∑x xpX(x) = E(X) and ∑x pX(x) = 1 we have

E(aX + b) = aE(X) + b, ∀a, b ∈ R.


It is equally easy to check that for g, h : R→ R, we have

E{g(X) + h(X)} = E{g(X)}+ E{h(X)}.


Var(X)

Consider another special case of g(X), namely

g(X) = {X− E(X)}2.

The expectation of this function wrt PX gives a measure of dispersionor variability of the random variable X around its mean, called thevariance and denoted VarX(X) (or sometimes σ2

X):

Defn.

VarX(X) = EX[{X− EX(X)}2].


We can expand the expression {X− E(X)}2 and exploit the linearity ofexpectation to get an alternative formula for the variance.

{X− E(X)}2 = X2 − 2E(X)X + {E(X)}2

⇒ Var(X) = E[X2 − {2E(X)}X + {E(X)}2]

= E(X2)− 2E(X)E(X) + {E(X)}2

and hence

Var(X) = E(X2)− {E(X)}2.


Variance of a Linear Function of a Random Variable

We saw earlier that for constants a, b ∈ R the linear combination aX + bhad expectation aE(X) + b. What about the variance?

It is easy to show that the corresponding result is

Var(aX + b) = a2Var(X), ∀a, b ∈ R.


sd(X)

The standard deviation of a random variable X, written sdX(X) (orsometimes σX), is the square root of the variance.

Defn.

sdX(X) =√

VarX(X).


Skewness

The skewness (γ1) of a discrete random variable X is given by

Defn.

γ1 =EX[{X− EX(X)}3]

sdX(X)3 .

That is, if µ = E(X) and σ = sd(X),

γ1 =E[(X− µ)3]

σ3 .


Examples1 If X is a r.v. taking the integer value scored with a single roll of a

fair die, then

Var(X) =6

∑x=1

x2p(x)− 3.52

= 12.16+ 22.

16+ . . . + 62.

16− 3.52 = 1.25.

2 If now X is a score from a student answering a single multiplechoice question with four options, with 3 marks awarded for acorrect answer, -1 for a wrong answer and 0 for no answer, what isthe standard deviation if they answer at random?

E(X2) = 32.P(Correct) + (−1)2.P(Incorrect) = 9.14+ 1.

34= 3

⇒ sd(X) =√

3− 02 =√

3.


Links with Statistics

We have met three important quantities for a random variable, definedthrough expectation - the mean µ, the variance σ2 and the standarddeviation σ.

Again we can see a duality with the corresponding numericalsummaries for data which we met - the sample mean x̄, the samplevariance s2 and the sample standard deviation s.

The duality is this: If we were to consider the data sample as thepopulation and draw a random member from that sample as a randomvariable, this r.v. would have cdf Fn(x), the empirical cdf. The mean ofthe r.v. µ = x̄, variance σ2 = s2 and standard deviation σ = s.


Expectation of Sums of Random Variables

Let X1, X2, . . . , Xn be n random variables, perhaps with differentdistributions and not necessarily independent.

Let Sn =n

∑i=1

Xi be the sum of those variables, andSn

nbe their average.

Then the mean of Sn is given by

E(Sn) =n

∑i=1

E(Xi), E(

Sn

n

)=

∑ni=1 E(Xi)

n.


Variance of Sums of Random Variables

However, for the variance of Sn, only if X1, X2, . . . , Xn are independentdo we have

Var(Sn) =n

∑i=1

Var(Xi), Var(

Sn

n

)=

∑ni=1 Var(Xi)

n2 .

So if X1, X2, . . . , Xn are independent and identically distributed withE(Xi) = µX and Var(Xi) = σ2

X we get

E(

Sn

n

)= µX, Var

(Sn

n

)=

σ2Xn

.


Bernoulli(p)

Consider an experiment with only two possible outcomes, encoded asa random variable X taking value 1, with probability p, or 0, withprobability (1− p), accordingly.(Ex.: Tossing a coin, X = 1 for a head, X = 0 for tails, p = 1

2 .)Then we say X ∼ Bernoulli(p) and note the pmf to be

p(x) = px(1− p)1−x, x = 0, 1.

Using the formulae for mean and variance, it follows that

µ = p, σ2 = p(1− p).


Example: Bernoulli(¼) pmf


Binomial(n, p)

Consider n identical, independent Bernoulli(p) trials X1, . . . , Xn.Let X = ∑n

i=1 Xi be the total number of 1s observed in the n trials.(Ex.: Tossing a coin n times, X is the number of heads obtained, p = 1

2 .)

Then X is a random variable taking values in {0, 1, 2, . . . , n}, and wesay X ∼ Binomial(n, p).

From the Binomial Theorem we find the pmf to be

p(x) = (nx)p

x(1− p)n−x, x = 0, 1, 2, . . . , n.


To calculate the Binomial pmf we recall that(

nx

)=

n!x!(n− x)!

and

x! =x

∏i=1

i. (Note 0! = 1.)

It can be shown, either directly from the pmf or from the results forsums of random variables, that the mean and variance are

µ = np, σ2 = np(1− p).

Similarly, the skewness is given by

γ1 =1− 2p√np(1− p)

.


Example: Binomial(20,¼) pmf

Go to Poi(5) pmf


ExampleSuppose that 10 users are authorised to use a particular computersystem, and that the system collapses if 7 or more users attempt to logon simultaneously. Suppose that each user has the same probabilityp = 0.2 of wishing to log on in each hour.What is the probability that the system will crash in a given hour?

Solution: The probability that exactly x users will want to log on in anyhour is given by Binomial(n, p) = Binomial(10, 0.2).Hence the probability of 7 or more users wishing to log on in any houris

p(7) + p(8) + p(9) + p(10)

=

(107

)0.270.83 + . . . +

(1010

)0.2100.80

= 0.00086.


Two more examples

A manufacturing plant produces chips with a defect rate of 10%.The quality control procedure consists of checking samples of size50. Then the distribution of the number of defectives is expectedto be Binomial(50,0.1).When transmitting binary digits through a communicationchannel, the number of digits received correctly out of ntransmitted digits, can be modelled by a Binomial(n, p), where p isthe probability that a digit is transmitted incorrectly.

Note: Recall the independence condition necessary for these models tobe reasonable.


Geometric(p)

Consider a potentially infinite sequence of independent Bernoulli(p)random variables X1, X2, . . ..Suppose we define a quantity X by

X = min{i|Xi = 1}

to be the index of the first Bernoulli trial to result in a 1.(Ex.: Tossing a coin, X is the number of tosses until the first head isobtained, p = 1

2 .)

Then X is a random variable taking values in Z+ = {1, 2, . . .}, and wesay X ∼ Geometric(p).


Clearly the pmf is given by

p(x) = p(1− p)x−1, x = 1, 2, . . ..

The mean and variance are

µ =1p

, σ2 =1− p

p2 .

The skewness is given by

γ1 =2− p√

1− p,

and so is always positive.


Example: Geometric(¼) pmf


Alternative Formulation

If X ∼ Geometric(p), let us consider Y = X− 1.Then Y is a random variable taking values in N = {0, 1, 2, . . .}, andcorresponds to the number of independent Bernoulli(p) trials before weobtain our first 1. (Some texts refer to this as the Geometricdistribution.)Note we have pmf

pY(y) = p(1− p)y, y = 0, 1, 2, . . .,

and the mean becomes

µY =1− p

p.

while the variance and skewness are unaffected by the shift.


Example

Suppose people have problems logging onto a particular website onceevery 5 attempts, on average.

1 Assuming the attempts are independent, what is the probabilitythat an individual will not succeed until the 4th?p =

45= 0.8. p(4) = (1− p)3p = 0.230.8 = 0.0064.

2 On average, how many trials must one make until succeeding?

Mean=1p=

54= 1.25.

3 What’s the probability that the first successful attempt is the 7th orlater?

p(7) + p(8) + p(9) + . . . =p(1− p)6

1− (1− p)= (1− p)6 = 0.26.


Example (contd. from Binomial)

Again suppose that 10 users are authorised to use a particularcomputer system, and that the system collapses if 7 or more usersattempt to log on simultaneously. Suppose that each user has the sameprobability p = 0.2 of wishing to log on in each hour.

Using the Binomial distribution we found the probability that thesystem will crash in any given hour to be 0.00086.

Using the Geometric distribution formulae, we are able to answerquestions such as: On average, after how many hours will the systemcrash?

Mean = 1p = 1

0.00086 = 1163 hours.


Example: Mad(?) Dictators and Birth Control

A dictator, keen to maximise the ratio of males to females in hiscountry (so he could build up his all male army) ordered that eachcouple should keep having children until a boy was born and thenstop.

Calculate the expected number of boys that a couple will have, and theexpected number of girls, given that P(boy)=½.


Assume for simplicity that each couple can have arbitrarily manychildren (although this is not necessary to get the following results).Then since each couple stops when 1 boy is born, the expected numberof boys per couple is 1.

On the other hand, if Y is the number of girls given birth to by acouple, Y clearly follows the alternative formulation for theGeometric(½) distribution.

So the expected number of girls for a couple is1− 1

212

= 1.


Poi(λ)

Let X be a random variable on N = {0, 1, 2, . . .} with pmf

p(x) =e−λλx

x!, x = 0, 1, 2, . . .,

for some λ > 0.

Then X is said to follow a Poisson distribution with rate parameter λand we write X ∼ Poi(λ).

Poisson random variables are concerned with the number of randomevents occurring per unit of time or space, when there is a constantunderlying probability ‘rate’ of events occurring across this unit.


Examples

the number of minor car crashes per day in the U.K.;the number of mistakes on each of my slides;the number of potholes in each mile of road;the number of jobs which arrive at a database server per hour;the number of particles emitted by a radioactive substance in agiven time.


An interesting property of the Poisson distribution is that it has equalmean and variance, namely

µ = λ, σ2 = λ.

The skewness is given by

γ1 =1√λ

,

so is always positive but decreasing as λ ↑.


Example: Poi(5) pmf (again)

Go to Binomial(20,¼) pmf


Poisson Approximation to the Binomial

Notice the similarity between the pmf plots we’ve seen forBinomial(20,¼) and Poi(5).

It can be shown that for Binomial(n, p), when p is small and n is large,this distribution can be well approximated by the Poisson distributionwith rate parameter np, Poi(np).

p in the above is not small, we would typically prefer p < 0.1 for theapproximation to be useful.

The usefulness of this approximation is in using probability tables;tabulating a single Poisson(λ) distribution encompasses an infinitenumber of possible corresponding Binomial distributions,Binomial(n, λ

n ).


Ex.A manufacturer produces VLSI chips, of which 1% are defective. Findthe probability that in a box of 100 chips none are defective.

We want p(0) from Binomial(100,0.01). Since n is large and p is small,we can approximate this distribution by Poi(100× 0.01) ≡ Poi(1).

Then p(0) ≈ e−1λ0

0!= 0.3679.


Fitting a Poisson Distribution to Data

Ex.The number of particles emitted by a radioactive substance whichreached a Geiger counter was measured for 2608 time intervals, eachof length 7.5 seconds.The (real) data are given in the table below:

x 0 1 2 3 4 5 6 7 8 9 ≥10nx 57 203 383 525 532 408 273 139 45 27 16

Do these data correspond to 2608 independent observations of anidentical Poisson random variable?


The total number of particles, ∑x xnx, is 10,094, and the total number ofintervals observed, n = ∑x nx, is 2608, so that the average numberreaching the counter in an interval is 10094

2608 = 3.870.

Since the mean of Poi(λ) is λ, we can try setting λ = 3.87 and see howwell this fits the data.

For example, considering the case x = 0, for a single experimentinterval the probability of observing 0 particles would bep(0) = e−3.873.870

0! = 0.02086. So over n = 2608 repetitions, our(Binomial) expectation of the number of 0 counts would ben× p(0) = 54.4.


Similarly for x = 1, 2, . . ., we obtain the following table of expectedvalues from the Poi(3.87) model:

x 0 1 2 3 4 5 6 7 8 9 ≥10O(nx) 57 203 383 525 532 408 273 139 45 27 16E(nx) 54.4 210.5 407.4 525.5 508.4 393.5 253.8 140.3 67.9 29.2 17.1

(O=Observed, E=Expected).

The two sets of numbers appear sufficiently close to suggest thePoisson approximation is a good one. Later, when we come to look athypothesis testing, we will see how to make such judgementsquantitatively.


U({1, 2, . . . , n})

Let X be a random variable on {1, 2, . . . , n} with pmf

p(x) =1n

, x = 1, 2, . . . , n.

Then X is said to follow a discrete uniform distribution and we writeX ∼ U({1, 2, . . . , n}).

The mean and variance are

µ =n + 1

2, σ2 =

n2 − 112

.

and the skewness is clearly zero.


Discrete Random Variables - Imperial College Londonnaheard/Comp245/discrete_random... · 2018. 1....

Documents

Transcript of Discrete Random Variables - Imperial College Londonnaheard/Comp245/discrete_random... · 2018. 1....