CS433 Modeling and Simulation Lecture 04 Statistical Models 1 Dr. Anis Koubâa Al-Imam Mohammad Ibn...

CS433Modeling and Simulation

Lecture 04

Statistical Models

1

Dr. Anis Koubâa

Al-Imam Mohammad Ibn Saud UniversityAl-Imam Mohammad Ibn Saud University

http://10.2.230.10:4040/akoubaa/cs433/

27 Oct 2008

2

Goals for Today

2

Understand the difference between discrete and continuous random variables

Review of the most common statistical models

Understand how to determine the empirical distribution from a statistical sample.

3

Topics

3

Discrete Random Variable Continuous Random Variable Discrete Probability Distributions

Binomial Distribution Bernoulli Distribution Discrete Poisson Distribution

Continuous Probability Distribution Uniform Exponential Normal Weibull Lognormal

Empirical Distributions

4

Discrete and Continuous Random Variables

5

Discrete Random Variables X is a discrete random variable if the number of

possible values of X (the sample space) is finite. Example: Consider jobs arriving at a job shop.

Let X be the number of jobs arriving each week at a job shop. Rx = possible values of X (range space of X) = {0,1,2,…}

p(xi) = probability the random variable is xi = p(X = xi)

The collection of pairs [xi, p(xi)], i = 1,2,…, is called the probability distribution of X,

p(xi) is called the probability mass function (PMF) of X.

Characteristics of the PMF: p(xi), i = 1,2, … must satisfy:

1

1. ( ) 0, for all

2. ( ) 1

i

ii

p x i

p x

6

Continuous Random Variables

X is a continuous random variable if its range space Rx is an interval or a collection of intervals.

The probability that X lies in the interval [a,b] is given by:

Where f(x) is the probability density function (PDF). Characteristics of the PDF: f(x) must satisfies:

Properties

X

R

X

Rxxf

dxxf

Rxxf

X

in not is if ,0)( 3.

1)( 2.

in allfor , 0)( 1.

b

adxxfbXaP )()(

)()()()( .2

0)( because ,0)( 1.0

00

bXaPbXaPbXaPbXaP

dxxfxXPx

x

7

Discrete versus Continuous Random Variables

Discrete Random VariableContinuous Random Variable

Finite Sample Space e.g. {0, 1, 2, 3}

Infinite Sample Space e.g. [0,1], [2.1, 5.3]

1

1. ( ) 0, for all

2. ( ) 1

i

ii

p x i

p x

i ip x P X x

Cumulative Distribution Function (CDF)

f xProbability Density Function (PDF)

X

R

X

Rxxf

dxxf

Rxxf

X

in not is if ,0)( 3.

1)( 2.

in allfor , 0)( 1.

p X x

( )i

ix xp X x p x

0

xp X x f t dt

b

ap a X b f x dx

Probability Mass Function (PMF)

8

Five Minutes Break

You are free to discuss with your classmates about the previous slides, or to refresh a bit, or to ask questions.

Administrative issues• Groups Formation• Choose a “class coordinator”

9

Expectation

The expected value (the mean) of X is denoted by E(X) If X is discrete

If X is continuous

A measure of the central tendency The variance of X is denoted by V(X) or var(X) or 2

Definition: Also, A measure of the spread or variation of the possible values of X

around the mean The standard deviation of X is denoted by

Definition: square root of V(X) Expressed in the same units as the mean

i

ii xpxxE all

)()(

dxxxfxE )()(

2V X E X E X

22V X E X E X

10

Example: Continuous Random Variables

Example: modeling the lifetime of a device

Time is a continuous random variable Random Time is typically modeled as exponential distribution We assume that with average lifetime of a device is 2 years

Probability that the device’s life is between 2 and 3 years is:

otherwise ,0

0 x,2

1)(

2/xexf

14.02

1)32(

3

2

2/ dxexP x

11


Cumulative Distribution Function: A device has the CDF:

The probability that the device lasts for less than 2 years:

The probability that it lasts between 2 and 3 years:

2/

0

2/ 12

1)( xx t edtexF

632.01)2()0()2()20( 1 eFFFXP

145.0)1()1()2()3()32( 1)2/3( eeFFXP

12


Example: The mean of life of the previous device is:

To compute variance of X, we first compute E(X2):

Hence, the variance and standard deviation of the device’s life are:

/ 2 /2

0 00

1 /2( ) 2

2x xx

E X xe dx e dxxe

82/22

1)(

0

2/

00

2/22

dxexdxexXE xx ex

2)(

428)( 2

XV

XV

Expected Value and Variance

13

Discrete Probability Distributions

Bernoulli Trials Binomial Distribution Geometric Distribution Poisson Distribution Poisson Process

14

Discrete Distributions

14

Discrete random variables are used to describe random phenomena in which only integer values can occur.

In this section, we will learn about: Bernoulli trials and Bernoulli distribution Binomial distribution Geometric and negative binomial

distribution Poisson distribution

15

Modeling of Random Events with Two-States

Bernoulli Trials Binomial Distribution

16

Bernoulli Trials

In the theory of probability and statistics, a Bernoulli trial is an experiment whose outcome is random and can be either of two possible outcomes, "success" and "failure".

In practice it refers to a single experiment, which can have one of two possible outcomes. These events can be phrased into “yes” or “no” questions: Did the coin land heads? Was the newborn child or a girl? Were a person's eyes green?

17

Bernoulli Distribution Consider an experiment consisting of n trials, each

can be a success or a failure. Let Xj = 1 if the jth experiment is a success

and Xj = 0 if the jth experiment is a failure

The Bernoulli distribution (one trial):

Bernoulli process It is the n Bernoulli trials where trials are independent:

, 1, 1, 2,...,

PMF: ( ) ( ) 1 , 0 1 2

0, otherwise

j

j j j j

p x j n

p x p x p q x , j , ,..., n

Expected Value: jE X p 2Variance : 1jV X p p

1 1 1 2, ,..., , ...n np X X X p X p X p X

18

Binomial Distribution

A binomial random variable is the number of successes in a series of n trials. Example: the number of 'heads' occurring when a coin is

tossed 50 times. A discrete random variable X is said to follow a

Binomial distribution with parameters n and p, written X ~ Bi(n,p) or X ~ B(n,p) if it has the probability distribution:

where x = 0, 1, 2, ......., n n = 1, 2, 3, ....... p = success probability; 0 < p < 1

1n kkn

P X k p pk

!

! !

n n

k k n k

Where

Expected Value: E X n p

2Variance : 1V X n p p

19


19

The trials must meet the following requirements: the total number of trials is fixed in

advance; there are just two outcomes of each

trial; success and failure; the outcomes of all the trials are

statistically independent; all the trials have the same

probability of success.

20

The number of successes in n Bernoulli trials, X, has a binomial distribution.

The formula can be understood as follows: we want k successes (pk) and n − k failures (1 − p)n − k. However, the k successes can occur anywhere among the n trials, and there are C(n, k) different ways of distributing k successes in a sequence of n trials.

otherwise ,0

,...,2,1,0k , )(

nqpk

nkXp

knk

The number of outcomes having

the required number of

successes and failures

Probability that there are

x successes and (n-x) failures


21

End of Part 01

Administrative issues• Groups Formation• Choose a “class coordinator”

22

Modeling of Discrete Random Time

Geometric Distribution

23

Geometric Distribution

Geometric Distribution represents the number X of Bernoulli trials to achieve the FIRST SUCCESS. It is used to represent random time until a first transition occurs

1 , 0,1,2,...,PMF: ( )

0, otherwise

kq p k np X k

1Expected Value : E X

p

22 2

1Variance :

q pV X

p p

PMF

k

CDF: F 1 1k

X p X k p

24

Negative Binomial Distribution

25

Negative Binomial Distribution

The negative binomial distribution is a discrete probability distribution that can be used to describe the distribution arising from an experiment consisting of a sequence of independent trials, subject to several constraints.

Firstly each trial results in success or failure, the probability of success for each trial, p, is constant across the experiment and finally the experiment continues until a fixed number of successes have been achieved.

Negative Binomial Distribution The number of Bernoulli trials, X, until the kth success If X is a negative binomial distribution with parameters p

and r, then: 11 , 1, 2,3...

, :

0, otherwise

krk rp p k

PMF r p p X k k

1Expected Value :

pE X r

p

2

2

1Variance :

r pV X

p

26

Modeling of Random Number of Arrivals/Events

Poisson Distribution Poisson Process

27

Poisson Distribution

the Poisson distribution is a discrete probability distribution that expresses the probability of a number of events occurring in a fixed period of time if these events occur with a known average rate and independently of the time since the last event.

Poisson random variable represents the count of the number of events that occur in a certain time interval or spatial area.

Example: The number of cars passing a fixed point in a 5 minute

interval, The number of calls received by a switchboard during a

given period of time. The number of message coming to a router in a given

period of time

28

Discrete Poisson Distribution

A discrete random variable X is said to follow a Poisson distribution with parameter , written X ~ Po(), if it has probability distribution

The PMF represents the probability that there are k arrivals in a certain period of time.

where X = 0, 1, 2, ..., n > 0 is called the arrival rate.

PMF: exp!

k

P X kk

29

Poisson distribution describes many random processes quite well and is mathematically quite simple.

The Poisson distribution with the parameter is characterized by:

exp for 0,1,2, ....PMF: !

0, otherwise

k

kp k P X k k

0

CDF: exp!

k i

i

F k p X ki


Expected value: E X

Variance: V X

PMF

CDF

30


The following requirements must be met in the

Poisson Distribution:

the length of the observation period is fixed in

advance;

the events occur at a constant average rate;

the number of events occurring in disjoint

intervals are statistically independent.

31

Example: Poisson Distribution

The number of cars that enter the parking follows a Poisson distribution with a mean rate equal to = 20 cars/hour

The probability of having exactly 15 cars entering the parking in one hour:

or

The probability of having more than 3 cars entering the parking in one hour:

1520

15 15 exp 20 0.05164915!

p P X

15 15 14 0.156513 0.104864 0.051649p F F

3 1 3 1 3

1 0 1 2 3

0.9999967

p X p X F

p p p p

USE EXCEL/MATLAB

FOR COMPUTATIONS

32

Example: Poisson Distribution

Probability Mass FunctionPoisson ( = 20 cars/hour)

Cumulative Distribution FunctionPoisson ( = 20 cars/hour)

0

20 exp 20

!

k i

i

F k p X ki

20exp 20

!

k

p X kk

33

Five Minutes Break


Administrative issues• Groups Formation

34


Poisson Distribution Poisson Process

35

Poisson Process

Wikipedia: A Poisson process, named after the French mathematician Siméon-Denis Poisson (1781 – 1840), is the stochastic process in which events (e.g. arrivals) occur continuously and independently of one another.

Formal Definition: The Poisson Process is a counting function {N(), ≥0} where N() is the number of events that have occurred up to time t , i.e. in the interval [0,].

Fact: The number of events between time a and time b is given as N(b) − N(a) and has a Poisson distribution.

The Poisson process is a continuous-time process: Time is continuous Its discrete-time counterpart is the Bernoulli process

Bernoulli process is a discrete-time stochastic process consisting of a sequence of independent random variables taking values over two symbols.

36

Examples of using Poisson Process

The number of web page requests arriving at a server may be characterized by a Poisson process except for unusual circumstances such as coordinated denial of service attacks.

The number of telephone calls arriving at a switchboard, or at an automatic phone-switching system, may be characterized by a Poisson process.

The number of raindrops falling over a wide spatial area may be characterized by a spatial Poisson process.

The arrival of "customers" is commonly modelled as a Poisson process in the study of simple queueing systems.

The execution of trades on a stock exchange, as viewed on a tick by tick basis, is a Poisson process.

37

(Homogenous) Poisson Process

The homogeneous Poisson process is characterized by a CONSTANT rate parameter λ, also known as intensity, such that the number of events in time interval follows a Poisson distribution with associated parameter .

Formally, A counting process is a (homogenous) Poisson process with mean rate if:

describes the number of events in time interval The mean and the variance are equal

for 0 and 0,1,2,...

( )PMF: ( ) exp

!

n

t n

p N t N t n p N nn

E N V N

, 0N

N t N t ,t t

,t t

38

(Homogenous) Poisson Process

Properties of Poisson process Arrivals occur one at a time (not

simultaneous) has stationary increments,

which means The number of arrivals in time s to t is also

Poisson-distributed with mean has independent increments

N t N s N t s

t s

, 0N

, 0N

39

Inter-arrival time: time between two consecutive arrivals The inter-arrival times of a Poisson process are random.

What is its distribution? Consider the inter-arrival times of a Poisson process (A1, A2, …),

where Ai is the elapsed time between arrival i and arrival i+1

The first arrival occurs after time t MEANS that there are no arrivals in the interval [0,t], As a consequence:

39

1 0 expp A t p N t t

1 11 1 expp A t p A t t

Inter-Arrival Times of a Poisson Process

The Inter-arrival times of a Poisson process are exponentially distributed and independent with

mean 1/

Brian

Poi is not an abbreviation of Poisson that I have ever seen

40

Splitting and Pooling

Splitting A Poisson process can be split into two Poisson processes:

The first with a probability p and the second with probability 1-p.

where and are both Poisson processes with rates and

Pooling The summation of two Poisson processes is a Poisson process , where is a Poisson processes with

rates

N(t) ~ Poi()

N1(t) ~ Poi[p]

N2(t) ~ Poi[(1-p)]

p

(1-p)

N(t) ~ Poi(12)

N1(t) ~ Poi[]

N2(t) ~ Poi[]

1

2

1 2N t N t N t

p 1 p 1N t 2N t 1 2N t N t N t

N t 1 2

41


Poisson Distribution Non Homogenous Poisson Process

42

Non Homogenous (Non-stationary) Poisson Process (NSPP)

The non homogeneous Poisson process is characterized by a VARIABLE rate parameter (t)λ , the arrival rate at time t. In general, the rate parameter may change over time.

The stationary increments, property is not satisfied

The expected number of events (e.g. arrival) between time s and time t is,

t

s t sλ(u) du

, : s t N t N s N t s

1 2 3

43

Example: Non-stationary Poisson Process (NSPP)

The number of cars that cross the intersection of King Fahd Road and Al-Ourouba Road is distributed according to a non homogenous Poisson process with a mean (t) defined as follows:

Let us consider the time 8 am as t=0. Q1. Compute the average arrival number of cars at 11H30? Q2. Determine the equation that gives the probability of having

only 10000 car arrivals between 12 pm and 16 pm. Q3. What is the distribution and the average (in seconds) of the

inter-arrival time of two cars between 8 am and 9 am?

80 cars/mn 8 960 cars/mn 9 1150 car/mn 11 1570 car/mn 15 17

if am t amif am t pm

tif am t pmif pm t pm

44

Example: Non-stationary Poisson Process (NSPP)

Q1. Compute the average arrival number of cars at 11H30?

Q2. Determine the equation that gives the probability of having only 10000 car arrivals between 12 pm and 16 pm.

We know that the number of cars between 12 pm and 16 pm, i.e. follows a Poisson distribution. During 12 pm and 16pm, the average number of cars is

Thus,

Q3. What is the distribution and the average (in seconds) of the inter-arrival time of two cars between 8 am and 9 am? (Homework)

11:30

8:00,11:30 8:00

9:00 11:00 11:30

8:00 9:00 11:00

80cars/mn 60mn 60cars/mn 120mn 50cars/mn 30mn 13500 cars

λ(u) du

λ(u) du λ(u) du λ(u) du

10000

1320016 12 10000 exp 13200

10000!p N N

16 12N N

12:00 16:00 180 50 60 70 13200 cars

45

Two Minutes Break


Administrative issues• Groups Formation

46

Continuous Probability Distributions

Uniform Distribution Exponential Distribution Normal (Gaussian) Distribution Weibull DistributionLognormal Distribution

47

Continuous Distributions

Continuous random variables can be used to describe random phenomena in which the variable can take on any value in some interval.

In this section, the distributions studied are: Uniform Exponential Normal Weibull Lognormal

48

Uniform Distribution

49

Continuous Uniform Distribution

The continuous uniform distribution is a family of probability distributions such that for each member of the family, all intervals of the same length on the distribution's support are equally probable

A random variable X is uniformly distributed on the interval [a,b], U(a,b), if its PDF and CDF are:

1,

PDF: ( )0, otherwise

a x bf x b a

0,

CDF: ( ) ,

1,

x a

x aF x a x b

b ax b

Expected value: 2

a bE X

2

Variance: 12

a bV X

50

Uniform Distribution

Properties is proportional to

the length of the interval

Special case: a standard uniform distribution U(0,1). Very useful for random

number generators in simulators

1 2p x X x

2 12 1

X XF X F X

b a

CDF

PDF

51

Exponential Distribution

Modeling Random Time

52


The exponential distribution describes the times between events in a Poisson process, in which events occur continuously and independently at a constant average rate.

A random variable X is exponentially distributed with parameter > 0 if its PDF and CDF are: exp , 0PDF: ( )

0, otherwise

x xf x

0

0, 0CDF: ( )

1 , 0x t x

xF x

e dt e x

1Expected value: E X

2

2

1Variance: V X

1exp , 0

( )

0, otherwise

xx

f x

0, 0

( )1 exp , 0

x

F x xx

53


1exp , 0

( ) 20 20

0, otherwise

xx

f x

0, 0

( )1 exp , 0

20

x

F x xx

µ=20 µ=20

54


The memoryless property: In probability theory, memoryless is a property of certain probability distributions: the exponential distributions and the geometric distributions, wherein any derived probability from a set of random samples is distinct and has no information (i.e. "memory") of earlier samples.

Formally, the memoryless property is:For all s and t greater or equal to 0:

This means that the future event do not depend on the past event, but only on the present event The fact that Pr(X > 40 | X > 30) = Pr(X > 10) does not mean that the

events X > 40 and X > 30 are independent; i.e. it does not mean that Pr(X > 40 | X > 30) = Pr(X > 40).

|p X s t X s p X t

55


The memoryless property: can be read as “the probability that you will wait more than s+t minutes given that you have already been waiting t minutes is equal to the probability that you will wait s minutes.”

In other words “The probability that you will wait s more minutes given that you have already been waiting t minutes is the same as the probability that you had wait for more than s minutes from the beginning.”

The fact that Pr(X > 40 | X > 30) = Pr(X > 10) does not mean that the events X > 40 and X > 30 are independent; i.e. it does not mean that Pr(X > 40 | X > 30) = Pr(X > 40).

|p X s t X s p X t

56

Example: Exponential Distribution

The time needed to repair the engine of a car is exponentially distributed with a mean time equal to 3 hours. The probability that the car spends more than 3 hours in

reparation

The probability that the car repair time lasts between 2 to 3 hours is:

The probability that the repair time lasts for another hour given it has been operating for 2.5 hours:

Using the memoryless property of the exponential distribution, we have:

33 1 3 1 3 1 1 exp 0.368

3p X p X F

3 3 2 0.145p X F F

12.5 1 | 2.5 1 1 1 exp 0.717

3p X X p X p X

57

Normal (Gaussian) Distribution

58

Normal Distribution

The Normal distribution, also called the Gaussian distribution, is an important family of continuous probability distributions, applicable in many fields.

Each member of the family may be defined by two parameters, location and scale: the mean ("average", μ) and variance (standard deviation squared, σ2) respectively.

The importance of the normal distribution as a model of quantitative phenomena in the natural and behavioral sciences is due in part to the Central Limit Theorem.

It is usually used to model system error (e.g. channel error), the distribution of natural phenomena, height, weight, etc.

59

Normal or Gaussian Distribution

A continuous random variable X, taking all real values in the range (-∞,+∞) is said to follow a Normal distribution with parameters µ and σ if it has the following PDF and CDF:

where

The Normal distribution is denoted as This probability density function (PDF) is

a symmetrical, bell-shaped curve, centered at its expected value µ. The variance is 2.

2

1 1PDF: exp

22

xf x

2~ ,X N

1CDF: 1

2 2

xF x erf

2

0

2Error Function: exp

x

erf x t

60

Normal distribution

Example The simplest case of the normal distribution, known as the

Standard Normal Distribution, has expected value zero and variance one. This is written as N(0,1).

61

Normal Distribution

Evaluating the distribution: Independent of and using the standard normal

distribution:

Transformation of variables: let

z t dtez 2/2

2

1)( where,

)()(

2

1

)(

/)(

/)( 2/2

xx

x z

dzz

dze

xZPxXPxF

~ 0,1Z N

XZ

62

Normal Distribution

Example: The time required to load an oceangoing vessel, X, is distributed as N(12,4) The probability that the vessel is loaded in less than 10

hours:

Using the symmetry property, (1) is the complement of (-1)

1587.0)1(2

1210)10(

F

63

Other Distributions

Weibull Distribution Lognormal Distribution

64

Weibull Distribution A random variable X has a Weibull distribution if its pdf has

the form:

3 parameters: Location parameter: Scale parameter: Shape parameter.

Example: = 0 and = 1:

Lifetime of objects

otherwise ,0

,exp)(

1

xxx

xf

)(

When = 1, X ~ exp( = 1/)

65

Lognormal Distribution A random variable X has a lognormal distribution

if its pdf has the form:

Mean E(X) = e+2/2 Variance V(X) = e+2/2 (e2 - 1)

Relationship with normal distribution When Y ~ N(, 2), then X = eY ~ lognormal(, 2) Parameters and 2 are not the mean and variance of

the lognormal general reliability analysis

otherwise 0,

0 ,2

lnexp

2

1)(

2

2x

σ

μx

σxπxf

65

=1, 2=0.5,1,2.

66

Empirical Distribution

67


An Empirical Distribution is a distribution whose parameters are the observed values in a sample of data. May be used when it is impossible or unnecessary

to establish that a random variable has any particular parametric distribution.

Advantage: no assumption beyond the observed values in the sample.

Disadvantage: sample might not cover the entire range of possible values.

68


In statistics, an empirical distribution function is a cumulative probability distribution function that concentrates probability 1/n at each of the n numbers in a sample.

Let X1, X2, …, Xn be iid random variables in with the CDF equal to F(x).

The empirical distribution function Fn(x) based on sample X1,

X2, …, Xn is a step function defined by

where I(A) is the indicator of event A. For a fixed value x, I(Xi≤x) is a Bernoulli random variable

with parameter p=F(x), hence nFn(x) is a binomial random variable with mean nF(x) and variance nF(x)(1-F(x)).

1

number of element in the sample 1 n

n ii

xF x I X x

n n

1 if

0 otherwisei

i

X xI X x

69

End of Chapter 4

CS433 Modeling and Simulation Lecture 04 Statistical Models 1 Dr. Anis Koubâa Al-Imam Mohammad Ibn...

Documents

Transcript of CS433 Modeling and Simulation Lecture 04 Statistical Models 1 Dr. Anis Koubâa Al-Imam Mohammad Ibn...