Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’...
-
date post
20-Dec-2015 -
Category
Documents
-
view
217 -
download
1
Transcript of Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’...
![Page 1: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/1.jpg)
Introduction to Probability Theory
Rong Jin
![Page 2: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/2.jpg)
Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions
![Page 3: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/3.jpg)
Definition of Probability Experiment: toss a coin twice Sample space: possible outcomes of an experiment
S = {HH, HT, TH, TT} Event: a subset of possible outcomes
A={HH}, B={HT, TH} Probability of an event : an number assigned to an
event Pr(A) Axiom 1: Pr(A) 0 Axiom 2: Pr(S) = 1 Axiom 3: For every sequence of disjoint events
Example: Pr(A) = n(A)/N: frequentist statistics
Pr( ) Pr( )i iiiA A
![Page 4: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/4.jpg)
Joint Probability For events A and B, joint probability
Pr(AB) stands for the probability that both events happen.
Example: A={HH}, B={HT, TH}, what is the joint probability Pr(AB)?
![Page 5: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/5.jpg)
Independence Two events A and B are independent in case
Pr(AB) = Pr(A)Pr(B)
A set of events {Ai} is independent in case
Pr( ) Pr( )i iiiA A
![Page 6: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/6.jpg)
Independence Two events A and B are independent in case
Pr(AB) = Pr(A)Pr(B)
A set of events {Ai} is independent in case
Example: Drug test
Pr( ) Pr( )i iiiA A
Women Men
Success 200 19
Failure 1800 1
A = {A patient is a Women}
B = {Drug fails}
Will event A be independent from event B ?
![Page 7: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/7.jpg)
Independence Consider the experiment of tossing a coin twice Example I:
A = {HT, HH}, B = {HT} Will event A independent from event B?
Example II: A = {HT}, B = {TH} Will event A independent from event B?
Disjoint Independence
If A is independent from B, B is independent from C, will A be independent from C?
![Page 8: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/8.jpg)
If A and B are events with Pr(A) > 0, the conditional probability of B given A is
Conditioning
Pr( )Pr( | )
Pr( )
ABB A
A
![Page 9: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/9.jpg)
If A and B are events with Pr(A) > 0, the conditional probability of B given A is
Example: Drug test
Conditioning
Pr( )Pr( | )
Pr( )
ABB A
A
Women Men
Success 200 19
Failure 1800 1
A = {A patient is a Women}
B = {Drug fails}
Pr(B|A) = ?
Pr(A|B) = ?
![Page 10: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/10.jpg)
If A and B are events with Pr(A) > 0, the conditional probability of B given A is
Example: Drug test
Given A is independent from B, what is the relationship between Pr(A|B) and Pr(A)?
Conditioning
Pr( )Pr( | )
Pr( )
ABB A
A
Women Men
Success 200 19
Failure 1800 1
A = {Patient is a Women}
B = {Drug fails}
Pr(B|A) = ?
Pr(A|B) = ?
![Page 11: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/11.jpg)
Which Drug is Better ?
![Page 12: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/12.jpg)
Simpson’s Paradox: View I
Drug I Drug II
Success 219 1100
Failure 1801 1190
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 10%
Pr(C|B) ~ 50%
Drug II is better than Drug I
![Page 13: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/13.jpg)
Simpson’s Paradox: View II
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%
![Page 14: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/14.jpg)
Simpson’s Paradox: View II
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%
Male Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 100%
Pr(C|B) ~ 50%
![Page 15: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/15.jpg)
Simpson’s Paradox: View II
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%
Male Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 100%
Pr(C|B) ~ 50%
Drug I is better than Drug II
![Page 16: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/16.jpg)
Conditional Independence Event A and B are conditionally independent given
C in case
Pr(AB|C)=Pr(A|C)Pr(B|C) A set of events {Ai} is conditionally independent
given C in case
Pr( | ) Pr( | )i iiiA C A C
![Page 17: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/17.jpg)
Conditional Independence (cont’d) Example: There are three events: A, B, C
Pr(A) = Pr(B) = Pr(C) = 1/5 Pr(A,C) = Pr(B,C) = 1/25, Pr(A,B) = 1/10 Pr(A,B,C) = 1/125 Whether A, B are independent? Whether A, B are conditionally independent
given C? A and B are independent A and B are
conditionally independent
![Page 18: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/18.jpg)
Outline Important concepts in probability theory Bayes’ rule Random variables and distributions
![Page 19: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/19.jpg)
Pr( ) Pr( | ) Pr( )Pr( | )
Pr( ) Pr( )i i i
iB A A B B
B AA A
Bayes’ Rule Suppose that B1, B2, … Bk form a partition of S:
Suppose that Pr(A) > 0. Then
; i j iiB B B S
![Page 20: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/20.jpg)
Pr( ) Pr( | ) Pr( )Pr( | )
Pr( ) Pr( )i i i
iB A A B B
B AA A
Bayes’ Rule Suppose that B1, B2, … Bk form a partition of S:
Suppose that Pr(A) > 0. Then
Example:
; i j iiB B B S
Pr(W|R) R R
W 0.7 0.4
W 0.3 0.6
R: The weather rains
W: The grass is wet
Pr(R|W) = ?
Pr(R) = 0.8
![Page 21: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/21.jpg)
Bayes’ Rule: More Complicated Suppose that B1, B2, … Bk form a partition of S:
Suppose that Pr(Bi) > 0 and Pr(A) > 0. Then
; i j iiB B B S
1
1
Pr( | ) Pr( )Pr( | )
Pr( )
Pr( | ) Pr( )
Pr( )
Pr( | ) Pr( )
Pr( ) Pr( | )
i ii
i ik
jj
i ik
j jj
A B BB A
A
A B B
AB
A B B
B A B
![Page 22: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/22.jpg)
Bayes’ Rule: More Complicated Suppose that B1, B2, … Bk form a partition of S:
Suppose that Pr(Bi) > 0 and Pr(A) > 0. Then
; i j iiB B B S
1
1
Pr( | ) Pr( )Pr( | )
Pr( )
Pr( | ) Pr( )
Pr( )
Pr( | ) Pr( )
Pr( ) Pr( | )
i ii
i ik
jj
i ik
j jj
A B BB A
A
A B B
AB
A B B
B A B
![Page 23: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/23.jpg)
Bayes’ Rule: More Complicated Suppose that B1, B2, … Bk form a partition of S:
Suppose that Pr(A) > 0. Then
; i j iiB B B S
1
1
Pr( | ) Pr( )Pr( | )
Pr( )
Pr( | ) Pr( )
Pr( )
Pr( | ) Pr( )
Pr( ) Pr( | )
i ii
i ik
jj
i ik
j jj
A B BB A
A
A B B
AB
A B B
B A B
![Page 24: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/24.jpg)
Bayes’ RuleR R
W 0.7 0.4
W 0.3 0.6
R: The weather rains
W: The grass is wet
R W
Information
Pr(W|R)
Inference
Pr(R|W)
![Page 25: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/25.jpg)
Pr( | ) Pr( )Pr( | )
Pr( )
E H HH E
E
Bayes’ RuleR R
W 0.7 0.4
W 0.3 0.6
R: The weather rains
W: The grass is wet
Hypothesis H Evidence EInformation: Pr(E|H)
Inference: Pr(H|E) PriorLikelihoodPosterior
![Page 26: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/26.jpg)
A More Complicated ExampleR The weather rains
W The grass is wet
U People bring umbrella
Pr(UW|R)=Pr(U|R)Pr(W|R)
Pr(UW| R)=Pr(U| R)Pr(W| R)
R
W U
Pr(W|R) R R
W 0.7 0.4
W 0.3 0.6
Pr(U|R) R R
U 0.9 0.2
U 0.1 0.8
Pr(U|W) = ?
Pr(R) = 0.8
![Page 27: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/27.jpg)
Outline Important concepts in probability theory Bayes’ rule Random variable and probability distribution
![Page 28: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/28.jpg)
Random Variable and Distribution A random variable X is a numerical outcome of a
random experiment The distribution of a random variable is the collection
of possible outcomes along with their probabilities: Discrete case: Continuous case:
Pr( ) ( )X x p x
Pr( ) ( )b
aa X b p x dx
![Page 29: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/29.jpg)
Random Variable: Example Let S be the set of all sequence of three rolls of a die.
Let X be the sum of the number of dots on the three rolls.
What are the possible values for X? Pr(X = 6) = ?, Pr(X = 10) = ?
![Page 30: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/30.jpg)
Expectation A random variable X~Pr(X=x). Then, its expectation is
In an empirical sample, x1, x2,…, xN,
Continuous case:
Expectation of sum of random variables
[ ] Pr( )x
E X x X x
1
1[ ]
Nii
E X xN
[ ] ( )E X xp x dx
1 2 1 2[ ] [ ] [ ]E X X E X E X
![Page 31: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/31.jpg)
Expectation: Example Let S be the set of all sequence of three rolls of a die.
Let X be the sum of the number of dots on the three rolls.
What is E(X)?
Let S be the set of all sequence of three rolls of a die. Let X be the product of the number of dots on the three rolls.
What is E(X)?
![Page 32: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/32.jpg)
Variance The variance of a random variable X is the
expectation of (X-E[x])2 :2
2 2
2 2
2 2
( ) (( [ ]) )
( [ ] 2 [ ])
( [ ] )
[ ] [ ]
Var X E X E X
E X E X XE X
E X E X
E X E X
![Page 33: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/33.jpg)
Bernoulli Distribution The outcome of an experiment can either be success
(i.e., 1) and failure (i.e., 0). Pr(X=1) = p, Pr(X=0) = 1-p, or
E[X] = p, Var(X) = p(1-p)
1( ) (1 )x xp x p p
![Page 34: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/34.jpg)
Binomial Distribution n draws of a Bernoulli distribution
Xi~Bernoulli(p), X=i=1n
Xi, X~Bin(p, n) Random variable X stands for the number of times
that experiments are successful.
E[X] = Np, Var(X) = Np(1-p)
(1 ) 1,2,...,Pr( ) ( )
0 otherwise
x n xnp p x n
X x p x x
![Page 35: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/35.jpg)
Plot of Binomial Distribution
![Page 36: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/36.jpg)
Poisson Distribution Coming from Binomial distribution
Fix the expectation =np Let the number of trials nA Binomial distribution will become a Poisson distribution
E[X] = , Var(X) =
0 or Pr( ) ( ) !
0 o.w.
xxe x x Z
X x p x x
![Page 37: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/37.jpg)
Plot of Poisson Distribution
![Page 38: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/38.jpg)
Normal (Gaussian) Distribution X~N(,)
E[X]= , Var(X)= 2
If X1~N(1,1) and X2~N(2,2), X= X1+ X2 ?
2
22
2
22
1 ( )( ) exp
22
1 ( )Pr( ) ( ) exp
22
b b
a a
xp x
xa X b p x dx dx
![Page 39: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/39.jpg)
Example In China, people usually favor boys. So, many people will not stop bearing
children unless they get a boy. Will the male population be larger than the
female’s?
![Page 40: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/40.jpg)
Statistical Inference Problem:
Likelihood function Approach: Maximum likelihood estimation (MLE), or
maximize log-likelihood
![Page 41: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/41.jpg)
Example I: Flip Coins
![Page 42: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/42.jpg)
Example I: Flip Coins (cont’d)
![Page 43: Introduction to Probability Theory Rong Jin. Outline Basic concepts in probability theory Bayes’ rule Random variable and distributions.](https://reader037.fdocuments.us/reader037/viewer/2022110322/56649d415503460f94a1c353/html5/thumbnails/43.jpg)
Example II: Normal Distribution