Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē...

30

Transcript of Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē...

Page 1: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated
PG2922
File Attachment
Thumbnailjpg

PROBABILITY AND RANDOMPROCESSES

PROBABILITY AND RANDOMPROCESSES

SECOND EDITION

VENKATARAMA KRISHNANProfessor Emeritus of Electrical and Computer EngineeringCenter for Advanced Computation and TelecommunicationsUniversity of Massachusetts Lowell

With Contribution from

KAVITHA CHANDRAProfessor of Electrical and Computer EngineeringCenter for Advanced Computation and TelecommunicationsUniversity of Massachusetts Lowell

Copyright copy 2016 by John Wiley amp Sons Inc All rights reserved

Published by John Wiley amp Sons Inc Hoboken New JerseyPublished simultaneously in Canada

No part of this publication may be reproduced stored in a retrieval system or transmitted in any form or byany means electronic mechanical photocopying recording scanning or otherwise except as permittedunder Section 107 or 108 of the 1976 United States Copyright Act without either the prior writtenpermission of the Publisher or authorization through payment of the appropriate per-copy fee to theCopyright Clearance Center Inc 222 Rosewood Drive Danvers MA 01923 (978) 750-8400 fax (978)750-4470 or on the web at wwwcopyrightcom Requests to the Publisher for permission should beaddressed to the Permissions Department John Wiley amp Sons Inc 111 River Street Hoboken NJ 07030(201) 748-6011 fax (201) 748-6008 or online at httpwwwwileycomgopermissions

Limit of LiabilityDisclaimer of Warranty While the publisher and author have used their bestefforts in preparing this book they make no representations or warranties with respect to the accuracyor completeness of the contents of this book and specifically disclaim any implied warranties ofmerchantability or fitness for a particular purpose No warranty may be created or extended by salesrepresentatives or written sales materials The advice and strategies contained herein may not be suitable foryour situation You should consult with a professional where appropriate Neither the publisher nor authorshall be liable for any loss of profit or any other commercial damages including but not limited to specialincidental consequential or other damages

For general information on our other products and services or for technical support please contact ourCustomer Care Department within the United States at (800) 762-2974 outside the United States at(317) 572-3993 or fax (317) 572-4002

Wiley also publishes its books in a variety of electronic formats Some content that appears in print maynot be available in electronic formats For more information about Wiley products visit our web site atwwwwileycom

Library of Congress Cataloging-in-Publication Data

Krishnan Venkatarama 1929ndashProbability and random processes Venkatarama Krishnan ndash Second edition

pages cmIncludes bibliographical references and indexISBN 978-1-118-92313-9 (cloth alk paper)

1 Probabilities 2 Stochastic processes 3 EngineeringndashStatistical methods 4 SciencendashStatisticalmethods I TitleQA273K74 20155192ndashdc23

2015023986

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

2 2016

Vishnu sahasranamam

Shree rāma rāma rāmētiramē rāmē manoramē

Sahasra nāmatathtulyamrāmanāma varānanē

This book is respectfully dedicated to thememory of my mother

and the memory of my father

Avvaiyar Tamil Poet

If a king and a well-learned person are balancedA well-learned person weighs more than the kingWhereas the king is revered only in his countryA well-learned person is valued wherever he goes

On a well-learned person

CONTENTS

Preface for the Second Edition xii

Preface for the First Edition xiv

1 Sets Fields and Events 1

11 Set Definitions 112 Set Operations 213 Set Algebras Fields and Events 5

2 Probability Space and Axioms 7

21 Probability Space 722 Conditional Probability 923 Independence 1124 Total Probability and Bayesrsquo Theorem 12

3 Basic Combinatorics 16

31 Basic Counting Principles 1632 Permutations 1633 Combinations 18

4 Discrete Distributions 23

41 Bernoulli Trials 2342 Binomial Distribution 2343 Multinomial Distribution 2644 Geometric Distribution 2645 Negative Binomial Distribution 2746 Hypergeometric Distribution 2847 Poisson Distribution 3048 NewtonndashPepys Problem and its Extensions 3349 Logarithmic Distribution 40

491 Finite Law (Benfordrsquos Law) 40492 Infinite Law 43

410 Summary of Discrete Distributions 44

5 Random Variables 45

51 Definition of Random Variables 4552 Determination of Distribution and Density Functions 4653 Properties of Distribution and Density Functions 5054 Distribution Functions from Density Functions 51

6 Continuous Random Variables and Basic Distributions 54

61 Introduction 5462 Uniform Distribution 5463 Exponential Distribution 5564 Normal or Gaussian Distribution 57

7 Other Continuous Distributions 63

71 Introduction 6372 Triangular Distribution 6373 Laplace Distribution 6374 Erlang Distribution 6475 Gamma Distribution 6576 Weibull Distribution 6677 Chi-Square Distribution 6778 Chi and Other Allied Distributions 6879 Student-t Density 71710 Snedecor F Distribution 72711 Lognormal Distribution 72712 Beta Distribution 73713 Cauchy Distribution 74714 Pareto Distribution 75715 Gibbs Distribution 75716 Mixed Distributions 75717 Summary of Distributions of Continuous Random Variables 76

8 Conditional Densities and Distributions 78

81 Conditional Distribution and Density for PA 0 7882 Conditional Distribution and Density for PA = 0 8083 Total Probability and Bayesrsquo Theorem for Densities 83

9 Joint Densities and Distributions 85

91 Joint Discrete Distribution Functions 8592 Joint Continuous Distribution Functions 8693 Bivariate Gaussian Distributions 90

10 Moments and Conditional Moments 91

101 Expectations 91102 Variance 92103 Means and Variances of Some Distributions 93104 Higher-Order Moments 94105 Correlation and Partial Correlation Coefficients 95

1051 Correlation Coefficients 951052 Partial Correlation Coefficients 106

viii CONTENTS

11 Characteristic Functions and Generating Functions 108

111 Characteristic Functions 108112 Examples of Characteristic Functions 109113 Generating Functions 111114 Examples of Generating Functions 112115 Moment Generating Functions 113116 Cumulant Generating Functions 115117 Table of Means and Variances 116

12 Functions of a Single Random Variable 118

121 Random Variable g(X) 118122 Distribution of Y = g(X) 119123 Direct Determination of Density fY(y) from fX(x) 129124 Inverse Problem Finding g(X) given fX(x) and fY(y) 132125 Moments of a Function of a Random Variable 133

13 Functions of Multiple Random Variables 135

131 Function of Two Random Variables Z = g(XY) 135132 Two Functions of Two Random Variables Z = g(XY) W = h(XY) 143133 Direct Determination of Joint Density fZW(zw) from fXY(xy) 146134 Solving Z = g(XY) Using an Auxiliary Random Variable 150135 Multiple Functions of Random Variables 153

14 Inequalities Convergences and Limit Theorems 155

141 Degenerate Random Variables 155142 Chebyshev and Allied Inequalities 155143 Markov Inequality 158144 Chernoff Bound 159145 CauchyndashSchwartz Inequality 160146 Jensenrsquos Inequality 162147 Convergence Concepts 163148 Limit Theorems 165

15 Computer Methods for Generating Random Variates 169

151 Uniform-Distribution Random Variates 169152 Histograms 170153 Inverse Transformation Techniques 172154 Convolution Techniques 178155 AcceptancendashRejection Techniques 178

16 Elements of Matrix Algebra 181

161 Basic Theory of Matrices 181162 Eigenvalues and Eigenvectors of Matrices 186163 Vector and Matrix Differentiation 190164 Block Matrices 194

17 Random Vectors and Mean-Square Estimation 196

171 Distributions and Densities 196

CONTENTS ix

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 2: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

PROBABILITY AND RANDOMPROCESSES

PROBABILITY AND RANDOMPROCESSES

SECOND EDITION

VENKATARAMA KRISHNANProfessor Emeritus of Electrical and Computer EngineeringCenter for Advanced Computation and TelecommunicationsUniversity of Massachusetts Lowell

With Contribution from

KAVITHA CHANDRAProfessor of Electrical and Computer EngineeringCenter for Advanced Computation and TelecommunicationsUniversity of Massachusetts Lowell

Copyright copy 2016 by John Wiley amp Sons Inc All rights reserved

Published by John Wiley amp Sons Inc Hoboken New JerseyPublished simultaneously in Canada

No part of this publication may be reproduced stored in a retrieval system or transmitted in any form or byany means electronic mechanical photocopying recording scanning or otherwise except as permittedunder Section 107 or 108 of the 1976 United States Copyright Act without either the prior writtenpermission of the Publisher or authorization through payment of the appropriate per-copy fee to theCopyright Clearance Center Inc 222 Rosewood Drive Danvers MA 01923 (978) 750-8400 fax (978)750-4470 or on the web at wwwcopyrightcom Requests to the Publisher for permission should beaddressed to the Permissions Department John Wiley amp Sons Inc 111 River Street Hoboken NJ 07030(201) 748-6011 fax (201) 748-6008 or online at httpwwwwileycomgopermissions

Limit of LiabilityDisclaimer of Warranty While the publisher and author have used their bestefforts in preparing this book they make no representations or warranties with respect to the accuracyor completeness of the contents of this book and specifically disclaim any implied warranties ofmerchantability or fitness for a particular purpose No warranty may be created or extended by salesrepresentatives or written sales materials The advice and strategies contained herein may not be suitable foryour situation You should consult with a professional where appropriate Neither the publisher nor authorshall be liable for any loss of profit or any other commercial damages including but not limited to specialincidental consequential or other damages

For general information on our other products and services or for technical support please contact ourCustomer Care Department within the United States at (800) 762-2974 outside the United States at(317) 572-3993 or fax (317) 572-4002

Wiley also publishes its books in a variety of electronic formats Some content that appears in print maynot be available in electronic formats For more information about Wiley products visit our web site atwwwwileycom

Library of Congress Cataloging-in-Publication Data

Krishnan Venkatarama 1929ndashProbability and random processes Venkatarama Krishnan ndash Second edition

pages cmIncludes bibliographical references and indexISBN 978-1-118-92313-9 (cloth alk paper)

1 Probabilities 2 Stochastic processes 3 EngineeringndashStatistical methods 4 SciencendashStatisticalmethods I TitleQA273K74 20155192ndashdc23

2015023986

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

2 2016

Vishnu sahasranamam

Shree rāma rāma rāmētiramē rāmē manoramē

Sahasra nāmatathtulyamrāmanāma varānanē

This book is respectfully dedicated to thememory of my mother

and the memory of my father

Avvaiyar Tamil Poet

If a king and a well-learned person are balancedA well-learned person weighs more than the kingWhereas the king is revered only in his countryA well-learned person is valued wherever he goes

On a well-learned person

CONTENTS

Preface for the Second Edition xii

Preface for the First Edition xiv

1 Sets Fields and Events 1

11 Set Definitions 112 Set Operations 213 Set Algebras Fields and Events 5

2 Probability Space and Axioms 7

21 Probability Space 722 Conditional Probability 923 Independence 1124 Total Probability and Bayesrsquo Theorem 12

3 Basic Combinatorics 16

31 Basic Counting Principles 1632 Permutations 1633 Combinations 18

4 Discrete Distributions 23

41 Bernoulli Trials 2342 Binomial Distribution 2343 Multinomial Distribution 2644 Geometric Distribution 2645 Negative Binomial Distribution 2746 Hypergeometric Distribution 2847 Poisson Distribution 3048 NewtonndashPepys Problem and its Extensions 3349 Logarithmic Distribution 40

491 Finite Law (Benfordrsquos Law) 40492 Infinite Law 43

410 Summary of Discrete Distributions 44

5 Random Variables 45

51 Definition of Random Variables 4552 Determination of Distribution and Density Functions 4653 Properties of Distribution and Density Functions 5054 Distribution Functions from Density Functions 51

6 Continuous Random Variables and Basic Distributions 54

61 Introduction 5462 Uniform Distribution 5463 Exponential Distribution 5564 Normal or Gaussian Distribution 57

7 Other Continuous Distributions 63

71 Introduction 6372 Triangular Distribution 6373 Laplace Distribution 6374 Erlang Distribution 6475 Gamma Distribution 6576 Weibull Distribution 6677 Chi-Square Distribution 6778 Chi and Other Allied Distributions 6879 Student-t Density 71710 Snedecor F Distribution 72711 Lognormal Distribution 72712 Beta Distribution 73713 Cauchy Distribution 74714 Pareto Distribution 75715 Gibbs Distribution 75716 Mixed Distributions 75717 Summary of Distributions of Continuous Random Variables 76

8 Conditional Densities and Distributions 78

81 Conditional Distribution and Density for PA 0 7882 Conditional Distribution and Density for PA = 0 8083 Total Probability and Bayesrsquo Theorem for Densities 83

9 Joint Densities and Distributions 85

91 Joint Discrete Distribution Functions 8592 Joint Continuous Distribution Functions 8693 Bivariate Gaussian Distributions 90

10 Moments and Conditional Moments 91

101 Expectations 91102 Variance 92103 Means and Variances of Some Distributions 93104 Higher-Order Moments 94105 Correlation and Partial Correlation Coefficients 95

1051 Correlation Coefficients 951052 Partial Correlation Coefficients 106

viii CONTENTS

11 Characteristic Functions and Generating Functions 108

111 Characteristic Functions 108112 Examples of Characteristic Functions 109113 Generating Functions 111114 Examples of Generating Functions 112115 Moment Generating Functions 113116 Cumulant Generating Functions 115117 Table of Means and Variances 116

12 Functions of a Single Random Variable 118

121 Random Variable g(X) 118122 Distribution of Y = g(X) 119123 Direct Determination of Density fY(y) from fX(x) 129124 Inverse Problem Finding g(X) given fX(x) and fY(y) 132125 Moments of a Function of a Random Variable 133

13 Functions of Multiple Random Variables 135

131 Function of Two Random Variables Z = g(XY) 135132 Two Functions of Two Random Variables Z = g(XY) W = h(XY) 143133 Direct Determination of Joint Density fZW(zw) from fXY(xy) 146134 Solving Z = g(XY) Using an Auxiliary Random Variable 150135 Multiple Functions of Random Variables 153

14 Inequalities Convergences and Limit Theorems 155

141 Degenerate Random Variables 155142 Chebyshev and Allied Inequalities 155143 Markov Inequality 158144 Chernoff Bound 159145 CauchyndashSchwartz Inequality 160146 Jensenrsquos Inequality 162147 Convergence Concepts 163148 Limit Theorems 165

15 Computer Methods for Generating Random Variates 169

151 Uniform-Distribution Random Variates 169152 Histograms 170153 Inverse Transformation Techniques 172154 Convolution Techniques 178155 AcceptancendashRejection Techniques 178

16 Elements of Matrix Algebra 181

161 Basic Theory of Matrices 181162 Eigenvalues and Eigenvectors of Matrices 186163 Vector and Matrix Differentiation 190164 Block Matrices 194

17 Random Vectors and Mean-Square Estimation 196

171 Distributions and Densities 196

CONTENTS ix

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 3: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

PROBABILITY AND RANDOMPROCESSES

SECOND EDITION

VENKATARAMA KRISHNANProfessor Emeritus of Electrical and Computer EngineeringCenter for Advanced Computation and TelecommunicationsUniversity of Massachusetts Lowell

With Contribution from

KAVITHA CHANDRAProfessor of Electrical and Computer EngineeringCenter for Advanced Computation and TelecommunicationsUniversity of Massachusetts Lowell

Copyright copy 2016 by John Wiley amp Sons Inc All rights reserved

Published by John Wiley amp Sons Inc Hoboken New JerseyPublished simultaneously in Canada

No part of this publication may be reproduced stored in a retrieval system or transmitted in any form or byany means electronic mechanical photocopying recording scanning or otherwise except as permittedunder Section 107 or 108 of the 1976 United States Copyright Act without either the prior writtenpermission of the Publisher or authorization through payment of the appropriate per-copy fee to theCopyright Clearance Center Inc 222 Rosewood Drive Danvers MA 01923 (978) 750-8400 fax (978)750-4470 or on the web at wwwcopyrightcom Requests to the Publisher for permission should beaddressed to the Permissions Department John Wiley amp Sons Inc 111 River Street Hoboken NJ 07030(201) 748-6011 fax (201) 748-6008 or online at httpwwwwileycomgopermissions

Limit of LiabilityDisclaimer of Warranty While the publisher and author have used their bestefforts in preparing this book they make no representations or warranties with respect to the accuracyor completeness of the contents of this book and specifically disclaim any implied warranties ofmerchantability or fitness for a particular purpose No warranty may be created or extended by salesrepresentatives or written sales materials The advice and strategies contained herein may not be suitable foryour situation You should consult with a professional where appropriate Neither the publisher nor authorshall be liable for any loss of profit or any other commercial damages including but not limited to specialincidental consequential or other damages

For general information on our other products and services or for technical support please contact ourCustomer Care Department within the United States at (800) 762-2974 outside the United States at(317) 572-3993 or fax (317) 572-4002

Wiley also publishes its books in a variety of electronic formats Some content that appears in print maynot be available in electronic formats For more information about Wiley products visit our web site atwwwwileycom

Library of Congress Cataloging-in-Publication Data

Krishnan Venkatarama 1929ndashProbability and random processes Venkatarama Krishnan ndash Second edition

pages cmIncludes bibliographical references and indexISBN 978-1-118-92313-9 (cloth alk paper)

1 Probabilities 2 Stochastic processes 3 EngineeringndashStatistical methods 4 SciencendashStatisticalmethods I TitleQA273K74 20155192ndashdc23

2015023986

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

2 2016

Vishnu sahasranamam

Shree rāma rāma rāmētiramē rāmē manoramē

Sahasra nāmatathtulyamrāmanāma varānanē

This book is respectfully dedicated to thememory of my mother

and the memory of my father

Avvaiyar Tamil Poet

If a king and a well-learned person are balancedA well-learned person weighs more than the kingWhereas the king is revered only in his countryA well-learned person is valued wherever he goes

On a well-learned person

CONTENTS

Preface for the Second Edition xii

Preface for the First Edition xiv

1 Sets Fields and Events 1

11 Set Definitions 112 Set Operations 213 Set Algebras Fields and Events 5

2 Probability Space and Axioms 7

21 Probability Space 722 Conditional Probability 923 Independence 1124 Total Probability and Bayesrsquo Theorem 12

3 Basic Combinatorics 16

31 Basic Counting Principles 1632 Permutations 1633 Combinations 18

4 Discrete Distributions 23

41 Bernoulli Trials 2342 Binomial Distribution 2343 Multinomial Distribution 2644 Geometric Distribution 2645 Negative Binomial Distribution 2746 Hypergeometric Distribution 2847 Poisson Distribution 3048 NewtonndashPepys Problem and its Extensions 3349 Logarithmic Distribution 40

491 Finite Law (Benfordrsquos Law) 40492 Infinite Law 43

410 Summary of Discrete Distributions 44

5 Random Variables 45

51 Definition of Random Variables 4552 Determination of Distribution and Density Functions 4653 Properties of Distribution and Density Functions 5054 Distribution Functions from Density Functions 51

6 Continuous Random Variables and Basic Distributions 54

61 Introduction 5462 Uniform Distribution 5463 Exponential Distribution 5564 Normal or Gaussian Distribution 57

7 Other Continuous Distributions 63

71 Introduction 6372 Triangular Distribution 6373 Laplace Distribution 6374 Erlang Distribution 6475 Gamma Distribution 6576 Weibull Distribution 6677 Chi-Square Distribution 6778 Chi and Other Allied Distributions 6879 Student-t Density 71710 Snedecor F Distribution 72711 Lognormal Distribution 72712 Beta Distribution 73713 Cauchy Distribution 74714 Pareto Distribution 75715 Gibbs Distribution 75716 Mixed Distributions 75717 Summary of Distributions of Continuous Random Variables 76

8 Conditional Densities and Distributions 78

81 Conditional Distribution and Density for PA 0 7882 Conditional Distribution and Density for PA = 0 8083 Total Probability and Bayesrsquo Theorem for Densities 83

9 Joint Densities and Distributions 85

91 Joint Discrete Distribution Functions 8592 Joint Continuous Distribution Functions 8693 Bivariate Gaussian Distributions 90

10 Moments and Conditional Moments 91

101 Expectations 91102 Variance 92103 Means and Variances of Some Distributions 93104 Higher-Order Moments 94105 Correlation and Partial Correlation Coefficients 95

1051 Correlation Coefficients 951052 Partial Correlation Coefficients 106

viii CONTENTS

11 Characteristic Functions and Generating Functions 108

111 Characteristic Functions 108112 Examples of Characteristic Functions 109113 Generating Functions 111114 Examples of Generating Functions 112115 Moment Generating Functions 113116 Cumulant Generating Functions 115117 Table of Means and Variances 116

12 Functions of a Single Random Variable 118

121 Random Variable g(X) 118122 Distribution of Y = g(X) 119123 Direct Determination of Density fY(y) from fX(x) 129124 Inverse Problem Finding g(X) given fX(x) and fY(y) 132125 Moments of a Function of a Random Variable 133

13 Functions of Multiple Random Variables 135

131 Function of Two Random Variables Z = g(XY) 135132 Two Functions of Two Random Variables Z = g(XY) W = h(XY) 143133 Direct Determination of Joint Density fZW(zw) from fXY(xy) 146134 Solving Z = g(XY) Using an Auxiliary Random Variable 150135 Multiple Functions of Random Variables 153

14 Inequalities Convergences and Limit Theorems 155

141 Degenerate Random Variables 155142 Chebyshev and Allied Inequalities 155143 Markov Inequality 158144 Chernoff Bound 159145 CauchyndashSchwartz Inequality 160146 Jensenrsquos Inequality 162147 Convergence Concepts 163148 Limit Theorems 165

15 Computer Methods for Generating Random Variates 169

151 Uniform-Distribution Random Variates 169152 Histograms 170153 Inverse Transformation Techniques 172154 Convolution Techniques 178155 AcceptancendashRejection Techniques 178

16 Elements of Matrix Algebra 181

161 Basic Theory of Matrices 181162 Eigenvalues and Eigenvectors of Matrices 186163 Vector and Matrix Differentiation 190164 Block Matrices 194

17 Random Vectors and Mean-Square Estimation 196

171 Distributions and Densities 196

CONTENTS ix

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 4: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

Copyright copy 2016 by John Wiley amp Sons Inc All rights reserved

Published by John Wiley amp Sons Inc Hoboken New JerseyPublished simultaneously in Canada

No part of this publication may be reproduced stored in a retrieval system or transmitted in any form or byany means electronic mechanical photocopying recording scanning or otherwise except as permittedunder Section 107 or 108 of the 1976 United States Copyright Act without either the prior writtenpermission of the Publisher or authorization through payment of the appropriate per-copy fee to theCopyright Clearance Center Inc 222 Rosewood Drive Danvers MA 01923 (978) 750-8400 fax (978)750-4470 or on the web at wwwcopyrightcom Requests to the Publisher for permission should beaddressed to the Permissions Department John Wiley amp Sons Inc 111 River Street Hoboken NJ 07030(201) 748-6011 fax (201) 748-6008 or online at httpwwwwileycomgopermissions

Limit of LiabilityDisclaimer of Warranty While the publisher and author have used their bestefforts in preparing this book they make no representations or warranties with respect to the accuracyor completeness of the contents of this book and specifically disclaim any implied warranties ofmerchantability or fitness for a particular purpose No warranty may be created or extended by salesrepresentatives or written sales materials The advice and strategies contained herein may not be suitable foryour situation You should consult with a professional where appropriate Neither the publisher nor authorshall be liable for any loss of profit or any other commercial damages including but not limited to specialincidental consequential or other damages

For general information on our other products and services or for technical support please contact ourCustomer Care Department within the United States at (800) 762-2974 outside the United States at(317) 572-3993 or fax (317) 572-4002

Wiley also publishes its books in a variety of electronic formats Some content that appears in print maynot be available in electronic formats For more information about Wiley products visit our web site atwwwwileycom

Library of Congress Cataloging-in-Publication Data

Krishnan Venkatarama 1929ndashProbability and random processes Venkatarama Krishnan ndash Second edition

pages cmIncludes bibliographical references and indexISBN 978-1-118-92313-9 (cloth alk paper)

1 Probabilities 2 Stochastic processes 3 EngineeringndashStatistical methods 4 SciencendashStatisticalmethods I TitleQA273K74 20155192ndashdc23

2015023986

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

2 2016

Vishnu sahasranamam

Shree rāma rāma rāmētiramē rāmē manoramē

Sahasra nāmatathtulyamrāmanāma varānanē

This book is respectfully dedicated to thememory of my mother

and the memory of my father

Avvaiyar Tamil Poet

If a king and a well-learned person are balancedA well-learned person weighs more than the kingWhereas the king is revered only in his countryA well-learned person is valued wherever he goes

On a well-learned person

CONTENTS

Preface for the Second Edition xii

Preface for the First Edition xiv

1 Sets Fields and Events 1

11 Set Definitions 112 Set Operations 213 Set Algebras Fields and Events 5

2 Probability Space and Axioms 7

21 Probability Space 722 Conditional Probability 923 Independence 1124 Total Probability and Bayesrsquo Theorem 12

3 Basic Combinatorics 16

31 Basic Counting Principles 1632 Permutations 1633 Combinations 18

4 Discrete Distributions 23

41 Bernoulli Trials 2342 Binomial Distribution 2343 Multinomial Distribution 2644 Geometric Distribution 2645 Negative Binomial Distribution 2746 Hypergeometric Distribution 2847 Poisson Distribution 3048 NewtonndashPepys Problem and its Extensions 3349 Logarithmic Distribution 40

491 Finite Law (Benfordrsquos Law) 40492 Infinite Law 43

410 Summary of Discrete Distributions 44

5 Random Variables 45

51 Definition of Random Variables 4552 Determination of Distribution and Density Functions 4653 Properties of Distribution and Density Functions 5054 Distribution Functions from Density Functions 51

6 Continuous Random Variables and Basic Distributions 54

61 Introduction 5462 Uniform Distribution 5463 Exponential Distribution 5564 Normal or Gaussian Distribution 57

7 Other Continuous Distributions 63

71 Introduction 6372 Triangular Distribution 6373 Laplace Distribution 6374 Erlang Distribution 6475 Gamma Distribution 6576 Weibull Distribution 6677 Chi-Square Distribution 6778 Chi and Other Allied Distributions 6879 Student-t Density 71710 Snedecor F Distribution 72711 Lognormal Distribution 72712 Beta Distribution 73713 Cauchy Distribution 74714 Pareto Distribution 75715 Gibbs Distribution 75716 Mixed Distributions 75717 Summary of Distributions of Continuous Random Variables 76

8 Conditional Densities and Distributions 78

81 Conditional Distribution and Density for PA 0 7882 Conditional Distribution and Density for PA = 0 8083 Total Probability and Bayesrsquo Theorem for Densities 83

9 Joint Densities and Distributions 85

91 Joint Discrete Distribution Functions 8592 Joint Continuous Distribution Functions 8693 Bivariate Gaussian Distributions 90

10 Moments and Conditional Moments 91

101 Expectations 91102 Variance 92103 Means and Variances of Some Distributions 93104 Higher-Order Moments 94105 Correlation and Partial Correlation Coefficients 95

1051 Correlation Coefficients 951052 Partial Correlation Coefficients 106

viii CONTENTS

11 Characteristic Functions and Generating Functions 108

111 Characteristic Functions 108112 Examples of Characteristic Functions 109113 Generating Functions 111114 Examples of Generating Functions 112115 Moment Generating Functions 113116 Cumulant Generating Functions 115117 Table of Means and Variances 116

12 Functions of a Single Random Variable 118

121 Random Variable g(X) 118122 Distribution of Y = g(X) 119123 Direct Determination of Density fY(y) from fX(x) 129124 Inverse Problem Finding g(X) given fX(x) and fY(y) 132125 Moments of a Function of a Random Variable 133

13 Functions of Multiple Random Variables 135

131 Function of Two Random Variables Z = g(XY) 135132 Two Functions of Two Random Variables Z = g(XY) W = h(XY) 143133 Direct Determination of Joint Density fZW(zw) from fXY(xy) 146134 Solving Z = g(XY) Using an Auxiliary Random Variable 150135 Multiple Functions of Random Variables 153

14 Inequalities Convergences and Limit Theorems 155

141 Degenerate Random Variables 155142 Chebyshev and Allied Inequalities 155143 Markov Inequality 158144 Chernoff Bound 159145 CauchyndashSchwartz Inequality 160146 Jensenrsquos Inequality 162147 Convergence Concepts 163148 Limit Theorems 165

15 Computer Methods for Generating Random Variates 169

151 Uniform-Distribution Random Variates 169152 Histograms 170153 Inverse Transformation Techniques 172154 Convolution Techniques 178155 AcceptancendashRejection Techniques 178

16 Elements of Matrix Algebra 181

161 Basic Theory of Matrices 181162 Eigenvalues and Eigenvectors of Matrices 186163 Vector and Matrix Differentiation 190164 Block Matrices 194

17 Random Vectors and Mean-Square Estimation 196

171 Distributions and Densities 196

CONTENTS ix

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 5: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

Vishnu sahasranamam

Shree rāma rāma rāmētiramē rāmē manoramē

Sahasra nāmatathtulyamrāmanāma varānanē

This book is respectfully dedicated to thememory of my mother

and the memory of my father

Avvaiyar Tamil Poet

If a king and a well-learned person are balancedA well-learned person weighs more than the kingWhereas the king is revered only in his countryA well-learned person is valued wherever he goes

On a well-learned person

CONTENTS

Preface for the Second Edition xii

Preface for the First Edition xiv

1 Sets Fields and Events 1

11 Set Definitions 112 Set Operations 213 Set Algebras Fields and Events 5

2 Probability Space and Axioms 7

21 Probability Space 722 Conditional Probability 923 Independence 1124 Total Probability and Bayesrsquo Theorem 12

3 Basic Combinatorics 16

31 Basic Counting Principles 1632 Permutations 1633 Combinations 18

4 Discrete Distributions 23

41 Bernoulli Trials 2342 Binomial Distribution 2343 Multinomial Distribution 2644 Geometric Distribution 2645 Negative Binomial Distribution 2746 Hypergeometric Distribution 2847 Poisson Distribution 3048 NewtonndashPepys Problem and its Extensions 3349 Logarithmic Distribution 40

491 Finite Law (Benfordrsquos Law) 40492 Infinite Law 43

410 Summary of Discrete Distributions 44

5 Random Variables 45

51 Definition of Random Variables 4552 Determination of Distribution and Density Functions 4653 Properties of Distribution and Density Functions 5054 Distribution Functions from Density Functions 51

6 Continuous Random Variables and Basic Distributions 54

61 Introduction 5462 Uniform Distribution 5463 Exponential Distribution 5564 Normal or Gaussian Distribution 57

7 Other Continuous Distributions 63

71 Introduction 6372 Triangular Distribution 6373 Laplace Distribution 6374 Erlang Distribution 6475 Gamma Distribution 6576 Weibull Distribution 6677 Chi-Square Distribution 6778 Chi and Other Allied Distributions 6879 Student-t Density 71710 Snedecor F Distribution 72711 Lognormal Distribution 72712 Beta Distribution 73713 Cauchy Distribution 74714 Pareto Distribution 75715 Gibbs Distribution 75716 Mixed Distributions 75717 Summary of Distributions of Continuous Random Variables 76

8 Conditional Densities and Distributions 78

81 Conditional Distribution and Density for PA 0 7882 Conditional Distribution and Density for PA = 0 8083 Total Probability and Bayesrsquo Theorem for Densities 83

9 Joint Densities and Distributions 85

91 Joint Discrete Distribution Functions 8592 Joint Continuous Distribution Functions 8693 Bivariate Gaussian Distributions 90

10 Moments and Conditional Moments 91

101 Expectations 91102 Variance 92103 Means and Variances of Some Distributions 93104 Higher-Order Moments 94105 Correlation and Partial Correlation Coefficients 95

1051 Correlation Coefficients 951052 Partial Correlation Coefficients 106

viii CONTENTS

11 Characteristic Functions and Generating Functions 108

111 Characteristic Functions 108112 Examples of Characteristic Functions 109113 Generating Functions 111114 Examples of Generating Functions 112115 Moment Generating Functions 113116 Cumulant Generating Functions 115117 Table of Means and Variances 116

12 Functions of a Single Random Variable 118

121 Random Variable g(X) 118122 Distribution of Y = g(X) 119123 Direct Determination of Density fY(y) from fX(x) 129124 Inverse Problem Finding g(X) given fX(x) and fY(y) 132125 Moments of a Function of a Random Variable 133

13 Functions of Multiple Random Variables 135

131 Function of Two Random Variables Z = g(XY) 135132 Two Functions of Two Random Variables Z = g(XY) W = h(XY) 143133 Direct Determination of Joint Density fZW(zw) from fXY(xy) 146134 Solving Z = g(XY) Using an Auxiliary Random Variable 150135 Multiple Functions of Random Variables 153

14 Inequalities Convergences and Limit Theorems 155

141 Degenerate Random Variables 155142 Chebyshev and Allied Inequalities 155143 Markov Inequality 158144 Chernoff Bound 159145 CauchyndashSchwartz Inequality 160146 Jensenrsquos Inequality 162147 Convergence Concepts 163148 Limit Theorems 165

15 Computer Methods for Generating Random Variates 169

151 Uniform-Distribution Random Variates 169152 Histograms 170153 Inverse Transformation Techniques 172154 Convolution Techniques 178155 AcceptancendashRejection Techniques 178

16 Elements of Matrix Algebra 181

161 Basic Theory of Matrices 181162 Eigenvalues and Eigenvectors of Matrices 186163 Vector and Matrix Differentiation 190164 Block Matrices 194

17 Random Vectors and Mean-Square Estimation 196

171 Distributions and Densities 196

CONTENTS ix

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 6: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

CONTENTS

Preface for the Second Edition xii

Preface for the First Edition xiv

1 Sets Fields and Events 1

11 Set Definitions 112 Set Operations 213 Set Algebras Fields and Events 5

2 Probability Space and Axioms 7

21 Probability Space 722 Conditional Probability 923 Independence 1124 Total Probability and Bayesrsquo Theorem 12

3 Basic Combinatorics 16

31 Basic Counting Principles 1632 Permutations 1633 Combinations 18

4 Discrete Distributions 23

41 Bernoulli Trials 2342 Binomial Distribution 2343 Multinomial Distribution 2644 Geometric Distribution 2645 Negative Binomial Distribution 2746 Hypergeometric Distribution 2847 Poisson Distribution 3048 NewtonndashPepys Problem and its Extensions 3349 Logarithmic Distribution 40

491 Finite Law (Benfordrsquos Law) 40492 Infinite Law 43

410 Summary of Discrete Distributions 44

5 Random Variables 45

51 Definition of Random Variables 4552 Determination of Distribution and Density Functions 4653 Properties of Distribution and Density Functions 5054 Distribution Functions from Density Functions 51

6 Continuous Random Variables and Basic Distributions 54

61 Introduction 5462 Uniform Distribution 5463 Exponential Distribution 5564 Normal or Gaussian Distribution 57

7 Other Continuous Distributions 63

71 Introduction 6372 Triangular Distribution 6373 Laplace Distribution 6374 Erlang Distribution 6475 Gamma Distribution 6576 Weibull Distribution 6677 Chi-Square Distribution 6778 Chi and Other Allied Distributions 6879 Student-t Density 71710 Snedecor F Distribution 72711 Lognormal Distribution 72712 Beta Distribution 73713 Cauchy Distribution 74714 Pareto Distribution 75715 Gibbs Distribution 75716 Mixed Distributions 75717 Summary of Distributions of Continuous Random Variables 76

8 Conditional Densities and Distributions 78

81 Conditional Distribution and Density for PA 0 7882 Conditional Distribution and Density for PA = 0 8083 Total Probability and Bayesrsquo Theorem for Densities 83

9 Joint Densities and Distributions 85

91 Joint Discrete Distribution Functions 8592 Joint Continuous Distribution Functions 8693 Bivariate Gaussian Distributions 90

10 Moments and Conditional Moments 91

101 Expectations 91102 Variance 92103 Means and Variances of Some Distributions 93104 Higher-Order Moments 94105 Correlation and Partial Correlation Coefficients 95

1051 Correlation Coefficients 951052 Partial Correlation Coefficients 106

viii CONTENTS

11 Characteristic Functions and Generating Functions 108

111 Characteristic Functions 108112 Examples of Characteristic Functions 109113 Generating Functions 111114 Examples of Generating Functions 112115 Moment Generating Functions 113116 Cumulant Generating Functions 115117 Table of Means and Variances 116

12 Functions of a Single Random Variable 118

121 Random Variable g(X) 118122 Distribution of Y = g(X) 119123 Direct Determination of Density fY(y) from fX(x) 129124 Inverse Problem Finding g(X) given fX(x) and fY(y) 132125 Moments of a Function of a Random Variable 133

13 Functions of Multiple Random Variables 135

131 Function of Two Random Variables Z = g(XY) 135132 Two Functions of Two Random Variables Z = g(XY) W = h(XY) 143133 Direct Determination of Joint Density fZW(zw) from fXY(xy) 146134 Solving Z = g(XY) Using an Auxiliary Random Variable 150135 Multiple Functions of Random Variables 153

14 Inequalities Convergences and Limit Theorems 155

141 Degenerate Random Variables 155142 Chebyshev and Allied Inequalities 155143 Markov Inequality 158144 Chernoff Bound 159145 CauchyndashSchwartz Inequality 160146 Jensenrsquos Inequality 162147 Convergence Concepts 163148 Limit Theorems 165

15 Computer Methods for Generating Random Variates 169

151 Uniform-Distribution Random Variates 169152 Histograms 170153 Inverse Transformation Techniques 172154 Convolution Techniques 178155 AcceptancendashRejection Techniques 178

16 Elements of Matrix Algebra 181

161 Basic Theory of Matrices 181162 Eigenvalues and Eigenvectors of Matrices 186163 Vector and Matrix Differentiation 190164 Block Matrices 194

17 Random Vectors and Mean-Square Estimation 196

171 Distributions and Densities 196

CONTENTS ix

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 7: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

5 Random Variables 45

51 Definition of Random Variables 4552 Determination of Distribution and Density Functions 4653 Properties of Distribution and Density Functions 5054 Distribution Functions from Density Functions 51

6 Continuous Random Variables and Basic Distributions 54

61 Introduction 5462 Uniform Distribution 5463 Exponential Distribution 5564 Normal or Gaussian Distribution 57

7 Other Continuous Distributions 63

71 Introduction 6372 Triangular Distribution 6373 Laplace Distribution 6374 Erlang Distribution 6475 Gamma Distribution 6576 Weibull Distribution 6677 Chi-Square Distribution 6778 Chi and Other Allied Distributions 6879 Student-t Density 71710 Snedecor F Distribution 72711 Lognormal Distribution 72712 Beta Distribution 73713 Cauchy Distribution 74714 Pareto Distribution 75715 Gibbs Distribution 75716 Mixed Distributions 75717 Summary of Distributions of Continuous Random Variables 76

8 Conditional Densities and Distributions 78

81 Conditional Distribution and Density for PA 0 7882 Conditional Distribution and Density for PA = 0 8083 Total Probability and Bayesrsquo Theorem for Densities 83

9 Joint Densities and Distributions 85

91 Joint Discrete Distribution Functions 8592 Joint Continuous Distribution Functions 8693 Bivariate Gaussian Distributions 90

10 Moments and Conditional Moments 91

101 Expectations 91102 Variance 92103 Means and Variances of Some Distributions 93104 Higher-Order Moments 94105 Correlation and Partial Correlation Coefficients 95

1051 Correlation Coefficients 951052 Partial Correlation Coefficients 106

viii CONTENTS

11 Characteristic Functions and Generating Functions 108

111 Characteristic Functions 108112 Examples of Characteristic Functions 109113 Generating Functions 111114 Examples of Generating Functions 112115 Moment Generating Functions 113116 Cumulant Generating Functions 115117 Table of Means and Variances 116

12 Functions of a Single Random Variable 118

121 Random Variable g(X) 118122 Distribution of Y = g(X) 119123 Direct Determination of Density fY(y) from fX(x) 129124 Inverse Problem Finding g(X) given fX(x) and fY(y) 132125 Moments of a Function of a Random Variable 133

13 Functions of Multiple Random Variables 135

131 Function of Two Random Variables Z = g(XY) 135132 Two Functions of Two Random Variables Z = g(XY) W = h(XY) 143133 Direct Determination of Joint Density fZW(zw) from fXY(xy) 146134 Solving Z = g(XY) Using an Auxiliary Random Variable 150135 Multiple Functions of Random Variables 153

14 Inequalities Convergences and Limit Theorems 155

141 Degenerate Random Variables 155142 Chebyshev and Allied Inequalities 155143 Markov Inequality 158144 Chernoff Bound 159145 CauchyndashSchwartz Inequality 160146 Jensenrsquos Inequality 162147 Convergence Concepts 163148 Limit Theorems 165

15 Computer Methods for Generating Random Variates 169

151 Uniform-Distribution Random Variates 169152 Histograms 170153 Inverse Transformation Techniques 172154 Convolution Techniques 178155 AcceptancendashRejection Techniques 178

16 Elements of Matrix Algebra 181

161 Basic Theory of Matrices 181162 Eigenvalues and Eigenvectors of Matrices 186163 Vector and Matrix Differentiation 190164 Block Matrices 194

17 Random Vectors and Mean-Square Estimation 196

171 Distributions and Densities 196

CONTENTS ix

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 8: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

11 Characteristic Functions and Generating Functions 108

111 Characteristic Functions 108112 Examples of Characteristic Functions 109113 Generating Functions 111114 Examples of Generating Functions 112115 Moment Generating Functions 113116 Cumulant Generating Functions 115117 Table of Means and Variances 116

12 Functions of a Single Random Variable 118

121 Random Variable g(X) 118122 Distribution of Y = g(X) 119123 Direct Determination of Density fY(y) from fX(x) 129124 Inverse Problem Finding g(X) given fX(x) and fY(y) 132125 Moments of a Function of a Random Variable 133

13 Functions of Multiple Random Variables 135

131 Function of Two Random Variables Z = g(XY) 135132 Two Functions of Two Random Variables Z = g(XY) W = h(XY) 143133 Direct Determination of Joint Density fZW(zw) from fXY(xy) 146134 Solving Z = g(XY) Using an Auxiliary Random Variable 150135 Multiple Functions of Random Variables 153

14 Inequalities Convergences and Limit Theorems 155

141 Degenerate Random Variables 155142 Chebyshev and Allied Inequalities 155143 Markov Inequality 158144 Chernoff Bound 159145 CauchyndashSchwartz Inequality 160146 Jensenrsquos Inequality 162147 Convergence Concepts 163148 Limit Theorems 165

15 Computer Methods for Generating Random Variates 169

151 Uniform-Distribution Random Variates 169152 Histograms 170153 Inverse Transformation Techniques 172154 Convolution Techniques 178155 AcceptancendashRejection Techniques 178

16 Elements of Matrix Algebra 181

161 Basic Theory of Matrices 181162 Eigenvalues and Eigenvectors of Matrices 186163 Vector and Matrix Differentiation 190164 Block Matrices 194

17 Random Vectors and Mean-Square Estimation 196

171 Distributions and Densities 196

CONTENTS ix

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 9: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

172 Moments of Random Vectors 200173 Vector Gaussian Random Variables 204174 Diagonalization of Covariance Matrices 207175 Simultaneous Diagonalization of Covariance Matrices 209176 Linear Estimation of Vector Variables 210

18 Estimation Theory 212

181 Criteria of Estimators 212182 Estimation of Random Variables 213183 Estimation of Parameters (Point Estimation) 218184 Interval Estimation (Confidence Intervals) 225185 Hypothesis Testing (Binary) 231186 Bayesian Estimation 238

19 Random Processes 250

191 Basic Definitions 250192 Stationary Random Processes 258193 Ergodic Processes 269194 Estimation of Parameters of Random Processes 273

1941 Continuous-Time Processes 2731942 Discrete-Time Processes 280

195 Power Spectral Density 2871951 Continuous Time 2871952 Discrete Time 294

196 Adaptive Estimation 298

20 Classification of Random Processes 320

201 Specifications of Random Processes 3202011 Discrete-State Discrete-Time (DSDT) Process 3202012 Discrete-State Continuous-Time (DSCT) Process 3202013 Continuous-State Discrete-Time (CSDT) Process 3202014 Continuous-State Continuous-Time (CSCT) Process 320

202 Poisson Process 321203 Binomial Process 329204 Independent Increment Process 330205 Random-Walk Process 333206 Gaussian Process 338207 Wiener Process (Brownian Motion) 340208 Markov Process 342209 Markov Chains 3472010 Birth and Death Processes 3572011 Renewal Processes and Generalizations 3662012 Martingale Process 3702013 Periodic Random Process 3742014 Aperiodic Random Process (KarhunenndashLoeve Expansion) 377

21 Random Processes and Linear Systems 383

211 Review of Linear Systems 383212 Random Processes through Linear Systems 385213 Linear Filters 393214 Bandpass Stationary Random Processes 401

x CONTENTS

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 10: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

22 Wiener and Kalman Filters 413

221 Review of Orthogonality Principle 413222 Wiener Filtering 414223 Discrete Kalman Filter 425224 Continuous Kalman Filter 433

23 Probability Modeling in Traffic Engineering 437

231 Introduction 437232 Teletraffic Models 437233 Blocking Systems 438234 State Probabilities for Systems with Delays 440235 Waiting-Time Distribution for MMcinfin Systems 441236 State Probabilities for MDc Systems 443237 Waiting-Time Distribution for MDcinfin System 446238 Comparison of MMc and MDc 448

References 451

24 Probabilistic Methods in Transmission Tomography 452

241 Introduction 452242 Stochastic Model 453243 Stochastic Estimation Algorithm 455244 Prior Distribution PM 457245 Computer Simulation 458246 Results and Conclusions 460247 Discussion of Results 462

References 462

APPENDICES

A A Fourier Transform Tables 463

B Cumulative Gaussian Tables 467

C Inverse Cumulative Gaussian Tables 472

D Inverse Chi-Square Tables 474

E Inverse Student-t Tables 481

F Cumulative Poisson Distribution 484

G Cumulative Binomial Distribution 488

H Computation of Roots of D(z) = 0 494

References 495

Index 498

CONTENTS xi

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 11: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

PREFACE FOR THE SECOND EDITION

In this second edition additional sections have been incor-porated to make it a more complete tome In Chapter 4 ahistorical account of how Isaac Newton solved a binomialprobability problem posed (1693) by the diarist SamuelPepys when the field of probability was relatively unknownis described In Chapter 10 Pearson correlation coefficientintroduced earlier in the chapter has been expanded toinclude Spearman and Kendall correlation coefficients withtheir properties and usages discussed In Chapter 14 con-vergences have been generalized In Chapter 19 on estima-tion two adaptive estimation techniques such as RecursiveLeast Squares and Least Mean Squares have been added InChapter 20 on random processes additional topics such asBirth and Death Processes and Renewal Processes usefulfor analyzing queuing have been incorporated Chapter 23is a new chapter on Probability Modeling of TeletrafficEngineering written by my colleague Kavitha Chandradiscussing various probability models in teletrafficengineering

This new edition now contains 455 carefully detailed fig-ures and 377 representative examples some with multiplesolutions with every step explained clearly enhancing the clar-ity of presentation to yield a more comprehensive referencetome It explains in detail the essential topics of applied math-ematical functions to problems that engineers and researcherssolve daily in the course of their work The usual topics likeset theory combinatorics random variables discrete and con-tinuous probability distribution functions moments and con-vergence of random variables autocovariance and crosscovariance functions stationarity concepts Wiener and Kal-man filtering and tomographic imaging of the first edition

have all been retained Graphical Fourier transform tablesand a number of probability tables with accuracy up to ninedecimal places are given in the seven appendices A neweighth appendix on finding the roots of the probability gener-ating function has also been added The index has been care-fully prepared to enhance the utility of this reference

With these added material on theory and applications ofprobability the second edition of Probability and RandomProcesses is a more comprehensive reference tome for practi-cing scientists and engineers in economics science andtechnology

As in the first edition the additional graphs in this editionwere also created with Mathcad software All algebraiccalculations were also verified with Mathcad softwareMathcad is a registered trademark of Parametric TechnologyCorporation Inc httpwwwptccomproductmathcad

Acknowledgments

Since the publication of the first edition of this book manyother books on Probability have appeared and they haveinfluenced me to a considerable extent in the choice ofadditional topics I once again extend my scientific debt ofgratitude to all these authors

I acknowledge with great pleasure the support given to meby my colleague Kavitha Chandra who not only contributeda chapter but also helped me to a considerable extent in thepreparation of the manuscript My grateful thanks are dueto Kari Capone who encouraged me to bring out a secondedition and did the entire spade work to make this project

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 12: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

a reality The courteous and thoughtful encouragement byAlex Castro during the final phases of the preparation of thismanuscript is a pleasure for me to acknowledge The effi-ciency with which Ramya Srinivasan the production editorwent about the production job is indeed a pleasure to recordShe also has the knack of smoothing ruffled feathers whichhad kept me in an even keel Finally I also acknowledge withthanks the clean and efficient job performed by F Pascal Rajthe assistant account manager and his team with their pro-active suggestions

The unequivocal support given to me by my wife Kamalahas been a great source of comfort

This is my third book with Wiley and I am always amazedat the way they can create a book with clockwork precisionout of the rough manuscript

VENKATARAMA KRISHNAN

Chelmsford MAMarch 2015

PREFACE FOR THE SECOND EDITION xiii

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 13: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

PREFACE FOR THE FIRST EDITION

Many good textbooks exist on probability and random pro-cesses written at the undergraduate level to the research levelHowever there is no one handy and ready book that explainsmost of the essential topics such as random variablesand most of their frequently used discrete and continuousprobability distribution functions moments transformationand convergences of random variables characteristic andgenerating functions estimation theory and the associatedorthogonality principle vector random variables randomprocesses and their autocovariance and cross-covariancefunctions stationarity concepts and random processesthrough linear systems and the associated Wiener and Kalmanfilters Engineering practitioners and students alike have todelve through several books to get the required formulas ortables either to complete a project or to finish a homeworkassignment This book may alleviate this diffculty to someextent and provide access to a compendium of most distribu-tion functions used by communication engineers queuingtheory specialists signal processing engineers biomedicalengineers and physicists Probability tables with accuracyup to nine decimal places are given in the appendixes toenhance the utility of this book A particular feature is the pres-entation of commonly occurring Fourier transforms whereboth the time and frequency functions are drawn to scale

Most of the theory has been explained with figures drawnto scale To understand the theory better more than 300examples are given with every step explained clearly Fol-lowing the adage that a figure is worth more than a thousandwords most of the examples are also illustrated with figuresdrawn to scale resulting in more than 400 diagrams Thisbook will be of particular value to graduate and undergradu-ate students in electrical computer and civil engineering aswell as students in physics and applied mathematics for sol-ving homework assignments and projects It will certainly be

useful to communication and signal processing engineersand computer scientists in an industrial setting It will alsoserve as a good reference for research workers in biostatisticsand financial market analysis

The salient features of this book are

bull Functional and statistical independence of random vari-ables are explained

bull Ready reference to commonly occurring density anddistribution functions and their means and variances isprovided

bull A section on Benfords logarithmic law which is used indetecting tax fraud is included

bull More than 300 examples many of them solved in differ-ent ways to illustrate the theory and various applicationsof probability are presented

bull Most examples have been substantiated with graphsdrawn to scale

bull More than400 figureshavebeendrawntoscale to improvethe clarity of understanding the theory and examples

bull Bounds on the tails of Gaussian probability have beencarefully developed

bull A chapter has been devoted to computer generation ofrandom variates

bull Another chapter has been devoted to matrix algebra sothat some estimation problems such as the Kalman filtercan be cast in matrix framework

bull Estimation problems have been given considerableexposure

bull Random processes defined and classified

bull A section on martingale processes with examples hasbeen included

xiv

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 14: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

bull Markov chains with interesting examples have beendiscussed

bull Wiener and Kalman filtering have been discussed with anumber of examples

bull Important properties are summarized in tables

bull The final chapter is on applications of probability totomographic imaging

bull The appendixes consist of probability tables withnine-place decimal accuracy for the most common prob-ability distributions

bull An extremely useful feature are the Fourier transformtables with both time and frequency functions graphedcarefully to scale

After the introduction of functional independence inSection 12 the differences between functional and statisticalindependences are discussed in Section 23 and clarifiedin Example 232 The foundation for permutations andcombinations are laid out in Chapter 3 followed by discretedistributions in Chapter 4 with an end-of-chapter summary ofdiscrete distributions Random variables are defined inChapter 5 and most of the frequently occurring continuousdistributions are explained in Chapters 6 and 7 Section 64discusses the new bounds on Gaussian tails A comprehen-sive table is given at the end of Chapter 7 for all the con-tinuous distributions and densities After discussion ofconditional densities joint densities and moments in Chap-ters 8ndash10 characteristic and other allied functions areexplained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and con-tinuous distributions Functions of random variables are dis-cussed in Chapters 12 and 13 with numerous examplesChapter 14 discusses the various bounds on probabilitiesalong with some commonly occurring inequalities Computergeneration of random variates is presented in Chapter 15Elements of matrix algebra along with vector and matrix dif-ferentiations are considered in Chapter 16 Vector randomvariables and diagonalization of covariance matrices andsimultaneous diagonalization of two covariance matricesare taken up in Chapter 17 Estimation and allied hypothesistesting are discussed in Chapter 18 After random processesare explained in Chapter 19 they are carefully classifiedin Chapter 20 with Section 2010 presenting martingale pro-cesses that find wide use in financial engineering Chapter 21discusses the effects of passing random processes throughlinear systems A number of examples illustrate the basicideas of Wiener and Kalman filters in Chapter 22 The finalchapter Chapter 23 presents a practical application of

probability to tomographic imaging The appendixes includeprobability tables up to nine decimal places for Gaussian chi-square Student-t Poisson and binomial distributions andFourier transform tables with graphs for time and frequencyfunctions carefully drawn to scale

This book is suitable for students and practicing engineerswho need a quick reference to any probability distribution ortable It is also useful for self-study where a number ofcarefully solved examples illustrate the applications of prob-ability Almost every topic has solved examples It can beused as an adjunct to any textbook on probability and mayalso be prescribed as a textbook with homework problemsdrawn from other sources

During the more than four decades that I have been con-tinuously teaching probability and random processes manyauthors who have written excellent textbooks have influ-enced me to a considerable extent Many of the ideas andconcepts that are expanded in this book are the direct resultof this influence I owe a scientific debt of gratitude to all theauthors who have contributed to this ubiquitous and excitingfield of probability Some of these authors are listed in thereference and the list is by no means exhaustive

Most of the graphs and all the probability tables in thisbook were created with Mathcad software All algebraiccalculations were verified with Mathcad software Mathcadand Mathsoft are registered trademarks of MathsoftEngineering and Education Inc httpwwwmathcad com

While facing extreme personal diffculties in writing thisbook the unequivocal support of my entire family have beena source of inspiration in finishing it

I acknowledge with great pleasure the support given to meby the Wiley staff George Telecki welcomed the idea of thisbook Rachel Witmer did more than her share of keeping mein good humor with her cheerful disposition in spite of theever-expanding deadlines and Kellsee Chu who did anexcellent job in managing the production of this bookFinally the enthusiastic support given to me by the SeriesEditor Emmanuel Desurvire was very refreshing This ismy second book with Wiley and the skills of their copyedi-tors and staff who transform highly mathematical manuscriptinto a finished book continue to amaze me

I have revised this book several times and corrected errorsNevertheless I cannot claim that it is error-free since correct-ing errors in any book is a convergent process I sincerelyhope that readers will bring to my attention any errors of com-mission or omission that they may come across

VENKATARAMA KRISHNANChelmsford MassachusettsMarch 2006

PREFACE FOR THE FIRST EDITION xv

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 15: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

1SETS FIELDS AND EVENTS

11 SET DEFINITIONS

The concept of sets play an important role in probability Wewill define a set in the following paragraph

Definition of Set

A set is a collection of objects called elements The elementsof a set can also be sets Sets are usually represented byuppercase letters A and elements are usually representedby lowercase letters a Thus

A= a1a2hellipan 1 1 1

will mean that the set A contains the elements a1 a2 hellip anConversely we can write that ak is an element of A as

ak A 1 1 2

and ak is not an element of A as

ak A 1 1 3

A finite set contains a finite number of elements for exampleS = 246 Infinite sets will have either countably infiniteelements such as A = x x is all positive integers oruncountably infinite elements such as B = x x is real num-ber le 20

Example 111 The set A of all positive integers less than 7is written as

A= x x is a positive integer lt 7 finite set

Example 112 The set N of all positive integers iswritten as

N = x x is all positive integers countably infinite set

Example 113 The set R of all real numbers is written as

R= x x is real uncountably infinite set

Example 114 The set R2 of real numbers x y is written as

R2 = xy x is real y is real

Example 115 The set C of all real numbers xy such thatx + y le 10 is written as

C = xy x + y le 10 uncountably infinite set

Venn Diagram

Sets can be represented graphically by means of a Venn dia-gram In this case we assume tacitly that S is a universalset under consideration In Example 115 the universal setS = x x is all positive integers We shall represent the setA in Example 111 by means of a Venn diagram of Fig 111

Empty Set

An empty is a set that contains no element It plays an impor-tant role in set theory and is denoted by The set A = 0 isnot an empty set since it contains the element 0

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 16: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

Cardinality

The number of elements in the set A is called the cardinalityof set A and is denoted by A If it is an infinite set then thecardinality is infin

Example 116 The cardinality of the setA = 246 is 3 orA = 3 The cardinality of set R = x x is real is infin

Example 117 The cardinality of the set A = x x is pos-itive integer lt7 is A = 6

Example 118 The cardinality of the set B = x x is a realnumber lt10 is infinity since there are infinite real num-bers lt10

Subset

A set B is a subset of A if every element in B is an element ofA and is written as B A B is a proper subset of A if everyelement of A is not in B and is written as B A

Equality of Sets

Two sets A and B are equal if B A and A B that is if everyelement of A is contained in B and every element of B iscontained in A In other words sets A and B contain exactlythe same elements Note that this is different from havingthe same cardinality that is containing the same numberof elements

Example 119 The set B = 135 is a proper subset ofA = 123456 whereas the set C = x x is a positive eveninteger le 6 and the set D = 246 are the same since theycontain the same elements The cardinalities of B C and Dare 3 and C = D

We shall now represent the sets A and B and the sets Cand D in Example 119 by means of the Venn diagram ofFig 112 on a suitably defined universal set S

Power Set

The power set of any set A is the set of all possible subsetsof A and is denoted by PS(A) Every power set of any set

A must contain the set A itself and the empty set If n isthe cardinality of the set A then the cardinality of the powerset PS(A) = 2n

Example 1110 If the set A = 123 then PS(A) = (123) (12) (23) (31) (1) (2) (3) The cardinality PS(A) = 8 = 23

12 SET OPERATIONS

Union

Let A and B be sets belonging to the universal set S Theunion of sets A and B is another set C whose elements arethose that are in either A or B and is denoted by A B Wherethere is no confusion it will also be represented as A + B

A B=A+B= x x Aor x B 1 2 1

Example 121 The union of sets A = 123 and B =2345 is the set C = A B = 12345

Intersection

The intersection of the sets A and B is another set C whoseelements are the same as those in both A and B and isdenoted by A B Where there is no confusion it will alsobe represented by AB

A B =AB = x x A and x B 1 2 2

FIGURE 111

FIGURE 112

2 SETS FIELDS AND EVENTS

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 17: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

Example 122 The intersection of the sets A and B inExample 121 is the set C = 23 Examples 121 and122 are shown in the Venn diagram of Fig 121

Mutually Exclusive Sets

Two sets A and B are called mutually exclusive if their inter-section is empty Mutually exclusive sets are also calleddisjoint

A B= 1 2 3

One way to determine whether two sets A and B are mutuallyexclusive is to check whether set B can occur when set A hasalready occurred and vice versa If it cannot then A andB are mutually exclusive For example if a single coin istossed the two sets heads and tails are mutually exclu-sive since tails cannot occur when heads has alreadyoccurred and vice versa

Independence

We will consider two types of independence The first isknown as functional independence [58]1 Two sets A and Bcan be called functionally independent if the occurrence of Bdoesnot in anyway influence the occurrence ofA andvice versaThe second one is statistical independence which is a differentconcept that will be defined later As an example the tossingof a coin is functionally independent of the tossing of a diebecause they do not depend on each other However the tossingof a coin and a die are not mutually exclusive since any onecan be tossed irrespective of the other By the same tokenpressure and temperature are not functionally independentbecause the physics of the problem namely Boylersquos law con-nects these quantities They are certainly notmutually exclusive

Cardinality of Unions and Intersections

We can now ascertain the cardinality of the union of two setsA and B that are not mutually exclusive The cardinality of the

union C = A B can be determined as follows If we add thecardinality A to the cardinality B we have added the car-dinality of the intersection A B twice Hence we have tosubtract once the cardinality A B as shown in Fig 122Or in other words

A B = A + B minus A B 1 2 4a

In Fig 122 the cardinality A = 9 and the cardinality B =11 and the cardinality A B is 11 + 9 minus 4 = 16

As a corollary if sets A and B are mutually exclusive thenthe cardinality of the union is the sum of the cardinalities or

A B = A + B 1 2 4b

The generalization of this result to an arbitrary union of n setsis called the inclusionndashexclusion principle given by

n

i= 1

Ai =n

i= 1

Ai minusn

i j= 1i j

Ai Aj +n

i j k = 1i j k

Ai Aj Ak

minus plusmn 1 nn

i j k = 1i j khellip n

Ai Aj Ak An 1 2 5a

If the sets Ai are mutually exclusive that is Ai Aj = fori j then we have

n

i= 1

Ai =n

i= 1

Ai 1 2 5b1Numbers in brackets refer to bibliographic entries in the Referencessection at the end of the book

FIGURE 121

FIGURE 122

SET OPERATIONS 3

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 18: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

This equation is illustrated in the Venn diagram for n = 3 inFig 123 where if A B C equals A + B + C then wehave added twice the cardinalites of A B B C andC A However if we subtract once A B B C and C A and write

A B C = A + B + C minus A B minus B C minus C A

then we have subtracted A B C thrice instead of twiceHence adding A B C we get the final result

A B C = A + B + C minus A B minus B C

minus C A + A B C 1 2 6

Example 123 In a junior class the number of students inelectrical engineering (EE) is 100 in math (MA) 50 and incomputer science (CS) 150 Among these 20 are taking bothEE and MA 25 are taking EE and CS and 10 are taking MAand CS Five of them are taking EE CS and MA Find thetotal number of students in the junior class

From the problem we can identify the sets A = EEstudents B = MA students and C = CS students and thecorresponding intersections are A B B C C A andA B C Here A = 100 B = 50 and C = 150 Weare also given A B = 20 B C = 10 and C A = 25and finally A B C = 5 Using Eq (126) the total numberof students are 100 + 50 + 150 minus 20 minus 10 minus 25 + 5 = 250

Complement

If A is a subset of the universal set S then the complement ofA denoted by Ā is the elements in S not contained in A or

A= SminusA= x x A S and x S 1 2 7

Example 124 If the universal set S = 123456 andA = 245 then the complement ofA is given byĀ = 136

Difference

The complement of a subset A S as given by Eq (127) isthe difference between the universal set S and the subset AThe difference between any two sets A and B denoted byA minus B is the set C containing those elements that are in Abut not in B and B minus A is the set D containing those elementsin B but not in A

C =AminusB= x x A and x B

D=BminusA= x x B and x A1 2 8

A minus B is also called the relative complement of B with respectto A and B minus A is called the relative complement of A withrespect to B It can be verified that A minus B B minus A

Example 125 If the set A is given as A = 2456 and B isgiven as B = 5678 then the difference A minus B = 24 andthe difference B minus A = 78 Clearly A minus B B minus A if A B

Symmetric Difference

The symmetric difference between sets A and B is written asAΔB and defined by

AΔB= AminusB BminusA

= x x A and x B or x x B and x A 1 2 9

Example 126 The symmetric difference between the setA given by A = 2456 and B given by B = 5678 inExample 125 is AΔB = 24 78 = 2478

The difference and symmetric difference for Examples125 and 126 are shown in Fig 124

Cartesian Product

This is a useful concept in set theory If A and B are two setsthe Cartesian product A times B is the set of ordered pairs (xy)such that x A and y B and is defined by

AtimesB= xy x A y B 1 2 10

When ordering is considered the cartesian product A timesB B times A The cardinality of a Cartesian product is the prod-uct of the individual cardinalities or A times B = A B

Example 127 If A = 234 and B = 56 then C = A timesB = (25) (26) (35) (36) (45) (46) with cardinality 3 times2 = 6 Similarly D = B times A = (52) (53) (54) (62) (63)

FIGURE 123

4 SETS FIELDS AND EVENTS

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 19: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

(64) with the same cardinality of 6 However C and D donot contain the same ordered pairs

Partition

A partition is a collection of disjoint sets Ai i = 1hellip n of aset S such that Ai over all i equals S and Ai Aj with i jempty Partitions play a useful role in conditional probabilityand Bayesrsquo theorem Figure 125 shows the concept of apartition

Example 128 If set S = 12345678910 then thecollections of sets A1 A2 A3 A4 where A1 = 135A2 = 79 A3 = 246 A4 = 810 is a partition of S asshown in Fig 126

A tabulation of set properties is shown on Table 121Among the identities De Morganrsquos laws are quite useful insimplifying set algebra

13 SET ALGEBRAS FIELDS AND EVENTS

Boolean Field

We shall now consider a universal set S and a collection ℱof subsets of S consisting of the sets Ai i = 12hellip nn + 1hellip The collection ℱ is called a Boolean field if thefollowing two conditions are satisfied

1 If Ai ℱ then Ai ℱ

2 If A1 ℱ and A2 ℱ then A1 A2 ℱ and hence ifAi i = 1hellipn ℱ then i = 1

n Ai ℱ

Let us see the consequences of these conditions being satis-fied If A1 ℱ and A1 ℱ then A1 A1 = S ℱ If S ℱthen S= ℱ If A1 A2 ℱ then by condition 1A1 A2 ℱ and by De Morganrsquos law A1 A2 ℱ HenceA1 A2 ℱ Also A1 minus A2 ℱ and (A1 minus A2) (A2 minus A1)ℱ Thus the conditions listed above are suffcient for any fieldℱ to be closed under all set operations

FIGURE 124

FIGURE 125FIGURE 126

TABLE 121 Table of set properties

Property Equation

Identity A = AA S = A

Domination A =A S = S

Idempotent A A = AA A = A

Complementation A=ACommutative A B = B A

A B = B A

Associative A (B C) = (A B) CA (B C) = (A B) C

Distributive A (B C) = (A B) (A C)A (B C) = (A B) (A C)

Noncommutative A minus B B minus A

De Morganrsquos law A B =A BA B =A B

SET ALGEBRAS FIELDS AND EVENTS 5

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 20: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

Sigma Field

The preceding definition of a field covers only finite setoperations Condition 3 for finite additivity may not holdfor infinite additivity For example the sum of 1 + 2 +22 + 23 + 24 = (25 minus 1)(2 minus 1) = 31 but the infinite sum1 + 2 + 22 + 23 + 24 + diverges Hence there is a needto extend finite additivity concept to infinite set operationsThus we have a σ field if the following extra condition isimposed for infinite additivity

4 If A1 A2 hellip An hellip ℱ then infini= 1Ai ℱ

Many σ fields may contain the subsets Ai of S but the smal-lest σ field containing the subsets Ai is called the Borel σfield The smallest σ field for S by itself is ℱ = S

Example 131 We shall assume that S = 123456Then the collection of the following subsets of S ℱ = S

(123) (456) is a field since it satisfies all the setoperations such as (123) (456) = S 123 = 456 However the collection of the following subsets of Sℱ1 = S 123 456 2 will not constitute a fieldbecause (2) (456) = (2456) is not in the field But wecan adjoin the missing sets and make ℱ1 into a field Thisis known as completion In the example above if we adjointhe collection of sets ℱ2 = 2456 13 to ℱ1 then theresulting collection ℱ =ℱ1 ℱ2 = S 123 456 2 2456 13 is a field

Event

Given the universal set S and the associated σ field ℱall subsets of S belonging to the field ℱ are called eventsAll events are subsets of S and can be assigned a probabilitybut not all subsets of S are events unless they belong to anassociated σ field

6 SETS FIELDS AND EVENTS

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 21: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

2PROBABILITY SPACE AND AXIOMS

21 PROBABILITY SPACE

A probability space is defined as the triplet Sℱ P whereS is the sample spaceℱ is the field of subsets of S and P is aprobability measure We shall now define these elements

Sample Space S

Finest-grain mutually exclusive and collectively exhaustivelisting of all possible outcomes of a mathematical experimentis called the sample space The outcomes can be either finitecountably infinite or uncountably infinite If S is countable wecall it a discrete sample space if it is uncountable we call itcontinuous sample space

Examples of Discrete Sample Space

Example 211 When a coin is tossed the outcomes areheads H or tails T Thus the sample space S = HT

Example 212 When a die is tossed the outcomes are123456 Thus the sample space is S = 123456

Example 213 The set of all positive integers is countableand the sample space is S = x x is a positive integer

Examples of Continuous Sample Space

Example 214 The set of all real numbers between 0 and 2is the sample space S = x 0 le x le 2

Example 215 The set of all real numbers x and y suchthat x is between 0 and 1 and y is between 1 and 2 is givenby S = x y 0 le x le 1 1 le y le 2

The continuous sample space given in Examples 214 and215 is shown in Fig 211

Field ℱ

A field has already been defined in Section 13 We shallassume that a proper Borel σ field can be defined for alldiscrete and continuous sample spaces

Probability Measure P

A probability measure is defined on the field ℱ of events ofthe sample space S It is a mapping of the sample space to thereal linebetween0and1 such that it satisfies the following threeKolmogorov axioms for all events belonging to the fieldℱ

Axiom 1

If Abelongs toℱ thenP A ge 0 nonnegativity 2 1 1

Axiom 2 The probability of the sample space equals 1

P S = 1 normalization 2 1 2

Axiom 3a If A and B are disjoint sets belonging toℱ thatis A B = then

P A B =P A +P B Boolen additivity 2 1 3

Axiom 3b If A1 A2 hellip is a sequence of sets belongingto the field ℱ such that Ai Aj = for all i j then

Probability and Random Processes Second Edition Venkatarama Krishnancopy 2016 John Wiley amp Sons Inc Published 2016 by John Wiley amp Sons Inc

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 22: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

P A1 A2 Ai =infin

i=1

P Aj sigma additivity

2 1 4

From these axioms we can write the following corollaries

Corollary 1

P = 0 2 1 5

Note that the probability of the empty event is zero butif the probability of an event A is zero we cannot con-clude that A = All sets with zero probability aredefined as null sets An empty set is also a null setbut null sets need not be empty

Corollary 2 Since A A= S and A A= we can write

P A = 1minusP A 2 1 6

Corollary 3 For any A and B that are subsets of S notmutually exclusive we can write

P A B =P A A B =P A +P A BP B =P A B A B =P A B +P A B

Eliminating P A B between the two equations weobtain

P A B =P A +P B minusP A B 2 1 7

Clearly if PA B = 0 implying A and B are mutuallyexclusive then Eq (217) reduces to Kolmogorovrsquosaxiom 3a If the probabilities PA and PB are equalthen sets A and B are equal in probability This does notmean that A is equal to B

The probability of the union of three sets can be calculatingfrom the Venn diagram for the three sets shown in Fig 123and redrawn in Fig 212

The results of the analysis for the cardinalities given inEq (126) can be extended to finding the P A B C We

subtract P A B +P B C +P C A from P A +P B +P C and add P A B C resulting in

P A B C =P A +P B +P C

minusP A B minusP B C minusP C A

+P A B C 2 1 8

Using the inclusionndashexclusion principle of Eq (125a) wecan also generalize Eq (218) to the probabilities of theunion of n sets A1 A2hellip An as follows

P ni=1Ai =

n

i=1

P Ai minusn

i j=1i j

P Ai Aj

+n

i j k=1i j k

P Ai Aj Ak

minus plusmn 1 nn

i j k=1i j khellip n

P Ai Aj Ak An

2 1 9

Example 216 A fair coin is tossed 3 times Since each tossis a functionally independent trial and forms a Cartesianproduct the cardinality of the three tosses is 2 times 2 times 2 = 8and the sample space consists of the following 8 events

S = HHH HHT HTT HTH THH

TTH THT TTT

In this sample space

FIGURE 211

FIGURE 212

8 PROBABILITY SPACE AND AXIOMS

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 23: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

P 2 heads =P HHH HHT HTH THH =48

P both heads and tails =

HHT HTT HTH THH TTH THT =68

Example 217 A telephone call occurs at random between900 am and 1100 am Hence the probability that thetelephone call occurs in the interval 0 lt t le t0 is given byt0120 Thus the probability that the call occurs between930 am and 1045 am is given by

P 30 lt t le 105 =105minus30120

=75120

=58

Example 218 Mortality tables give the probability den-sity that a person who is born today assuming that the lifespanis 100 years as

f t = 3 times 10minus9t2 100minus t2 u t

and the distribution function that the person will die at the aget0 years as

F t0 =P t le t0 =t0

03 times 10minus9t2 100minus t2 dt

These probabilities are shown in Fig 213 The probabilitythat this person will die between the ages 70 to 80 is givenby P 70 lt t le 80 = 0 1052 obtained from the followingintegral

P 70 lt t le 80 =F 80 minusF 70

=80

703 times 10minus9t2 100minus t2 dt = 0 1052

The density and distribution functions are shown inFig 213

22 CONDITIONAL PROBABILITY

The conditional probability of an event B conditioned on theoccurrence of another event B is defined by

P B A =P B A

P A

P B A =P B A P A

if P A 0 2 2 1

If P A = 0 then the conditional probability P B A is notdefined Since P B A is a probability measure it alsosatisfies Kolmogorov axioms

Conditional probability P B A can also be interpreted asthe probability of the event B in the reduced sample spaceS1 = A Sometimes it is simpler to use the reduced samplespace to solve conditional probability problems We shalluse both these interpretations in obtaining conditionalprobabilities as shown in the following examples

Example 221 In the tossing of a fair die we shall calcu-late the probability that 3 has occurred conditioned on the tossbeing odd We shall define the following events

A = odd number B= 3

P A = 12 and P B = 1

6 and the event A B = 3 and hence

P 3 odd =P 3

P odd=

1612

=13

Note that P 2 odd = 0 since 2 is an impossible event giventhat an odd number has occurred Also note that P 3 3 = 1FIGURE 213

CONDITIONAL PROBABILITY 9

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 24: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

The conditional probability of 3 occurring given that 3 hasoccurred is obviously equal to 1

We can also find the solution using the concept ofreduced sample space Since we are given that an oddnumber has occurred we can reduce the sample space fromS= 123456 to S1 = 135 as shown in Fig 221

The probability of 3 in this reduced sample space is 13

agreeing with the previous result

Example 222 Consider a family of exactly two childrenWe will find the probabilities (1) that both are girls (2) bothare girls given that one of them is a girl (3) both are girlsgiven that the elder child is a girl and (4) both are girls giventhat both are girls

The sample space S has 4 points gg gb bg bbwhere g isgirl and b is boy The event B = both children are girls Theconditioning event Awill depend on the following four cases

1 The event A = S Hence P B = 14

2 The event A = one child is a girl The reduced samplespace is S1 = gg gb bg and the conditional probabil-ity P B A = 1

3

3 The event A = elder child is a girl In this case thereduced sample space is S2 = gg gb andP B A = 1

2

4 The event = both are girls The reduced sample spacehas only one point namely S3 = gg and P B A = 1

As our knowledge of the experiment increases by theconditioning event the uncertainty decreases and the proba-bility increases from 025 through 0333 05 and finally to 1meaning that there is no uncertainty

Example 223 The game of craps as played in Las Vegashas the following rules A player rolls two dice He wins onthe first roll if he throws a 7 or a 11 He loses if the first throwis a 2 3 or 12 If the first throw is a 4 5 6 8 9 or 10 it iscalled a point and the game continues He goes on rollinguntil he throws the point for a win or a 7 for a loss We haveto find the probability of the player winning

We will solve this problem both from the definition ofconditional probability and the reduced sample spaceFigure 222 shows the number of ways the sums of the pipson the two dice can equal 2 3 4 5 6 7 8 9 10 11 12 andtheir probabilities

Solution Using Definition of Conditional ProbabilityThe probability of winning in the first throw is

P 7 +P 11 =636

+236

=836

= 0 22222

The probability of losing in the first throw is

P 2 +P 3 +P 12 =136

+236

+136

=436

= 0 11111

To calculate the probability of winning in the secondthrow we shall assume that i is the point withprobability p The probability of not winning inany given throw after the first is given byr =P not i and not 7 = 1minuspminus 1

6 We compute theconditional probability

P win i in first throw = p+ rp + r2p+

=p

1minusr=

p

p+ 16

as an infinite geometric series

P win in second or subsequent throws =p2

p + 16

The probability p of making the point i = 4568910 isobtained from Fig 222 Thus for i = 456 we obtain

FIGURE 222

FIGURE 221

10 PROBABILITY SPACE AND AXIOMS

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 25: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

P win after point i= 4 =336

2 336

+16

=136

P win after point i= 5 =436

2 436

+16

=245

P win after point i= 6 =536

2 536

+16

=25396

with similar probabilities for 8910 Thus the probabilityof winning in craps is

P win =P win in roll 1 +P win in roll 23hellip

=836

+ 2136

+245

+25396

=244495

= 0 4929

Solution Using Reduced Sample Space We can seefrom the figure that there are 6waysof rolling 7 for a loss(i minus 1 i = 4 5 6) ways of rolling the point for awin and(13 minus i i = 8 9 10) ways of rolling the point for a winAfter the point i has been rolled in the first throw thereduced sample space consists of (i minus 1 + 6) pointsfor i = 4 5 6 and (13 minus i + 6) points for i = 8 910 in which the game will terminate Thus

P win in roll 23hellip i in roll 1 =P W i

=

iminus1iminus1 + 6

i= 456

13minus i13minus i+ 6

i= 8910

Thus P W 4 = 39 P W 5 = 4

10 P W 6 = 511 P W

8 = 511 P W 9 = 4

10 P W 10 = 39 Hence probabil-

ity of a win after the first roll is P W i P i Hence

P win after first roll

= 239

336

+410

436

+511

536

=134495

P win =836

+ 239

336

+410

436

+511

536

=836

+134495

+244495

= 0 4929

a result obtained with comparative ease

Example 224 We shall continue Example 218 and findthe probability that a person will die between the ages 70 and80 given that this individual has lived upto the age 70 Wewill define the events A and B as follows

A= person has lived upto70B= person dies between ages 70 and 80

The required conditional probability equals the number ofpersons who die between the ages 70 and 80 divided bythe number of persons who have lived upto the age 70

P B A =P 70 lt t le 80 and t ge 70

P t ge 70=P 70 lt t le 80P t ge 70

=

80

703 times 10minus9t2 100minus t2 dt

100

703 times 10minus9t2 100minus t2 dt

=0 1050 163

= 0 644

This type of analysis is useful in setting up actuarial tables forlife insurance purposes

23 INDEPENDENCE

We have already seen the concept of functional independ-ence of sets in Chapter 1 We will now define statisticalindependence of events If A and B are two events in a fieldof eventsℱ then A and B are statistically independent if andonly if

P A B =P A P B 2 3 1

Statistical independence neither implies nor is implied byfunctional independence Unless otherwise indicated we willcall statistically independent events ldquoindependentrdquo withoutany qualifiers

Three events A1 A2 A3 are independent if they satisfy inaddition to

P A1 A2 A3 =P A1 P A2 P A3 2 3 2

the pairwise independence

P Ai Aj =P Ai P Aj i j = 123 i j 2 3 3

Note that if A and B are mutually exclusive then from Kol-mogorovrsquos axiom 3a we obtain

P A B =P A +P B 2 3 4

The concepts of independence and mutual exclusivity arefundamental to probability theory For any two sets A andB we recall from Eq (217) that

P A B =P A +P B minusP A B 2 1 7

INDEPENDENCE 11

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS

Page 26: Thumbnail - download.e-bookshelf.deVishnu sahasranamam Shree rāma rāma rāmēti ramē rāmē manoramē Sahasra nāmatathtulyam rāmanāma varānanē This book is respectfully dedicated

If A and B are mutually exclusive then P A B = 0 but ifthey are independent then P A B =P A P B If this hasto be zero for mutual exclusivity then one of the probabilitiesmust be zero Therefore we conclude that mutual exclusivityand independence are incompatible unless we are dealingwith null events

We shall now give examples to clarify the concepts offunctional independence and statistical independence

Example 231 The probability of an ace on the throw of adie is 1

6 The probability of a head on the throw of a coin is 12

Since the throw of a die is not dependent on the throw of acoin we have functional independence and the combinedsample space is the Cartesian product with 12 elements asshown in Table 231

We assume as usual that all the 12 elements are equiprob-able events and the probability of ace and head is 1

12 sincethere is only one occurrence of 1 H among the 12 pointsas seen in the first row and first column in Table 231 UsingEq (231) we have P 1 H =P 1 P H = 1

6 times12 =

112

illustrating that in this example functional independencecoincides with statistical independence

Example 232 Two dice one red and the other blue aretossed These tosses are functionally independent and wehave the Cartesian product of 6 times 6 = 36 elementary eventsin the combined sample space where each event is equiprob-able We seek the probability that an event B defined by thesum of the numbers showing on the dice equals 9 There arefour points (6 3) (5 4) (4 5) (3 6) and henceP B = 4

36 =19 We now condition the event B with an event

A defined as the red die shows odd numbers Clearly theevents A and B are functionally dependent The probabilityof the event A is P A = 18

36 =12 We want to determine

whether the events A and B are statistically independentThese events are shown in Fig 231 where the firstnumber is for the red die and the second number is for theblue die

From Fig 231 P A B =P 36 54 = 236 =

118

and =P A timesP B = 12 times

19 =

118 showing statistical independ-

enceWe compute P B A from the point of view of reducedsample space The conditioning event reduces the samplespace from 36 points to 18 equiprobable points and the eventB A = 36 54 Hence P B A = 2

18 =19 =P B

or the conditioning event has no influence on B Here eventhough the events A and B are functionally dependent theyare statistically independent

However if another set C is defined by the sum beingequal to 8 as shown in Fig 232 then P C = 5

36Here the events C and A are not statistically independ-

ent because P C P A = 536 times

12 =

572 P C A =P 35

53 = 218 =

472

In this example we have the case where the events A andC are neither statistically independent nor functionallyindependent

24 TOTAL PROBABILITY AND BAYESrsquoTHEOREM

Let Ai i= 1hellipn be a partition of the sample space and letB an arbitrary event in the sample space S as shown inFig 241

We will determine the total probability PB given theconditional probabilities P B A1 P B A2 hellip P B An The event B can be given in terms of the partition as

B= B S =B A1 A1 An

= B A1 B A2 B An2 4 1

TABLE 231

1 H 2 H 3 H 4 H 5 H 6 H

1 T 2 T 3 T 4 T 5 T 6 T

FIGURE 232

FIGURE 231

12 PROBABILITY SPACE AND AXIOMS