1Lecture 3• HW2 solutions

• Reviewing Lecture 2+ some comments

• About entropy and disorder

• More about Mathematica.

• Random and pseudo-random numbers.

• Our first probabilistic experiments with Mathematica

• Student’s presentation: paradoxes of probability

21. Reminder

Multiplication RuleIf N experiments are performed in order and the number of outcomes for the k-th experiment is nk then the total number of all possible outcomes is n1n2…nN

•Number of all possible permutations between N different objects is N! whereN! = 1 *2 *3 …. (N-2)* (N-1) *N is called “N factorial”. If there are some groups of “identical” * (or indistinguishable) objects (with m, s, p, etc. elements) the the total number of distinct permutations is N!/(m!s!p!…)

•Number of permutations of N objects taken m in a time is PN,m = N!/(N-m)!

Note 1: These formulas are used when the order of objects mattersNote 2: The convention is that 0!=1

Number of combinations of n objects taken m in a time: is Cn.m=Pn,m/m!= n!/[m!(n-m)!] (used when order of objects does not matter)

* The “identity” is defined by the context. The objects should not be physically identical to fit this requirement. (in fact, “physically identical” objects can only exist on the sub-atomic level). For instance, if we only care about the eye color, then all the blue-eyed people are identical in the context defined by this experiment.

3

Examples:

1.1 Counting sequences of distinguishable and indistinguishable objects.

How many different arrangements are there of the letters comprising the word “ALL”. The answer depends on whether or not you can tell the L’s apart. Suppose first that one L has subscript “1” and the other has subscript “2”. Then all the letter are distinguishable and there is W= 3! different arrangements:AL1L2, AL2L1, L1AL2, L2AL1, L1L2A, L2L1A

However, if two A’s are indistinguishable from each other and have no subscripts, then there are only W=3 distinct sequences:ALL, LAL, LLA. More generally, W = N!/NL!. The nominator is the number of permutations if all the characters were distinguishable and the denominator NL! corrects for “overcounting”.

4

Examples:

1.2 Bose-Einstein statistics. This example is discussed in Molecular Driving Forces by Ken A. Dill and Sarina Bromberg © 2003, p. 13(click here)

How many ways can n indistinguishable particles put into M boxes, with any number of particles per box. This problem is related to the properties of the microscopic particles called “bosons”, such as photons and He4 atoms.

(These particles possess integer spin, while the fermions, such as protons, electrons, possess half-integer spin).

The problems has its own value and we won’t discuss its relation to B-E in more details. Consider first 3 balls and two boxes.

5There are 4 ways to distribute 3 balls between 4 boxes (Fig. (a)) Fig (b) demonstrates that this problem can be solved by considering a linear array of n particles and m-1 movable walls.

W(3,2)= 4!/3! = 4. For n particles and m boxes the answer is W(n,M) = (n+M-1)!/[n!(M-1)!].

Try to understand it at home.

6

1. The binomial distribution

Conditions: Two outcomes, with probabilities p (success) and q=1-p (failure); the number of trials (experiments) is fixedQuestion : The probability of k successes in n experimentsAnswer: p(n,m)=Cn,k pkqn-k

Try it:

Try proving that in 3 flips of a coin the total probability of getting either 0 or 1 or 2 or 3 heads equals 1.Can you generalize it for n flips?

Some distributions for the repetitive independent experiments.

7

2. The multinomial distributionConditions: m possible outcomes with probabilities p1,p2,..,pm. The number of trials, n , is not fixed.

Question: What is the probability of getting exactly ni outcomes of type i in n=

n1+n2+..+nm experiments?Answer:

Some distributions for the repetitive independent experiments.

1 21 2 1 2

1 2

!( , ,... ) ( ) ( ) ( )! !... !

nn nn kp n n n p p pk kn n n

k

8Reminder 3. The geometric distribution

Conditions: p and q=1-p are the success and failure probabilities.Question : what is the probability that the first success occurs in the experiment number k?Answer: p(n=k)=qk-1p

4. The negative binomial distributionConditions: the same two sorts of outcomes, probabilities p and q. The number of trials, Tn , is not fixed.Question: What is the probability that Tn trials will be required for getting exactly n successful outcomes?Answer:

Note1: there was a typo in the lecture: Cn+k-1,k was erroneously written and Cn+k-1

Note2: I am grateful to Michael Z. for a wonderful example of the NBD:

In a computer game, one has to apprehend 7 Monsters randomly positioned in 20 rooms (only one Monster per room is allowed). What is the probability that the last (and the most dangerous) Monster will be hiding in the 12-th room?

Can you answer this question?

11

( ) ( ).

n kp T n k C p pn k n k

9Let’s discuss a question

What is the probability of getting 5 Heads tossing 10 coins.

Make your guess.

Is it 1?

0.5?

0.25?

0.1?

10

The solution is given by the Binomial distribution.

The probability of n heads out of 10 is

P(10,n)= C10,n(1/2)n(1/2)10-n=C10,n /210

Let’s solve it with Mathematica

11

12

As you can see, the probability function p(H) reaches its maximum at H= 5. However, p(H=5) does not equal 0.5, but rather ~0.25.

Why is that?

How p(n/2) depends on n?

Try finding it for n=20,50,100. What is the tendency?

Try plotting p(H) for these case, changing H from 0 to n.

13

In addition to Lect 2:

Hypergeometric distribution

In the Bernoulli process that lead to Binomial and Geometric distribution, it was assumed that the trials were independent.

Sometimes this is not the case.

For example, consider urn containing 5 white and 10 black marbles. What is the probability of obtaining exactly 2 white and 3 black marbles in in drawing 5 marbles without replacement from the urn.

It is clear that if we draw with replacement, the trials are independent and we can use the binomial distribution (what will be the answer?)

However, when we draw without replacement, the probability changes after each draw. We already solved this kind of problems before. Let’s now formalize the solution.

14We obtain our probability as follows:1. Find total number of ways we can draw 5 marbles out of 15 (C15,5).2. Find the total ways we can draw 2 white marbles out of 5 (C5,2)

3. The same for 3 black marbles out of 10 (C10,3).

4. Multiply (2) by (3) to find the total ways that we can be successful ((C5,2* C10,3).

4. Divide the total ways of being successful by the total ways of drawing 5 marbles ((4) divided by (1)] to find the probability.

That is:P(2 white and 3 black)= C5,2* C10,3/ C15,5~0.4

In general, if we letA= number in the population having a given characteristicB = the same with another characteristica= number of A found in n trialsb = number of B found in n trials

then

This is called hypergeometric distributionbaBA

bBaA

C

CCbaP

,

,,),(

15

Problem:

On a board of directors composed of 10 members, 6 in favor of certain decision and 4 oppose it.

What is the probability that a randomly chosen subcommittee of 5 will contain exactly 3 who favor the change?

What is the probability that a subcommittee of 5 will vote against the change?

(This sort of problems will be included in the test).

16A short insert about entropy and disorder

Why does the atmosphere get thinner with elevation?

There are two major “forces” in this problem. One is a force of gravity. It is responsible for the potential energy of the molecules, Ep(h)=gmn(h) h (3.1)

relative to the ground level (here m is the molecular mass for some of the

atmospheric gases and n(h) is its concentration at the elevation h) . This force is trying to collect all the molecules at the ground level.

Various Natural phenomena are often described in terms of optimization, or minimization of different “energy functions”. If the Nature just minimized the potential energy, all the atmosphere would’ve condensed on the surface, and life would become impossible.In fact, the Nature is trying to minimize a different function, F, which includes Ep(h) as just one of its components. In our problem, where the gas is considered ideal (the interaction between the molecules is neglected), F can be presented as

17

T is the temperature. The function S is called “entropy”. It plays the most important role in statistical physics and thermodynamics. In fact, it’s a gateway for the probability.Entropy describes a universal tendency towards disorder.

For the ideal gas model,

S=-k n(h)ln(n(h)) (3.3)

Inserting Eqs. (3.1) and (3.3) in (3.3) and using the conditions F=minn(F(n)), one can find

n(h)=n(0) Exp(-h/RT) (3.4)

F=Ep-T S (3.2)

18 μ is the molar mass, and R is the gas constant.It is clear from the Eq. 3.2 that the role of S increases proportionally to the temperature T. That’s why it’s instructive to analyze how the n(h) depends on T. Let’s consider the exponential distribution function (3.4) and draw it for three different temperatures T1<T2<T3.

19What lesson do we learn from this picture?

Each of the curves represents a probability of finding a gas molecule at elevation h. When T grows, the distribution becomes more uniform. At high temperatures,e.g. T=9T1 , the probability of finding a molecule at h=4 is almost as big as at h=0. The distribution becomes shallower and wider. This behavior can be characterized as an increasing disorder.

The tendency is quite universal. An example with a crystal of sugar placed in water is illustrative. The crystal, which symbolizes the order in Nature, turns into a solute, an absolutely disordered state, similar to the gas phase. Note: the generalized distribution (3.4),

n(r)=n(0) Exp(-EP(r)/kT) (3.5)

is known as “Boltzmann distributions”.

20A few more words about relation between the entropy and probability

“The shuffling of a deck of cards, the spray of an aerosol can, the flow of heat from warm things to cold things, the breaking of an egg; these are all examples of what physicists call " irreversible processes" . They occur very naturally, but all the king's horses and all the king's men can't undo them”.

When you drop an egg on the floor, it breaks. But dropping a broken egg on the floor doesn't cause it to become whole.      A new deck of cards comes with all the cards in order. Shuffling them mixes up the cards. Now take a deck of    cards that's mixed up and shuffle them. The cards do not come back to their original order.      Pick up a can of air freshener and push down the button. The air freshener spews out of the can and spreads out around the room. Now try gathering up the air freshener and putting it back in the can. Doesn't work, does it?     

21A few more words about relation between the entropy and probability

In the isolate system entropy never decreases:

All processes in the isolated system are directed towards the macroscopic state with higher probability.

Two examples: (1) Energy flow from a hot to a cold object(2) Gas always flow from the the higher

density to the lower

Let’s consider a second case in more details to illustrate a relation between the entropy and probability

22The probability of each “macro-state” is determined by the # of possible configurations (microstates) corresponding to this macro-state.

Events (states) with higher entropy have higher probability.

Example: connected containers with N distinguishable particles.

N= nL+nR. The “macrostate” can be described by (N,nL)

23

D

B CA

For (4,1) there are 4 microstates:

(A,BCD), (B,ACD),{C,ABD),{D,ABC)

Calculate how many states are there in configuration

(4,2):

2?

4?

8?

24

The answer is 6.

You probably noticed that

The total number of microstates is 2N.

Then we find for probability:

P(N,n)=CN,n/2N=N!/(2N n!(N-n)!)

Analyzing this expression you will find (similar to N coins) that n=N/2 corresponds to the most probable state.

)!(!

!),( ,

LLnN nNn

NCnN

L

25

The entropy is defined as

Thus, it is proportional to ln(P):

It is increases when the macroscopic state becomes more probable.

This statement describes one of the major links between the Nature and Probability.

)ln(

26

The fight between the Chaos and the Order can be discovered in every aspect of our life.

If we do not take special actions, the disorder around us will always increase: desk, clothing, home will become messier and messier. They never come back to the order by themselves. But does it have anything to do with Nature? Sure it does.

Why the disorder always increases if you give it a chance? The answer is quite simple: while the ordered state can be achieved by only a few arrangements (choices), the disorder is represented by practically infinite number of them.

27A child carefully builds a tower from N blocks, placing them in the alphabetical order. There is only one proper arrangement.The task is not an easy one. It takes a lot of time and efforts.

Another child simply touches it, and disorder is achieved in a moment. No time or efforts are required.

A few manifestations of the entropy laws.

• Easier to destroy than build

• The information is costly and vulnerable.

• Energy can not be used completely. Whenever there is a transfer of energy, some of the potential energy is lost as heat, the random movement of molecules. 

28A possible project: “Entropy, Information and probability”.Read Claude Shannon and related links

The Information is often described as “negentropy” (although sometimes this term can be misleading). It is associated with rear occurrences, unlikely events. In our example with Encyclopedia Britannica, the “information” can be associated with the correct order of the volumes on the shelf. As we have shown before, the probability of finding it at random is much less that a probability of pulling one particular atom from the body of our planet.

Scientific discoveries, elaborate architectural design and masterpieces of arts are so valuable exactly because they exist against the odds of random sampling. The probability that monkey hitting randomly a keyboard will type just “To be or not to be” is ~10-15 (check this estimate).

The project can run in any direction, including possible applications of the information theory in molecular biology, molecular computations, etc. (click here for references).

29Random and pseudorandom numbers

The experiments with real dice are very slow. To overcome this shortcomings, the computer experiments can be used. To deal with randomness, computers are equipped with so called "Random number generators".

They use the sequences of numbers that, on one hand, are fixed, and in this sense are completely deterministic, but, on the other hand, they display every sign of randomness.

Suggested reading: “Randomness” by Deborah J. Bennett, Ch. 8.

30One of the most popular methods for generating random numbers is the linear congruential generator. These generators use a method similar to the folding schemes in chaotic maps. The general formula is,

The values a, c and m are pre-selected constants. a is known as the multiplier, c is the increment, and m is the modulus.

The quality of the generator is strongly dependent upon the choice of these constants The method is appealing however, because once a good set of the parameters is found, it is very easy to program. One fairly obvious goal is to make the period (the time before the generator repeats the sequence) long, this implies that m should be as large as possible. This means that 16 bit random numbers generated by this method have at most a period of 216= 65,536, which is not nearly long enough for serious applications.

The choice of a = 1277, c = 0, m = 131072 looks okay for a while but eventually has problems. A scatter plot for this for 2000 pairs from this generator reveals linear bands emerging from the plot. A random walk plot shows that after a while the slope changes (it should be a constant slope).

1( )modk kI aI c m

31

In the Mathematica program that we will be using, the random numbers are generated by the function called Random[ ]. For instance, Random[Integer,{1,6}] produces a result of single through of die. To imitate n consecutive tests, one has to fill with random numbers a table: Table[Random[Integer,{1,6}], {n}].

The choice of a = 16807, c = 0, m = 2147483647 is a very good set of parameters for this generator Park & Miller (1988). This generator often known as the minimal standard random number generator, it is often (but not always) the generator that used for the built in random number function in compilers and other software packages.

To learn about the more advanced methods, and about the differences between the pseudo-random and Quasi-random numbers, read E,F. Carter.

32

Monte-Carlo estimate for π

Let’s apply our new skills to calculate the magic number π.Here is the idea behind. Suppose that we draw a circle of radius R=1.The

numeric value of its area is π. Now we insert this circle in a square 3*3. Suppose than we started shooting randomly in the square, and count the holes. Let Nin the number of bullets that hit inside the circle,while N is the total number of holes.At large N the ratio Nin / N should become

proportions to the π /9. This provides a

an opportunity of estimating π using the random number generator.

Here is how it can be done with Mathematica.

33

The rest of this class we will be practicing with Mathematica. Please, follow steps 1-4 from the Mathematica Lessons link.