Learning with Bayesian Networks
David Heckerman
Presented by Colin Rickert
Introduction to Bayesian Networks
Bayesian networks represent an advanced form of general Bayesian probability
A Bayesian network is a graphical model that encodes probabilistic relationships among variables of interest1
The model has several advantages for data analysis over rule based decision trees1
Outline
1. Bayesian vs. classical probability methods
2. Advantages of Bayesian techniques3. The coin toss prediction model from a
Bayesian perspective4. Constructing a Bayesian network with
prior knowledge5. Optimizing a Bayesian network with
observed knowledge (data)6. Exam questions
Bayesian vs. the Classical Approach
The Bayesian probability of an event x, represents the person’s degree of belief or confidence in that event’s occurrence based on prior and observed facts.
Classical probability refers to the true or actual probability of the event and is not concerned with observed behavior.
Bayesian vs. the Classical Approach
Bayesian approach restricts its prediction to the next (N+1) occurrence of an event given the observed previous (N) events.
Classical approach is to predict likelihood of any given event regardless of the number of occurrences.
Example
Imagine a coin with irregular surfaces such that the probability of landing heads or tails is not equal.
Classical approach would be to analyze the surfaces to create a physical model of how the coin is likely to land on any given throw.
Bayesian approach simply restricts attention to predicting the next toss based on previous tosses.
Advantages of Bayesian Techniques
How do Bayesian techniques compare to other learning models? 1. Bayesian networks can readily handle
incomplete data sets.
2. Bayesian networks allow one to learn about causal relationships
3. Bayesian networks readily facilitate use of prior knowledge
4. Bayesian methods provide an efficient method for preventing the over fitting of data (there is no need for pre-processing).
Handling of Incomplete Data
Imagine a data sample where two attribute values are strongly anti-correlated
With decision trees both values must be present to avoid confusing the learning model
Bayesian networks need only one of the values to be present and can infer the absence of the other: Imagine two variables, one for gun-owner
and the other for peace activist. Data should indicate that you do not
need to check both values
Learning about Causal Relationships
We can use observed knowledge to determine the validity of the acyclic graph that represents the Bayesian network.
For instance is running a cause of knee damage? Prior knowledge may indicate that this is
the case. Observed knowledge may strengthen or
weaken this argument.
Use of Prior Knowledge and Observed Behavior
Construction of prior knowledge is relatively straightforward by constructing “causal” edges between any two factors that are believed to be correlated.
Causal networks represent prior knowledge where as the weight of the directed edges can be updated in a posterior manner based on new data
Avoidance of Over Fitting Data
Contradictions do not need to be removed from the data.
Data can be “smoothed” such that all available data can be used
The “Irregular” Coin Toss from a Bayesian Perspective
Start with the set of probabilities = {1,…,n} for our hypothesis.
For coin toss we have only one representing our belief that we will toss a “heads”, 1- for tails.
Predict the outcome of the next (N+1) flip based on the previous N flips: for 1, … ,N D = {X1=x1,…, Xn=xn} Want to know probability that Xn+1=xn+1 = heads
represents information we have observed thus far (i.e. = {D}
Bayesian Probabilities Posterior Probability, p(|D,): Probability of a
particular value of given that D has been observed (our final value of ) . In this case = {D}.
Prior Probability, p(|): Prior Probability of a particular value of given no observed data (our previous “belief”)
Observed Probability or “Likelihood”, p(D|,): Likelihood of sequence of coin tosses D being observed given that is a particular value. In this case = {}.
p(D|): Raw probability of D
Bayesian Formulas for Weighted Coin Toss (Irregular Coin)
where
*Only need to calculate p( |D,) and p(|), the rest can be derived
Integration
To find the probability that Xn+1=heads, we must integrate over all possible values of to find the average value of which yields:
Expansion of Terms
1. Expand observed probability p(|D,):
2. Expand prior probability p(|):
*“Beta” function yields a bell curve upon integration which is a typical probability distribution. Can be viewed as our expectation of the shape of the curve.
Beta Function and Integration
Integrating gives the desired result:
Combine product of both functions to yield
Key Points
Multiply the results of the beta function (prior probability) with results of the coin toss function for (observed probability). Result is our confidence for this value of .
Integrating the product of the two with respect to over all values of 0<<1 , is necessary to yield the average value that best fits the observed facts + prior knowledge.
Bayesian Networks
1. Construct prior knowledge from graph of causal relationships among variables.
2. Update the weights of the edges to reflect confidence of that causal link based on observed data (i.e. posterior knowledge).
Example Network
Consider a credit fraud network designed to determine the probability of credit fraud based on certain events
Variables include: Fraud(f): whether fraud occurred or not Gas(g): whether gas was purchased within 24 hours Jewelry(J): whether jewelry was purchased in the last
24 hours Age(a): Age of card holder Sex(s): Sex of card holder
Task of determining which variables to include is not trivial, involves decision analysis.
Construct Graph Based on Prior Knowledge
If examining all possible causal networks, there are n! possibilities to consider when trying to find the best one
Search space can be reduced with domain knowledge:
If A is thought to be a “cause” of B then add edge A->B
For all nodes that do not have a causal link we can check for conditional independence between those nodes
Example
Using the above graph of expected causes, we can check for conditional independence of the following probabilities given initial sample data
p(a|f) = p(a)
p(s|f,a) = p(s)
p(g|f,a, s) = p(g|f)
p(j|f,a,s,g) = p(j|f,a,s)
Construction of “Posterior” knowledge based on observed data:
For every node i, we construct the vector of probabilities ij = {ij1,…,ijn} where ij is represented as row entry in a table of all possible combinations j of the parent nodes 1,…,n
The entries in this table are the weights that represent the degree of confidence that nodes 1,…,n influence node i (though we don’t know these values yet)
Determining Table Values for i
How do we determine the values for ij?
Perform multivariate integration to find the average ij for all i and j in a similar manner to the coin toss integration: Count all instances “m” that satisfy a
configuration ijk then observed probability for ijk becomes ijkm(1- ijk )n-m
Integrate over all vectors ijk to find the average value of each ijk
Question1: What is Bayesian Probability?
A person’s degree of belief in a certain event
i.e. Your own degree of certainty that a tossed coin will land “heads”
Question 2: What are the advantages and disadvantages of the Bayesian and classical approaches to probability?
Bayesian Probability: + Reflects an expert’s knowledge +Compiles with rules of probability - Arbitrary
Classical Probability: + Objective and unbiased - Generally not available
Question 3: Mention at least 3 Advantages of Bayesian analysis
Handle incomplete data sets Learning about causal relationships Combine domain knowledge and
data Avoid over fitting
Conclusion
Bayesian networks can be used to express expert knowledge about a problem domain even when a precise model does not exist
Top Related