Logistics - cs.cmu.edu
Transcript of Logistics - cs.cmu.edu
1
1
School of Computer Science
Introduction
Probabilistic Graphical Models (10Probabilistic Graphical Models (10--708)708)
Lecture 0, Sep 10, 2007
Eric XingEric XingReceptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
X1 X2
X3 X4 X5
X6
X7 X8
Reading:
Eric Xing 2
Class webpage:http://www.cs.cmu.edu/~epxing/Class/10708-07/
Logistics
2
Eric Xing 3
LogisticsNo formal text book, but draft chapters will be handed out in class:
M. I. Jordan, An Introduction to Probabilistic Graphical ModelsDaphne Koller and Nir Friedman, Structured Probabilistic Models
Mailing Lists: To contact the instructors: [email protected] Class announcements list: [email protected].
TA:Hetunandan Kamichetty, Doherty 4302C, Office hours: Wednesdays, 5:00-6:00 pm Dr. Ramesh Nallapati
Class Assistant:Monica Hopes, Wean Hall 4616, x8-5527
Eric Xing 4
Logistics4 homework assignments: 45% of grade
Theory exercisesImplementation exercises
Final project: 30% of gradeApplying PGM to your research area
NLP, IR, Computational biology, vision, robotics …Theoretical and/or algorithmic work
a more efficient approximate inference algorithma new sampling scheme for a non-trivial model …
Take home final: 25% of gradeTheory exercises and/or analysis
Policies …
3
Eric Xing 5
Past projects:
We will have a prize for the best project(s) …
Winner of the 2005 project:J. Yang, Y. Liu, E. P. Xing and A. Hauptmann, Harmonium-Based Models for Semantic Video Representation and Classification , Proceedings of The Seventh SIAM International Conference on Data Mining (SDM 2007). (Recipient of the BEST PAPER Award)
Other projects:Andreas Krause, Jure Leskovec and Carlos Guestrin, Data Association for Topic Intensity Tracking, 23rd International Conference on Machine Learning (ICML 2006).
Y. Shi, F. Guo, W. Wu and E. P. Xing,GIMscan: A New Statistical Method for Analyzing Whole-Genome Array CGH Data, The Eleventh Annual International Conference on Research in Computational Molecular Biology (RECOMB 2007).
Eric Xing 6
What is this?
Classical AI and ML research ignored this phenomena
The Problem (an example): you want to catch a flight at 10:00am from Pitt to SF, can I make it if I leave at 7am and take a 28X at CMU?
partial observability (road state, other drivers' plans, etc.)noisy sensors (radio traffic reports)uncertainty in action outcomes (flat tire, etc.)immense complexity of modeling and predicting traffic
Reasoning under uncertainty!
4
Eric Xing 7
A universal task …
Speech recognitionSpeech recognition
Information retrievalInformation retrieval
Computer visionComputer vision
Robotic controlRobotic control
PlanningPlanning
GamesGames
EvolutionEvolution
PedigreePedigree
Eric Xing 8
RepresentationHow to capture/model uncertainties in possible worlds?How to encode our domain knowledge/assumptions/constraints?
InferenceHow do I answers questions/queries according to my model and/or based given data?
LearningWhat model is "right" for my data?
The Fundamental Questions
??
?
?
X1 X2 X3 X4 X5
X6 X7
X8
X9
)|( :e.g. DiXP
);( maxarg :e.g. MMM
DFM∈
=
5
Eric Xing 9
Graphical ModelsGraphical models are a marriage between graph theory and probability theory
One of the most exciting developments in machine learning (knowledge representation, AI, EE, Stats,…) in the last two decades…
Some advantages of the graphical model point of viewInference and learning are treated togetherSupervised and unsupervised learning are merged seamlesslyMissing data handled nicely A focus on conditional independence and computational issuesInterpretability (if desired)
Are having significant impact in science, engineering and beyond!
X1 X2 X3 X4 X5
X6 X7
X8
X9
Eric Xing 10
What is a Graphical Model?The informal blurb:
It is a smart way to write/specify/compose/design exponentially-large probability distributions without paying an exponential cost, and at the same time endow the distributions with structured semantics
A more formal description:It refers to a family of distributions on a set of random variables that are compatible with all the probabilistic independence propositions encoded by a graph that connects these variables
A
C
F
G H
ED
BA
C
F
G H
ED
B A
C
F
G H
ED
BA
C
F
G H
ED
BA
C
F
G H
ED
B
)( 87654321 ,X,X,X,X,X,X,XX P),()(),(
)|()|()|()()()( :
65867436
25242132181
XXXPXXPXXXPXXPXXPXXXPXPXPXP =
6
Eric Xing 11
probabilisticprobabilisticgenerativegenerative
modelmodel
gene expression profilesgene expression profiles
Statistical Inference
Eric Xing 12
statisticalstatisticalinferenceinference
gene expression profilesgene expression profiles
Statistical Inference
7
Eric Xing 13
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
Multivariate Distribution in High-D Space
A possible world for cellular signal transduction:
Eric Xing 14
Representation: what is the joint probability dist. on multiple variables?
How many state configurations in total? --- 28
Are they all needed to be represented?Do we get any scientific/medical insight?
Learning: where do we get all this probabilities? Maximal-likelihood estimation? but how many data do we need?Where do we put domain knowledge in terms of plausible relationships between variables, and plausible values of the probabilities?
Inference: If not all variables are observable, how to compute the conditional distribution of latent variables given evidence?
Computing p(H|A) would require summing over all 26 configurations of the unobserved variables
),,,,,,,,( 87654321 XXXXXXXXP
Recap of Basic Prob. Concepts
A
C
F
G H
ED
BA
C
F
G H
ED
BA
C
F
G H
ED
BA
C
F
G H
ED
B
8
Eric Xing 15
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
What is a Graphical Model?--- example from a signal transduction pathway
A possible world for cellular signal transduction:
Eric Xing 16
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor B
Membrane
Cytosol
Nucleus
X1 X2
X3 X4 X5
X6
X7 X8
GM: Structure Simplifies Representation
Dependencies among variables
9
Eric Xing 17
If Xi's are conditionally independent (as described by a PGM), the joint can be factored to a product of simpler terms, e.g.,
Why we may favor a PGM?Incorporation of domain knowledge and causal (logical) structures
P(X1, X2, X3, X4, X5, X6, X7, X8)
= P(X1) P(X2) P(X3| X1) P(X4| X2) P(X5| X2)P(X6| X3, X4) P(X7| X6) P(X8| X5, X6)
Probabilistic Graphical Models
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
X1 X2
X3 X4 X5
X6
X7 X8
2+2+4+4+4+8+4+8=36, an 8-fold reduction from 28 in representation cost !
Stay tune for what are these independencies!
Eric Xing 18
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BXX11 XX22
XX33 XX44 XX55
XX66
XX77 XX88
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BXX11 XX22
XX33 XX44 XX55
XX66
XX77 XX88
GM: Data Integration
10
Eric Xing 19
If Xi's are conditionally independent (as described by a PGM), the joint can be factored to a product of simpler terms, e.g.,
Why we may favor a PGM?Incorporation of domain knowledge and causal (logical) structures
Modular combination of heterogeneous parts – data fusion
Probabilistic Graphical Models
2+2+4+4+4+8+4+8=36, an 8-fold reduction from 28 in representation cost !
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BXX11 XX22
XX33 XX44 XX55
XX66
XX77 XX88
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BXX11 XX22
XX33 XX44 XX55
XX66
XX77 XX88
XX11 XX22
XX33 XX44 XX55
XX66
XX77 XX88
P(X1, X2, X3, X4, X5, X6, X7, X8)
= P(X2) P(X4| X2) P(X5| X2) P(X1) P(X3| X1) P(X6| X3, X4) P(X7| X6) P(X8| X5, X6)
Eric Xing 20
∑∈′
′′=
Hhhphdp
hphdpdhp)()|(
)()|()|(
Posteriorprobability
Likelihood Priorprobability
Sum over space of hypotheses
Rational Statistical Inference
hh
dd
The Bayes Theorem:
This allows us to capture uncertainty about the model in a principled way
But how can we specify and represent a complicated model?Typically the number of genes need to be modeled are in the order of thousands!
11
Eric Xing 21
GM: MLE and Bayesian Learning
Probabilistic statements of Θ is conditioned on the values of the observed variables Aobs and prior p( |χ)
(A,B,C,D,E,…)=(T,F,F,T,F,…)A= (A,B,C,D,E,…)=(T,F,T,T,F,…)
……..(A,B,C,D,E,…)=(F,T,T,T,F,…)
A
C
F
G H
ED
BA
C
F
G H
ED
B A
C
F
G H
ED
BA
C
F
G H
ED
BA
C
F
G H
ED
B
0.9 0.1
c
dc
0.2 0.8
0.01 0.99
0.9 0.1
dcdd
c
DC P(F | C,D)0.9 0.1
c
dc
0.2 0.8
0.01 0.99
0.9 0.1
dcdd
c
DC P(F | C,D)
p(Θ; χ)
);()|();|( χχ ΘΘΘ ppp AA ∝
posterior likelihood prior
ΘΘΘΘ dpBayes ∫= ),|( χA
Eric Xing 22
If Xi's are conditionally independent (as described by a PGM), the joint can be factored to a product of simpler terms, e.g.,
Why we may favor a PGM?Incorporation of domain knowledge and causal (logical) structures
Modular combination of heterogeneous parts – data fusion
Bayesian PhilosophyKnowledge meets data
Probabilistic Graphical Models
2+2+4+4+4+8+4+8=36, an 8-fold reduction from 28 in representation cost !
θ α θ⇒⇒
P(X1, X2, X3, X4, X5, X6, X7, X8)
= P(X1) P(X2) P(X3| X1) P(X4| X2) P(X5| X2)P(X6| X3, X4) P(X7| X6) P(X8| X5, X6)
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
X1 X2
X3 X4 X5
X6
X7 X8
12
Eric Xing 23
Directed edges give causality relationships (Bayesian Network or Directed Graphical Model):
Undirected edges simply give correlations between variables (Markov Random Field or Undirected Graphical model):
Two types of GMs
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
X1 X2
X3 X4 X5
X6
X7 X8
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
Receptor A
Kinase C
TF F
Gene G Gene H
Kinase EKinase D
Receptor BX1 X2
X3 X4 X5
X6
X7 X8
X1 X2
X3 X4 X5
X6
X7 X8
P(X1, X2, X3, X4, X5, X6, X7, X8)
= P(X1) P(X2) P(X3| X1) P(X4| X2) P(X5| X2)P(X6| X3, X4) P(X7| X6) P(X8| X5, X6)
P(X1, X2, X3, X4, X5, X6, X7, X8)
= 1/Z exp{E(X1)+E(X2)+E(X3, X1)+E(X4, X2)+E(X5, X2)+ E(X6, X3, X4)+E(X7, X6)+E(X8, X5, X6)}
Eric Xing 24
Structure: DAG
• Meaning: a node is conditionally independentof every other node in the network outside its Markov blanket
• Local conditional distributions (CPD) and the DAGcompletely determine the joint dist.
• Give causality relationships, and facilitate a generativeprocess
X
Y1 Y2
Descendent
Ancestor
Parent
Children's co-parentChildren's co-parent
Child
Bayesian Networks
13
Eric Xing 25
Structure: undirected graph
• Meaning: a node is conditionally independent of every other node in the network given its Directed neighbors
• Local contingency functions (potentials) and the cliques in the graph completely determine the joint dist.
• Give correlations between variables, but no explicit way to generate samples
X
Y1 Y2
Markov Random Fields
Eric Xing 26
(Picture by Zoubin Ghahramani and Sam Roweis)
An (incomplete)
genealogy of graphical
models
14
Eric Xing 27
Computing statistical queries regarding the network, e.g.:Is node X independent on node Y given nodes Z,W ?What is the probability of X=true if (Y=false and Z=true)?What is the joint distribution of (X,Y) if Z=false?What is the likelihood of some full assignment?What is the most likely assignment of values to all or a subset the nodes of the network?
General purpose algorithms exist to fully automate such computation Computational cost depends on the topology of the networkExact inference:
The junction tree algorithmApproximate inference;
Loopy belief propagation, variational inference, Monte Carlo sampling
Probabilistic InferenceA
C
F
G H
ED
BA
C
F
G H
ED
BA
C
F
G H
ED
B
Eric Xing 28
They require a localist semantics for the nodes
They require a causal semantics for the edges
They are necessarily Bayesian
They are intractable
A few myths about graphical models
√√
××
××
√√
15
Eric Xing 29
Application of GMsMachine LearningComputational statistics
Computer vision and graphicsNatural language processing Informational retrievalRobotic control Decision making under uncertaintyError-control codesComputational biologyGenetics and medical diagnosis/prognosisFinance and economicsEtc.
Eric Xing 30
Speech recognition
A AA AX2 X3X1 XT
Y2 Y3Y1 YT...
...
Hidden Markov ModelHidden Markov Model
16
Eric Xing 31
A A A AAA A A A A A A A A A A AC G T AGA A A A G A G T C A A T
X
Y
θ1 θ2 θ3 θ4 θ5 θ6 θ7 θ8 θ1
Segmentation and Pattern Recog. ( in Bio, Vision, NLP)
b b c m1cb m1 m2 m3 m4 m5 m6 m7 m8 c c c
Eric Xing 32
Reinforcement learningPartially observed Markov decision processes (POMDP)
17
Eric Xing 33
Machine translation
SMT
The HMThe HM--BiTAMBiTAM model model (B. Zhao and E.P Xing, (B. Zhao and E.P Xing, ACL 2006)ACL 2006)
Eric Xing 34
Genetic pedigree
A0
A1
AgB0
B1
Bg
M0
M1
F0
F1
Fg
C0
C1
Cg
Sg
An allele networkAn allele network
18
Eric Xing 35
Evolution
ancestor
A C
QhQm
T years
?
AGAGAC
Tree ModelTree Model
Eric Xing 36
Solid State Physics
IsingIsing/Potts model/Potts model
19
Eric Xing 37
Computer Vision
Eric Xing 38
A Generative GM
P(I|Y)
Image Observation
P(Y|X; Ө)
Transformation
P(X|b)
Deformation
I:image observation
b: deformation parameter
: pose parameterθ
X: canonical shape
Y: transformed shape
UnimodalMultimodalExample-based
RigidPerspective
Boundary/RegionalSearching Path/Region
Y
Xθ
bb
θ
Y
X
I
20
Eric Xing 39
A Generative GM
P(I|Y)
Image Observation
P(Y|X; Ө)
Transformation
P(X|b)
Deformation
I:image observation
b: deformation parameter
: pose parameterθ
X: canonical shape
Y: transformed shape
Y
Xθ
bb
θ
Y
X
I
(Gu, Xing, & Kanade, CVPR07)
(Gu, & Kanade, CVPR07)
Eric Xing 40
Why graphical models
A language for communicationA language for computationA language for development
Origins: Wright 1920’sIndependently developed by Spiegelhalter and Lauritzen in statistics and Pearl in computer science in the late 1980’s
21
Eric Xing 41
Probability theory provides the glue whereby the parts are combined, ensuring that the system as a whole is consistent, and providing ways to interface models to data.
The graph theoretic side of graphical models provides both an intuitively appealing interface by which humans can model highly-interacting sets of variables as well as a data structure that lends itself naturally to the design of efficient general-purpose algorithms.
Many of the classical multivariate probabilistic systems studied in fields such as statistics, systems engineering, information theory, pattern recognition and statistical mechanics are special cases of the general graphical model formalism
The graphical model framework provides a way to view all of these systems as instances of a common underlying formalism.
--- M. Jordan
Why graphical models
Eric Xing 42
Plan for the ClassFundamentals of Graphical Models:
Bayesian Network and Markov Random FieldsContinuous and Hybrid models, exponential family, GLIMBasic representation, inference, and learning
Case studies: Popular Bayesian networks and MRFMultivariate Gaussian ModelsTemporal modelsTrees modelsIntractable popular BNs and MRFs: e.g., Dynamic Bayesian networks, Bayesian admixture models (LDA)
Approximate inferenceMonte Carlo algorithmsVatiational methods
Advanced topicsLearning in structured input-output spaceNonparametric Bayesian model
Applications
22
Eric Xing 43
NotationVariable, value and index
Random variable
Random vector
Random matrix
Parameters