An Introduction to Markov Logic Networks in Knowledge Bases
-
Upload
elaine-lowery -
Category
Documents
-
view
17 -
download
1
description
Transcript of An Introduction to Markov Logic Networks in Knowledge Bases
From Big Data to Big Wisdom
From Big Data to Big Wisdom
Data Facts Rules
What is a Knowledge Base?
What is a Knowledge Base?
What is a Knowledge Base?
What is a Knowledge Base?
What is a Knowledge Base?
What is a Knowledge Base?
Graph Representation
Triple Representation
• FatherOf(Darth Vader, Luke Skywalker)• memberOf(Luke Skywalker, Rebel Alliance)• hasWeapon(Luke Skywalker, lightsaber)• isAlliedWith(Luke Skywalker, Han Solo)• IsAlliedWith(Luke Skywalker, Chewbacca)• livesOn(Luke Skywalker, Tatooine)• hasAdviser(Luke Skywalker, Yoda)• …..
Large Scale KBs
• KnowitAll (Univ. of Washington)• 500M facts extracted from 100M webpages (ClueWeb)
• DBPedia (Univ. of Leipzig)• 3.5M entities, 700M facts extracted from Wikipedia
• YAGO (Max-Planck Institute)• 2M entities, 20M facts extracted from Wikipedia and WordNet
• Freebase (Google)• 20M entities, 300M facts from integrated data sources and humans
• NELL (Carnegie Mellon)• 850M facts extracted from 500M webpages
Explicit vs. Implicit Information
Does Obama live in the US?
Knowledge Expansion
Inferring New Facts
• Markov Network• Undirected graphical model• Markov Property: Probabilities for each node depend only on its neighbors• Most probable assignment of random variables defined by cliques• HARD to figure out all the clique potentials
• First Order Logic• Data described by predicates with arguments• Logical formulas denote inference rules over predicates• Deterministic, ie. no probability• EASY to specify logical formulas
• Markov Logic: Markov Networks + First Order Logic• Statistical power of MLNs + specification power of FOL
Markov Networks
Cancer
CoughAsthma
Smoking
Smoking Cancer Ф(S,C)
False False 4.5
False True 4.5
True False 2.7
True True 4.5
c
cc xZxP )(
1)(
x c
cc xZ )(
Weight of Feature i Feature i
Markov Networks
Cancer
CoughAsthma
Smoking
iii xfw
ZxP )(exp
1)(
otherwise0
CancerSmokingif1)CancerSmoking,(1f
5.41 w
First Order Logic
Fact A Fact B
Fact A Fact B Fact C^
First Order Logic
isA(Luke, Jedi) memberOf(Luke, Rebel Alliance)
First Order Logic
isAlliedWith(Han, Luke)
memberOf(Luke, Rebel Alliance)^
memberOf(Han, Rebel Alliance)
First Order Logic
isAlliedWith(X, Y)
memberOf(Y, Z)^
memberOf(X, Z)
Soft Rules vs. Hard Rules
bornIn(X, Y) locatedIn(Y, Z)^ bornIn(X, Z)
Soft Rules vs. Hard Rules
bornIn(X, Y) locatedIn(Y, Z)^ bornIn(X, Z)HARD
Soft Rules vs. Hard Rules
bornIn(X, Y) locatedIn(Y, Z)^ bornIn(X, Z)HARD
friendOf(X, Y) friendOf(Y, Z)^ friendOf(X, Z)
Soft Rules vs. Hard Rules
bornIn(X, Y) locatedIn(Y, Z)^ bornIn(X, Z)HARD
friendOf(X, Y) friendOf(Y, Z)^ friendOf(X, Z)SOF
T
Soft Rules vs. Hard Rules
bornIn(X, Y) locatedIn(Y, Z)^ bornIn(X, Z)HARD
friendOf(X, Y) friendOf(Y, Z)^ friendOf(X, Z)SOF
T
MLN Basic Idea: Give every formula a weight (higher weight stronger constraint)
Markov Logic Networks: Logic + ProbabilityA Markov Logic Network is a collection of pairs (F, W)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Building a Markov Network
• Nodes = all possible instantiations of a predicate• Edge between two nodes iff they appear together in some formula
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
c
cc xZxP )(
1)(
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
satisfiesit formulas of weightsexpP(world)
c
cc xZxP )(
1)(
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
Knowledge Base:
memberOf(Luke, Rebels)
memberOf(Han, Rebels)
hasLightSaber(Luke)
~memberOf(Luke, Empire)
Probability of Possible World
c
cc xZxP )(
1)(
Probability of Possible World
)exp()( j
jjcc fwx
c
cc xZxP )(
1)(
otherwise0
X)(Y,alliedWithY)(X,alliedWithif1))X(Y,alliedWithY),(X,alliedWith(1f
1001 w
iii xnw
ZxP )(exp
1)(
Probability of Possible World
Weight of formula i No. of true groundings of formula i in x
c
cc xZxP )(
1)(
)exp()( j
jjcc fwx
otherwise0
X)(Y,alliedWithY)(X,alliedWithif1))X(Y,alliedWithY),(X,alliedWith(1f
1001 w
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
iii xnw
ZxP )(exp
1)(
Inference in MLNs
• MAP Inference: Find the most likely possible world given some evidence
Inference in MLNs
• MAP Inference: Find the most likely possible world given some evidence
i
iiy
yxnw ),(maxarg
Inference in MLNs
• MAP Inference: Find the most likely possible world given some evidence
Solved with MaxWalkSAT
i
iiy
yxnw ),(maxarg
Inference in MLNs
• MAP Inference: Find the most likely possible world given some evidence
Solved with MaxWalkSAT
• Marginal Inference: Find the marginal distribution of a predicate (node) given all the others
i
iiy
yxnw ),(maxarg
MaxWalkSAT
for i ← 1 to max-tries do solution = random truth assignment for j ← 1 to max-flips do if ∑ weights(sat. clauses) > threshold then return solution c ← random unsatisfied clause with probability p flip a random variable in c else flip variable in c that maximizes ∑ weights(sat. clauses) return failure, best solution found
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
Knowledge Base:
memberOf(Luke, Rebels)
memberOf(Han, Rebels)
hasLightSaber(Luke)
~memberOf(Luke, Empire)
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
Knowledge Base:
memberOf(Luke, Rebels)
memberOf(Han, Rebels)
hasLightSaber(Luke)
~memberOf(Luke, Empire)
1
1
0
1Fill in Observed Data
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
Knowledge Base:
memberOf(Luke, Rebels)
memberOf(Han, Rebels)
hasLightSaber(Luke)
~memberOf(Luke, Empire)
1
1
0
1?
?
?
?
Find Missing Data
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
Knowledge Base:
memberOf(Luke, Rebels)
memberOf(Han, Rebels)
hasLightSaber(Luke)
~memberOf(Luke, Empire)
1
1
0
10
0
0
0
Initialization Assignment
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
Knowledge Base:
memberOf(Luke, Rebels)
memberOf(Han, Rebels)
hasLightSaber(Luke)
~memberOf(Luke, Empire)
1
1
0
10
0
0
0
Flip One At Random
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
Knowledge Base:
memberOf(Luke, Rebels)
memberOf(Han, Rebels)
hasLightSaber(Luke)
~memberOf(Luke, Empire)
1
1
0
10
0
0
1
Flip One At Random
memberOf(H, R)memberOf(L, R)
memberOf(H, E)
memberOf(L, E)
alliedWith(H, L)alliedWith(L, H)
hasLightSaber(H)hasLightSaber(L)
memberOf(X, Z) ^ memberOf(Y, Z) -> alliedWith(X, Y) 8.5
hasLightsaber(X) -> memberOf(X, Rebels) 0.5
alliedWith(X, Y) -> alliedWith(Y, X) 100
Han (H)Luke (L)Rebels (R)Empire (E)
Knowledge Base:
memberOf(Luke, Rebels)
memberOf(Han, Rebels)
hasLightSaber(Luke)
~memberOf(Luke, Empire)
1
1
0
11
1
0
1
Flip One At Random
Learning in MLNs
• Data is a relational database or knowledge base
Knowledge Base:
memberOf(Luke, Rebels) hasLightSaber(Luke)
memberOf(Han, Rebels) ~memberOf(Luke, Empire)
Learning in MLNs
• Data is a relational database or knowledge base
• Learning parameters (weights)• Similar to learning weights for Markov networks
Knowledge Base:
memberOf(Luke, Rebels) hasLightSaber(Luke)
memberOf(Han, Rebels) ~memberOf(Luke, Empire)
Learning in MLNs
• Data is a relational database or knowledge base
• Learning parameters (weights)• Similar to learning weights for Markov networks
• Learning structure (formulas)• Inductive logic programming
Knowledge Base:
memberOf(Luke, Rebels) hasLightSaber(Luke)
memberOf(Han, Rebels) ~memberOf(Luke, Empire)
No. of times clause i is true in data
Expected no. times clause i is true according to MLN
Learning Weights via Max-Likelihood
)()()(log xnExnxPw iwiwi
iii xnw
ZxP )(exp
1)(
Alchemy (Open Source Software)
• Full first-order logic syntax• Inference (MAP and marginal probabilities)• Lifted Inference• Weight Learning (generative and discriminative)• Structure Learning• Large set of tutorials for learning
• Information extraction, social network modeling, entity resolution
alchemy.cs.washington.edu
Current Problem: Scale to Large KBs!• KnowitAll (Univ. of Washington)
• 500M facts extracted from 100M webpages (ClueWeb)
• DBPedia (Univ. of Leipzig)• 3.5M entities, 700M facts extracted from Wikipedia
• YAGO (Max-Planck Institute)• 2M entities, 20M facts extracted from Wikipedia and WordNet
• Freebase (Google)• 20M entities, 300M facts from integrated data sources and humans
• NELL (Carnegie Mellon)• 850M facts extracted from 500M webpages