Mike Barnett RSDE Microsoft Research Nikolai Tillmann RSDE Microsoft Research TL51.
Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft...
-
date post
21-Dec-2015 -
Category
Documents
-
view
212 -
download
0
Transcript of Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft...
![Page 1: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/1.jpg)
Third Generation Machine
IntelligenceChristopher M. BishopMicrosoft Research, Cambridge
Microsoft Research Summer School 2009
![Page 2: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/2.jpg)
First Generation
“Artificial Intelligence” (GOFAI)
Within a generation ... the problem of creating ‘artificial intelligence’ will largely be solved
Marvin Minsky (1967)
Expert Systems– rules devised by humans
Combinatorial explosion
General theme: hand-crafted rules
![Page 3: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/3.jpg)
Second Generation
Neural networks, support vector machines
Difficult to incorporate complex domain knowledge
General theme: black-box statistical models
![Page 4: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/4.jpg)
Third Generation
General theme: deep integration of domainknowledge and statistical learning
Probabilistic graphical models– Bayesian framework– fast inference using local message-passing
Origins: Bayesian networks, decision theory, HMMs, Kalman filters, MRFs, mean field theory, ...
![Page 5: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/5.jpg)
Bayesian Learning
Consistent use of probability to quantify uncertainty
Predictions involve marginalisation, e.g.
![Page 6: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/6.jpg)
Why is prior knowledge important?
y
x
?
![Page 7: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/7.jpg)
Probabilistic Graphical Models
1. New insights into existing models
2. Framework for designing new models
3. Graph-based algorithms for calculation and computation (c.f. Feynman diagrams in physics)
4. Efficient software implementation
Directed graphs to specify the model
Factor graphs for inference and learning
Probability theory + graphs
![Page 8: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/8.jpg)
Directed Graphs
![Page 9: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/9.jpg)
Example: Time Series Modelling
![Page 10: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/10.jpg)
![Page 11: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/11.jpg)
Manchester Asthma and Allergies Study
Chris BishopIain BuchanMarkus SvensénVincent TanJohn Winn
![Page 12: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/12.jpg)
![Page 13: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/13.jpg)
Factor Graphs
![Page 14: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/14.jpg)
From Directed Graph to Factor Graph
![Page 15: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/15.jpg)
Local message-passing
Efficient inference by exploiting factorization:
![Page 16: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/16.jpg)
Factor Trees: Separation
v w x
f1(v,w) f2(w,x)
y
f3(x,y)
z
f4(x,z)
![Page 17: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/17.jpg)
Messages: From Factors To Variables
w x
f2(w,x)
y
f3(x,y)
z
f4(x,z)
![Page 18: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/18.jpg)
Messages: From Variables To Factors
x
f2(w,x)
y
f3(x,y)
z
f4(x,z)
![Page 19: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/19.jpg)
What if marginalisations are not tractable?
True distribution Monte Carlo Variational Bayes
Loopy belief propagation
Expectation propagation
![Page 20: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/20.jpg)
Illustration: Bayesian Ranking
Ralf HerbrichTom MinkaThore Graepel
![Page 21: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/21.jpg)
Two Player Match Outcome Model
y1
2
1 2
s1 s2
![Page 22: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/22.jpg)
Two Team Match Outcome Model
y1
2
t1
t2
s2
s3
s1
s4
![Page 23: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/23.jpg)
Multiple Team Match Outcome Model
s1
s2
s3
s4
t1
y1
2
t2
t3
y2
3
![Page 24: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/24.jpg)
Efficient Approximate Inference
s1
s2
s3
s4
t1
y1
2
t2
t3
y2
3
Gaussian Prior Factors
Ranking Likelihood Factors
![Page 25: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/25.jpg)
Convergence
0
5
10
15
20
25
30
35
40L
ev
el
0 100 200 300 400
Number of Games
char (Elo)
SQLWildman (Elo)
char (TrueSkill™)
SQLWildman (TrueSkill™)
![Page 26: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/26.jpg)
TrueSkillTM
![Page 27: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/27.jpg)
John WinnChris Bishop
![Page 28: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/28.jpg)
research.microsoft.com/infernet
Tom MinkaJohn WinnJohn GuiverAnitha Kannan
![Page 29: Third Generation Machine Intelligence Christopher M. Bishop Microsoft Research, Cambridge Microsoft Research Summer School 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032521/56649d5f5503460f94a3f1c8/html5/thumbnails/29.jpg)
Summary
New paradigm for machine intelligence built on:– a Bayesian formulation– probabilistic graphical models– fast inference using local message-passing
Deep integration of domain knowledge and statistical learning
Large-scale application: TrueSkillTM
Toolkit: Infer.NET