NIST Foundations of Ontological Analysis Chris Welty, Vassar College.
Lecture 6: Watson and the Social Web (2014), Chris Welty
-
Upload
lora-aroyo -
Category
Technology
-
view
559 -
download
3
description
Transcript of Lecture 6: Watson and the Social Web (2014), Chris Welty
© 2011 IBM Corporation
Watson and the Social Web
Chris Welty IBM Watson Group ibmwatson.com
Do Not Record. Do Not Distribute.
© 2011 IBM Corporation
What is Cognitive Computing?
§ Increasingly, machines are being asked to add their computational power to problems which are not inherently solvable
§ Traditionally, these problems came from AI – The hardest AI problems are the easiest for human intelligence:
vision, speech, natural language – these are not actually associated with “being intelligent”
– Human intelligence provides solutions, but does not scale § Cognitive Computing is founded on four principles
Learn & improve. Cognitive computing systems focus on inexact solutions to unsolvable problems that utilize machine learning and improve over time. Often they combine multiple approaches and must integrate them effectively. They must learn from humans, in more and more seamless ways.
Speed&Scale. Cognitive computing harnesses the clear advantage machines have over humans in their ability to perform mundane tasks of arbitrary complexity repeatedly, whether it is the scale of the data or the complexity of the task.
Interact in a natural way. Cognitive computing provides technologies that support a higher level of human cognition by adapting to human approaches and interfaces...over the next several decades it will incorporate essentially all the ways humans sense and interact.
Assist & augment human cognition. Cognitive computing addresses problems that lie squarely in the province of human intelligence, but where we can't handle the volume of information, penetrate the complexity or otherwise extend our reach (physically). The goal is to be useful, not universally correct.
or Computers can be incorrect and still prove useful!
© 2011 IBM Corporation
Examples of Cognitive Computing
§ Web Search
§ Image Search
§ Event Search
§ Recommendations
§ Natural Language Processing
© 2011 IBM Corporation
What is Watson?
§ Open Domain Question-Answering Machine § Given
– Rich Natural Language Questions – Over a Broad Domain of Knowledge
§ Delivers – Precise Answers: Determine what is being asked & give precise response – Accurate Confidences: Determine likelihood answer is correct – Consumable Justifications: Explain why the answer is right – Fast Response Time: Precision & Confidence in <3 seconds – At the level of human experts
– Proved its mettle in a televised match – Won a 2-game Jeopardy match against
the all-time winners – viewed by over 50,000,000
4
© 2011 IBM Corporation
What is Jeopardy?
§ Jeopardy! is an American quiz show
– 1964 – Today – Household name in U.S.
§ answer-and-question format – contestants are presented with
clues in the form of answers – must phrase their responses in
question form. – Open domain trivia questions,
speed is a big factor § Example
– Category: General Science – Clue: When hit by electrons, a
phosphor gives off electromagnetic energy in this form
– Answer: What is light?
© 2011 IBM Corporation
Social Computing: What’s the connection?
§ Social Web as Data Source: – The vast majority of sources Watson
used to answer questions came from community-created data
– Adapting Watson to a new problem requires the same kind of information about that problem
§ Social Machines: – Watson combined with people is a
powerful proposition
§ Social Web as Application: – Watson’s major advance is in
understanding natural language, the technology can be useful to augment social interaction
© 2011 IBM Corporation
$200 If you are looking at the wainscoating, you are looking in
this direction.
$1000 The first person
mentioned by name in ‘The Man in the Iron
Mask’ is this hero of a previous book by the
same author.
7
The Jeopardy! Challenge Hard for humans, hard for machines
Broad/Open Domain
Complex Language
High Precision
Accurate Confidence
High Speed
$600 In cell division, mitosis
splits the nucleus & cytokinesis splits this liquid cushioning the
nucleus
$800 The conspirators against
this man were wounded by each other while they
stabbed at him
But hard for different reasons.
For people, the challenge is knowing the answer For machines, the challenge is understanding the question
What is down? Who is
D’Artagnan?
What is cytoplasm?
Who is Julius Caesar?
© 2011 IBM Corporation
The Winner’s Cloud What It Takes to compete against Top Human Jeopardy! Players
Winning Human Performance
2007 QA Computer System
Grand Champion Human Performance
Top human players are remarkably
good.
Each dot – actual historical human Jeopardy! games
More Confident Less Confident
Develop against a metric!
© 2011 IBM Corporation
2007 QA Computer System
In 2007, we committed to making a Huge Leap!
More Confident Less Confident
Each dot – actual historical human Jeopardy! games
Computers? Not So Good.
Winning Human Performance
Grand Champion Human Performance
The Winner’s Cloud What It Takes to compete against Top Human Jeopardy! Players
© 2011 IBM Corporation
DeepQA: The Technology Behind Watson An example of a new software paradigm
. . .
Answer Scoring
Models
Answer & Confidence
Question
Evidence Sources
Models
Models
Models
Models
Models Primary Search
Candidate Answer
Generation
Hypothesis Generation
Hypothesis and Evidence Scoring
Final Confidence Merging & Ranking
Synthesis
Answer Sources
Question & Topic
Analysis
Question Decomposition
Evidence Retrieval
Deep Evidence Scoring
Hypothesis Generation
Hypothesis and Evidence Scoring
Learned Models help combine and
weigh the Evidence
DeepQA generates and scores many hypotheses using an extensible collection of Natural Language Processing, Machine Learning and Reasoning Algorithms. These gather and weigh evidence over both unstructured and structured content to
determine the answer with the best confidence. Content from Community Resources!
© 2011 IBM Corporation
Example Question
In 1894 C.W. Post created his warm cereal drink Postum in this Michigan city
Related Content (Structured & Unstructured)
Primary Search
1985
Post Foods
aramour
General Foods
Grand Rapids
…
Battle Creek
…
…
Candidate Answer Generation
1) Battle Creek (0.85) 2) Post Foods ( 0.20) 3) 1985 (0.05)
Merging & Ranking
Evidence Retrieval
Question Analysis
Keywords: 1894, C.W. Post, created … Lexical AnswerType: (Michingan city) Date(1894) Relations: Create(Post, cereal drink) …
[0.58 0 -1.3 … 0.97]
[0.71 1 13.4 … 0.72]
[0.12 0 2.0 … 0.40]
[0.84 1 10.6 … 0.21]
[0.33 0 6.3 … 0.83]
[0.21 1 11.1 … 0.92]
[0.91 0 -8.2 … 0.61]
[0.91 0 -1.7 … 0.60]
Evidence Scoring
Need thousands of Q/A pairs for training!
© 2011 IBM Corporation
Planet Fitness
Role of Answer Typing in QA
Type Information - a crucial hint to get the correct answer
ASTRONOMY: In 1610 Galileo named the moons of this planet for the Medici brothers
Telescope
Giovanni Medici
Sidereus Nuncius
Jupiter
Ganymede Telescope (Instrument)
Giovanni Medici (Person)
Sidereus Nuncius (Book)
Jupiter (Planet)
Ganymede (Moon)
Terms Associated with Clue Context (e.g. via Keyword Search)
Planet Fitness (Planet)
© 2011 IBM Corporation
§ This fish was thought to be exLnct millions of years ago unLl one was found off South Africa in 1938
§ Category: ENDS IN "TH" § Answer:
§ When hit by electrons, a phosphor gives off electromagneLc energy in this form
§ Category: General Science § Answer:
§ Secy. Chase just submiXed this to me for the third Lme-‐-‐guess what, pal. This Lme I'm accepLng it
§ Category: Lincoln Blogs § Answer:
The type of thing being asked for is often indicated but
can go from specific to very vague
coelacanth
light (or photons)
his resigna4on
13
Answer Typing for Jeopardy!?
© 2011 IBM Corporation
Broad Domain
Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text. Structured sources (DBs and KBs) provide background knowledge for interpreting the text.
We do NOT attempt to anticipate all questions and build databases.
We do NOT try to build a formal model of the world
© 2011 IBM Corporation
Sources for typing evidence
§ DbPedia & Freebase – Wide coverage of well-known entities – Taxonomy (MountainsOfNepal → Mountain) – Good type coverage, but not many synonyms
• E.g. what about “summit”
§ Wikpedia Categories – Wide coverage of entities and type name synonyms – Noisy (many errors)
§ Wikipedia Intro – First sentence always indicates the most common type of the entity – Highly reliable, low coverage of types
Communities can scale data collection!
© 2011 IBM Corporation
Typing Impact on Jeopardy! clues
61.5% 62.0% 62.5% 63.0% 63.5% 64.0% 64.5% 65.0% 65.5% 66.0% 66.5%
An ensemble of TyCor components
+ ~10%
© 2011 IBM Corporation
Many sources of evidence
In 1894 C.W. Post created his warm cereal drink Postum in this Michigan city
Related Content (Structured & Unstructured)
Primary Search
1985
Post Foods
aramour
General Foods
Grand Rapids
…
Battle Creek
…
…
Candidate Answer Generation
1) Battle Creek (0.85) 2) Post Foods ( 0.20) 3) 1985 (0.05)
Merging & Ranking
Evidence Retrieval
Question Analysis
Keywords: 1894, C.W. Post, created … Lexical AnswerType: (Michingan city) Date(1894) Relations: Create(Post, cereal drink) …
[0.58 0 -1.3 … 0.97]
[0.71 1 13.4 … 0.72]
[0.12 0 2.0 … 0.40]
[0.84 1 10.6 … 0.21]
[0.33 0 6.3 … 0.83]
[0.21 1 11.1 … 0.92]
[0.91 0 -8.2 … 0.61]
[0.91 0 -1.7 … 0.60]
Evidence Scoring
© 2011 IBM Corporation
Watson as part of a social machine
§ Watson makes mistakes: – This woman was the first to witness her husband resign from the U.S. Presidency.
– This U.S. City’s largest airport is named for a world-war II hero; its second largest for a world-war II battle.
§ These mistakes are typically obvious to people – Even when they don’t know the answer – Watson isn’t stupid, it solves problems differently – Often these multiple perspectives can combine productively
• E.g. add a “dismiss” button to the answer interface
Richard Nixon Dolly Madison
Pat Nixon
Watson can adapt and learn from its users!
© 2011 IBM Corporation
Cut to the chase….. Watson emerges victorious
© 2011 IBM Corporation
Technology marches forward…
© 2011 IBM Corporation
Adapt Watson
Models
Answer & Confidence
Question
Evidence Sources
Models
Models
Models
Models
Models
Answer Sources
. . .
Answer Scoring
Primary Search
Candidate Answer
Generation
Hypothesis Generation
Hypothesis and Evidence Scoring
Final Confidence Merging & Ranking
Synthesis Question &
Topic Analysis
Question Decomposition
Evidence Retrieval
Deep Evidence Scoring
Hypothesis Generation
Hypothesis and Evidence Scoring
Learned Models help combine and
weigh the Evidence
What does it take to use Watson in a new domain? (medical diagnosis, call centers, etc...)
Gathering significant numbers of question-answer pairs is proving to be one of the most significant challenges for adapting Watson. Can the social web help?
Community created!
© 2011 IBM Corporation
Integrating Watson in Social Interaction?
Did you hear about Bob?
No
He’s taking a year off to climb the tallest mountain!
The tallest mountain is Mount Everest.
Wow.
me
me
Jeff
Watson
Jeff
© 2011 IBM Corporation
Privacy – a blessing and a curse
Need to protect our data, but… Crime on the web, the social web, is very real
Identity theft Credit card, bank, insurance fraud Terrorist networks
Medical diagnosis
Monitoring your profile for health-related information ICT for depression
Calendar, appointments, traffic, spreading disease
© 2011 IBM Corporation
The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs).
Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range.
Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse…
Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities.
© 2011 IBM Corporation
The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs).
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% % Answered
© 2011 IBM Corporation
The arrival of Cognitive Computing
Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities.
UTI
Diabetes
Influenza
hypokalemia
Renal failure
esophogitis
Diagnosis Models Confidence
Most Confident Diagnosis: UTI
Symptoms
Tests/Findings Medica4ons
Family History
Notes/Hypotheses
Huge Volumes of Texts, Journals, References, DBs etc.
Pa4ent History
© 2011 IBM Corporation
The arrival of Cognitive Computing
Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range.
© 2011 IBM Corporation
The arrival of Cognitive Computing
Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse…
© 2011 IBM Corporation
The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs).
Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range.
Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse…
Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities.
© 2011 IBM Corporation
…and for Social Web
§ First and foremost, social web analytics (e.g. recommendations) and Social Computing in general lie clearly in the realm of Cognitive Computing
– Uncertainty, natural language, human intelligence – Inexact solutions that can improve with time, training – Problems & solutions need metrics to be solvable
§ All cognitive computing systems require ground truth data – This data is expensive to collect – Crowdsourcing is a key new technology/approach
§ The user interface moving closer to people – Natural language, speech, gestures – In addition, integrating the collection of training data seamlessly into the interface
is a key development § Cognitive computing systems require integration of multiple, disparate, data
sources – Structured, unstructured, semi-structured – curated, crowdsourced