1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic...
-
date post
20-Dec-2015 -
Category
Documents
-
view
224 -
download
0
Transcript of 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic...
![Page 1: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/1.jpg)
1
Natural Language Processing
Information RetrievalSpeech Recognition
Syntactic ParsingSemantic Interpretation
CSE 592 Applications of AIWinter 2003
![Page 2: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/2.jpg)
2
Example Applications
• Spelling and grammar checkers
• Finding information on the WWW
• Spoken language control systems: banking, shopping
• Classification systems for messages, articles
• Machine translation tools
![Page 3: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/3.jpg)
3
The Dream
![Page 4: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/4.jpg)
4
Information Retrieval
(Thanks to Adam Carlson)
![Page 5: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/5.jpg)
5
Motivation and Outline
• Background– Definitions
• The Problem– 100,000+ pages
• The Solution– Ranking docs– Vector space– Probabilistic
approaches
• Extensions– Relevance feedback, clustering, query expansion, etc.
![Page 6: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/6.jpg)
6
What is Information Retrieval
• Given a large repository of documents, how do I get at the ones that I want– Examples: Lexus/Nexus, Medical reports,
AltaVista
• Different from databases– Unstructured (or semi-structured) data– Information is (typically) text– Requests are (typically) word-based
![Page 7: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/7.jpg)
7
Information Retrieval Task
• Start with a set of documents• User specifies information need
– Keyword query, Boolean expression, high-level description
• System returns a list of documents– Ordered according to relevance
• Known as the ad-hoc retrieval problem
![Page 8: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/8.jpg)
8
Measuring Performance
• Precision– Proportion of selected
items that are correct
• Recall– Proportion of target
items that were selected
• Precision-Recall curve– Shows tradeoff
tn
fp tp fn
System returned these
Actual relevant docs
fptp
tp
fntp
tp
Precision
Recall
![Page 9: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/9.jpg)
9
Basic IR System
• Use word overlap to determine relevance– Word overlap alone is inaccurate
• Rank documents by similarity to query
• Computed using Vector Space Model
![Page 10: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/10.jpg)
10
Vector Space Model
• Represent documents as a matrix– Words are rows– Documents are columns– Cell i,j contains the number of times word i
appears in document j– Similarity between two documents is the
cosine of the angle between the vectors representing those words
![Page 11: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/11.jpg)
11
Vector Space Example
a: System and human system engineering testing of EPS
b: A survey of user opinion of computer system response time
c: The EPS user interface management system d: Human machine interface for ABC computer
applications e: Relation of user perceived response time to
error measurement f: The generation of random, binary, ordered
trees g: The intersection graph of paths in trees h: Graph minors IV: Widths of trees and well-
quasi-ordering i: Graph minors: A survey
a b c d e f g h IInterface 0 0 1 0 0 0 0 0 0User 0 1 1 0 1 0 0 0 0System 2 1 1 0 0 0 0 0 0Human 1 0 0 1 0 0 0 0 0Computer 0 1 0 1 0 0 0 0 0Response 0 1 0 0 1 0 0 0 0Time 0 1 0 0 1 0 0 0 0EPS 1 0 1 0 0 0 0 0 0Survey 0 1 0 0 0 0 0 0 1Trees 0 0 0 0 0 1 1 1 0Graph 0 0 0 0 0 0 1 1 1Minors 0 0 0 0 0 0 0 1 1
![Page 12: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/12.jpg)
12
Vector Space Example cont.
system
interfaceuser
a
c
b
||||)cos(
BA
BAAB
a b cInterface 0 0 1User 0 1 1System 2 1 1
![Page 13: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/13.jpg)
13
Similarity in Vector Space
n
iiAA
1
2||
nnBABABABA ...2211
BA
BAAB
)cos(
Measures word overlap
Normalizes for different length vectors
Other m
etrics
exist
![Page 14: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/14.jpg)
14
Answering a Query UsingVector Space
• Represent query as vector
• Compute distances to all documents
• Rank according to distance
• Example– “computer system”
Query a b c d e f g h IInterface 0 0 0 1 0 0 0 0 0 0User 0 0 1 1 0 1 0 0 0 0System 1 2 1 1 0 0 0 0 0 0Human 0 1 0 0 1 0 0 0 0 0Computer 1 0 1 0 1 0 0 0 0 0Response 0 0 1 0 0 1 0 0 0 0Time 0 0 1 0 0 1 0 0 0 0EPS 0 1 0 1 0 0 0 0 0 0Survey 0 0 1 0 0 0 0 0 0 1Trees 0 0 0 0 0 0 1 1 1 0Graph 0 0 0 0 0 0 0 1 1 1Minors 0 0 0 0 0 0 0 0 1 1
![Page 15: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/15.jpg)
15
Common Improvements
• The vector space model– Doesn’t handle morphology (eat, eats, eating)– Favors common terms
• Possible fixes– Stemming
• Convert each word to a common root form
– Stop lists– Term weighting
![Page 16: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/16.jpg)
16
Handling Common Terms
• Stop list– List of words to ignore
• “a”, “and”, “but”, “to”, etc.
• Term weighting– Words which appear everywhere aren’t very
good discriminators – give higher weight to rare words
![Page 17: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/17.jpg)
17
tf * idf
)/log(* kikik nNtfw
log
Tcontain that in documents ofnumber the
collection in the documents ofnumber total
in T termoffrequency document inverse
document in T termoffrequency
document in term
nNidf
Cn
CN
Cidf
Dtf
DkT
kk
kk
kk
ikik
ik
![Page 18: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/18.jpg)
18
Inverse Document Frequency
• IDF provides high values for rare words and low values for common words
41
10000log
698.220
10000log
301.05000
10000log
010000
10000log
For a collectionof 10000 documents
![Page 19: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/19.jpg)
19
Probabilistic IR
• Vector space model robust in practice• Mathematically ad-hoc
– How to generalize to more complex queries?(intel or microsoft) and (not stock)
• Alternative approach: model problem as finding documents with highest probability of being relevant to the query– Requires making some simplifying assumptions about
underlying probability distributions– In certain cases can be shown to yield same results as
vector space model
![Page 20: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/20.jpg)
20
Probability Ranking Principle
For a given query Q, find the documents D that maximize the odds that the document is relevant (R):
( | , ) ( | )( | , )
( | , ) ( | )
P r D Q P r DP Q D r
P r D Q P r D
![Page 21: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/21.jpg)
21
Probability Ranking Principle
For a given query Q, find the documents D that maximize the odds that the document is relevant (R):
( | , ) ( | )( | , )
( | , ) ( | )
P r D Q P r DP Q D r
P r D Q P r D
Probability of document relevance to any query – i.e., the inherent quality of the document
![Page 22: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/22.jpg)
22
Probability Ranking Principle
For a given query Q, find the documents D that maximize the odds that the document is relevant (R):
( | , ) ( | )( | , )
( | , ) ( | )
P r D Q P r DP Q D r
P r D Q P r D
Probability that if document is indeed relevant, then the query is in fact Q
But where do we get that number?
![Page 23: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/23.jpg)
23
Bayesian nets for text retrieval
d1 d2
w1 w3
c1 c3
q1 q2
q0
w2
c2
DocumentNetwork
QueryNetwork
Documents
Words
Concepts
Query operators(AND/OR/NOT)
Information need
![Page 24: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/24.jpg)
24
Bayesian nets for text retrieval
d1 d2
w1 w3
c1 c3
q1 q2
q0
w2
c2
DocumentNetwork
QueryNetwork
Documents
Words
Concepts
Query operators(AND/OR/NOT)
Information need
Computed once for entire collection
![Page 25: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/25.jpg)
25
Bayesian nets for text retrieval
d1 d2
w1 w3
c1 c3
q1 q2
q0
w2
c2
DocumentNetwork
QueryNetwork
Documents
Words
Concepts
Query operators(AND/OR/NOT)
Information need
Computed for each query
![Page 26: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/26.jpg)
26
Conditional Probability Tables
P(d) = prior probability document d is relevant Uniform model: P(d) = 1 / Number docs In general, document quality P(r | d)
P(w | d) = probability that a random word from document d is w Term frequency
P(c | w) = probability that a given document word w has same meaning as a query word c Thesarus
P(q | c1, c2, …) = canonical form of operators AND, OR, NOT, etc.
![Page 27: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/27.jpg)
27
Example
Hamlet Macbeth
reason double
reason two
OR NOT
AND
trouble
trouble
DocumentNetwork
QueryNetwork
User Query
![Page 28: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/28.jpg)
28
Details
Set head q0 of user query to “true” Compute posterior probability P(D | q0) “User information need” doesn’t have to
be a query - can be a user profile, e.g., other documents user has read
Instead of just words, can include phrases, inter-document links
Link matrices can be modified over time. User feedback The promise of “personalization”
![Page 29: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/29.jpg)
29
Extensions
• Meet demands of web-based systems
• Modified ranking functions for the web
• Relevance feedback
• Query expansion
• Document clustering
• Latent Semantic Indexing
• Other IR tasks
![Page 30: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/30.jpg)
30
IR on the Web
• Query AltaVista with “Java”– Almost 107 pages found
• Avoiding latency– User wants (initial) results fast
• Solution– Rank documents using word-overlap– Use special data structure - inverted index
![Page 31: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/31.jpg)
31
Improved Ranking on the Web
• Not just arbitrary documents
• Can use HTML tags and other properties– Query term in <TITLE></TITLE>– Query term in <IMG>, <HREF>, etc. tag– Check date of document (prefer recent docs)– PageRank (Google)
![Page 32: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/32.jpg)
32
PageRank
• Idea: Good pages link to other good pages– Round 1: count in-links Problems?– Round 2: sum weighted in-links– Round 3: and again, and again…
• Implementation: Repeated random walk on snapshot of the web– weight frequency
visited
![Page 33: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/33.jpg)
33
Relevance Feedback
• System returns initial set of documents
• User identifies relevant documents
• System refines query to get documents more like those identified by user– Add words common to relevant docs– Reposition query vector closer to relevant docs
• Lather, rinse, repeat…
![Page 34: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/34.jpg)
34
Query Expansion
• Given query, add words to improve recall– Workaround for synonym problem
• Example– boat boat OR ship
• Can involve user feedback or not
• Can use thesaurus or other online source– WordNet
![Page 35: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/35.jpg)
35
Document Clustering
• Group similar documents– Similar means “close in vector space”
• If a document is relevant, return whole cluster
• Can be combined with relevance feedback
• GROUPERhttp://www.cs.washington.edu/research/clustering
![Page 36: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/36.jpg)
36
Clustering Algorithms
• K-means
• Hierarchical Agglomerative Clustering
Initialize k cluster centersLoop
Assign all document to closest centerMove cluster centers to better fit assignment
Until little movement
Initialize each document to a singleton clusterLoop
Merge two closest clustersUntil k clusters exist
Clusters
Cluster centers
Many ways to measure
distance between cluste
rs
![Page 37: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/37.jpg)
37
Latent Semantic Indexing
• Creates modified vector space
• Captures transitive co-occurrence information– If docs A & B don’t share any words, with
each other, but both share lots of words with doc C, then A & B will be considered similar
• Simulates query expansion and document clustering (sort of)
![Page 38: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/38.jpg)
38
Variations on a Theme
• Text Categorization– Assign each document to a category
– Example: automatically put web pages in Yahoo hierarchy
• Routing & Filtering– Match documents with users
– Example: news service that allows subscribers to specify “send news about high-tech mergers”
![Page 39: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/39.jpg)
39
Speech Recognition
TO BE COMPLETED
![Page 40: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/40.jpg)
40
Syntactic ParsingSemantic Interpretation
TO BE COMPLETED
![Page 41: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/41.jpg)
41
NLP Research Areas
• Morphology: structure of words• Syntactic interpretation (parsing): create a parse
tree of a sentence.• Semantic interpretation: translate a sentence into
the representation language.– Pragmatic interpretation: incorporate current situation
into account.
– Disambiguation: there may be several interpretations. Choose the most probable
![Page 42: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/42.jpg)
42
Some Difficult Examples
• From the newspapers:– Squad helps dog bite victim.– Helicopter powered by human flies.– Levy won’t hurt the poor.– Once-sagging cloth diaper industry saved by full
dumps.
• Ambiguities:– Lexical: meanings of ‘hot’, ‘back’.– Syntactic: I heard the music in my room.– Referential: The cat ate the mouse. It was ugly.
![Page 43: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/43.jpg)
43
Parsing
• Context-free grammars:
EXPR -> NUMBEREXPR -> VARIABLEEXPR -> (EXPR + EXPR)EXPR -> (EXPR * EXPR)
• (2 + X) * (17 + Y) is in the grammar.• (2 + (X)) is not.• Why do we call them context-free?
![Page 44: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/44.jpg)
44
Using CFG’s for Parsing
• Can natural language syntax be captured using a context-free grammar?– Yes, no, sort of, for the most part, maybe.
• Words:– nouns, adjectives, verbs, adverbs.– Determiners: the, a, this, that– Quantifiers: all, some, none– Prepositions: in, onto, by, through– Connectives: and, or, but, while.– Words combine together into phrases: NP, VP
![Page 45: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/45.jpg)
45
An Example Grammar
• S -> NP VP• VP -> V NP• NP -> NAME• NP -> ART N• ART -> a | the• V -> ate | saw• N -> cat | mouse• NAME -> Sue | Tom
![Page 46: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/46.jpg)
46
Example Parse
• The mouse saw Sue.
![Page 47: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/47.jpg)
47
Ambiguity
• S -> NP VP• VP -> V NP • VP -> V NP NP• NP -> N• NP -> N N• NP -> Det NP• Det -> the• V -> ate | saw | bought• N -> cat | mouse |biscuits | Sue | Tom
“Sue bought the cat biscuits”
![Page 48: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/48.jpg)
48
Example: Chart Parsing
• Three main data structures: a chart, a key list, and a set of edges
• Chart:
1 32
1
4
2
3
4
Starting pointslengt
h
Name of terminal or non-terminal
![Page 49: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/49.jpg)
49
Key List and Edges
• Key list: Push down stack of chart entries– “the” “box” “floats”
• Edges: rules that can be applied to chart entries to build up larger entries
1 32
1
4
2
3
4length
the
box
floats
detthe o
![Page 50: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/50.jpg)
50
Chart Parsing Algorithm
• Loop while entries in key list– 1. Remove entry from key list
– 2. If entry already in chart, • Add edge list
• Break
– 3. Add entry from key list to chart
– 4. For all rules that begin with entry’s type, add an edge for that rule
– 5. For all edges that need the entry next, add an extended edge (see algorithm on right)
– 6. If the edge is finished, add an entry to the key list with type, start point, length, and edge list
• To extend an edge with chart entry c
– Create a new edge e’
– Set start (e’) to start (e)
– Set end(e’) to end(e)
– Set rule(e’) to rule(e) with “o” moved beyond c.
– Set the righthandside(e’) to the righthandside(e)+c
![Page 51: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/51.jpg)
51
Try it
• S -> NP VP
• VP -> V
• NP -> Det N
• Det -> the
• N -> box
• V -> floats
![Page 52: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/52.jpg)
52
Semantic Interpretation
• Our goal: to translate sentences into a logical form.
• But: sentences convey more than true/false:– It will rain in Seattle tomorrow.– Will it rain in Seattle tomorrow?
• A sentence can be analyzed by:– propositional content, and– speech act: tell, ask, request, deny, suggest
![Page 53: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/53.jpg)
53
Propositional Content
• We develop a logic-like language for representing propositional content:– Word-sense ambiguity – Scope ambiguity
• Proper names --> objects (John, Alon)• Nouns --> unary predicates (woman, house)• Verbs -->
– transitive: binary predicates (find, go)– intransitive: unary predicates (laugh, cry)
• Quantifiers: most, some• Example: Mary: Loves(John, Mary)
![Page 54: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/54.jpg)
54
From Syntax to Semantics
• ADD SLIDES ON SEMANTIC INTERPRETATION
![Page 55: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/55.jpg)
55
Word Sense Disambiguation
• ADD SLIDES!
![Page 56: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/56.jpg)
56
Statistical NLP
• Consider the problem of tagging part-of-speech:– “The box floats”– “The” Det; “Box” N; “Floats” V;
• Given a sentence w(1,n), where w(i) is the i-th word, we want to find tags t(i) assigned to each word w(i)
![Page 57: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/57.jpg)
57
The Equations
• Find the t(1,n) that maximizes– P[t(1,n)|w(1,n)]=P[w(1,n)|t(1,n)]/P(w(1,n))– So, only need to maximize P[w(1,n)|t(1,n)]
• Assume that – A word depends only on previous tag– A tag depends only on previous tag– We have:
• P[w(j)|w(1,j-1),t(1,j)]=P[w(j)|t(j)], and• P[t(j)|w(1,j-1),t(1,j-1)] = P(t(j)|t(j-1)]
– Thus, want to maximize• P[w(n)|t(n-1)]*P[t(n+1)|t(n)]*P[w(n-1)|t(n-2)]*P[t(n)|t(n-1)]…
![Page 58: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/58.jpg)
58
Example• “The box floats”: given a corpus (a training set)
– Assignment one:• T(1)=det, T(2) = V, T(3)=V
• P(V|det) is rather low, so is P(V|V). Thus is less likely compared to
– Assignment two: • T(t)=det, T(2) = N; t(3) = V
• P(N|det) is high, and P(V|N) is high, thus is more likely!
– In general, can use Hidden Markov Models to find probabilities
det N Vbox
floats
the
![Page 59: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/59.jpg)
59
Experiments
• Charniak and Colleagues did some experiments on a collection of documents called the “Brown Corpus”, where tags are assigned by hand.
• 90% of the corpus are used for training and the other 10% for testing
• They show they can get 95% correctness with HMM’s.
• A really simple algorithm: assign t to w by the highest probability tag P(t|w) 91% correctness!
![Page 60: 1 Natural Language Processing Information Retrieval Speech Recognition Syntactic Parsing Semantic Interpretation CSE 592 Applications of AI Winter 2003.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d4e5503460f94a2cf87/html5/thumbnails/60.jpg)
60
Natural Language Summary
• Parsing:– context free grammars with features.
• Semantic interpretation:– Translate sentences into logic-like language– Use additional domain knowledge for word-
sense disambiguation.– Use context to disambiguate references.