Post on 28-Mar-2015
Kees van Deemter, Dublin, Trinity College, May 2009
What utility can do for NLG: the case of vague language
Kees van Deemter
University of Aberdeen
Scotland, United Kingdom
Kees van Deemter, Dublin, Trinity College, May 2009
Two big issues
• Does NLG have proper foundations?– What is its mathematical core?
• Why is language vague?– When does vague language have benefits
over crisp language?
Kees van Deemter, Dublin, Trinity College, May 2009
1. NLG and utility
2. Why is language vague?
3. Why should NLG systems be vague?
4. Conclusions
Kees van Deemter, Dublin, Trinity College, May 2009
Why bother about “foundations” for NLG?
Kees van Deemter, Dublin, Trinity College, May 2009
Why bother about “foundations” for NLG?
• Computing is based on– theory of formal languages/computability– Hoare logic/program correctness
• Linguistics is based on– data & stats– the theory of formal languages– mathematical logic/set theory
Kees van Deemter, Dublin, Trinity College, May 2009
Why bother about “foundations” for NLG?
• Computing is based on– theory of formal languages/computability– Hoare logic/program correctness
• Linguistics is based on– data & stats– the theory of formal languages (syntax)– mathematical logic/set theory (semantics)
Kees van Deemter, Dublin, Trinity College, May 2009
Why bother about “foundations” for NLG?
• Computing is based on– theory of formal languages/computability– Hoare logic/program correctness
• Linguistics is based on– data & stats– the theory of formal languages (syntax)– mathematical logic/set theory (semantics)– how about pragmatics?
• Not just what’s true, but what’s appropriate given the circ’s.
Kees van Deemter, Dublin, Trinity College, May 2009
• An NLG program “translates” input to linguistic output
• Essentially the problem of choosing the best Form for a given Content. E.g.– McDonald 1987 – Bateman 1997 on Systemic Grammar
• What does this choice depend on?
Kees van Deemter, Dublin, Trinity College, May 2009
• What could this choice depend on? – Roughly: which utterance is most “useful”
• In other words, utility
• A perspective seldom pursued. Exceptions:– Kibble 2003 (IWCS-5, referring expressions)– Klabunde 2009 (ENLG-12)
• NLG programs become more expressive, so this perspective now becomes tempting
Kees van Deemter, Dublin, Trinity College, May 2009
Example: the hazards of road ice
Kees van Deemter, Dublin, Trinity College, May 2009
Kees van Deemter, Dublin, Trinity College, May 2009
Example: Roadgritting (Turner et al. 2009, this conference)
Compare1. “Roads above 500m are icy”2. “Roads in the Highlands are icy”
Decision-theoretic perspective:1. 100 false positives, 2 false negatives2. 10 false positives, 10 false negatives
Supposeeach false positive has utility of -0.1each false negative has utility of -2
Kees van Deemter, Dublin, Trinity College, May 2009
Example: Roadgritting (Turner et al. 2009, this conference)
Supposefalse positive has utility of -0.1
false negative has utility of -2
Then1: 100 false pos, 2 false neg = -14
2: 10 false pos, 10 false neg = -21
So summary 1 is preferred over summary 2.
Kees van Deemter, Dublin, Trinity College, May 2009
Communication in Game Theory (following Crawford & Sobel 1982)
• Set of contents C• Set of forms F• Set of actions A• Speaker strategy S : CF• Hearer strategy H : FA• Two utility functions
US : A[0,1]UH : A[0,1] (not necessarily the same!)
Kees van Deemter, Dublin, Trinity College, May 2009
A special case: Vagueness
• Two related questions:
1. Can Game Theory help us explain why language is so often vague?
2. Can Game Theory tell NLG systems when to use vagueness?
Kees van Deemter, Dublin, Trinity College, May 2009
• Vagueness: An expression is vague iffit has borderline cases.
– Example of a vague adjective: “poor” is vague because some people are borderline poor.
– Saying this differently: different thesholds for “poor” may be used
Kees van Deemter, Dublin, Trinity College, May 2009
Example: Is John poor?
£10.000 p/a
£30.000 p/a
£20.000 p/a
poverty Norm A
poverty Norm B
John
Norm A: “John is poor” is True
Norm B: “John is poor” is False
Kees van Deemter, Dublin, Trinity College, May 2009
• Other vague adjectives: ‘large’, ‘small’, ...
• Vague nouns: ‘girl’, ‘giant’, ‘island’,...
• Vague determiners: ‘many’, ‘few’, ...
• Vague adverbs: ‘often’, ‘slowly’, ...
Relevant for any NLG system with “continuous” input– weather forecasts (FOG, Sumtime-Mousam)– patient data (e.g. Babytalk)
Kees van Deemter, Dublin, Trinity College, May 2009
From Babytalk corpus
“BREATHING – Today he managed 1½ hours off CPAP in about 0.3 litres nasal prong oxygen, and was put back onto CPAP after a desaturation with bradycardia. However, over the day his oxygen requirements generally have come down from 30% to 25%. Oxygen saturation is very variable. Usually the desaturations are down to the 60s or 70s; some are accompanied by bradycardia and mostly they resolve spontaneously, though a few times his saturation has dipped to the 50s with bradycardia and gentle stimulation was given. He has needed oral suction 3 or 4 times today, oral secretions are thick.” [BT-Nurse scenario 1]
Kees van Deemter, Dublin, Trinity College, May 2009
• The question is: “When (if ever) is vague communication more useful than crisp communication?”
• The question is not: “Can vague communication be of some use?”
• Compare Rohit Parikh (2000)
• Ann calls Bob to bring “the blue book”, her only book on topology
Kees van Deemter, Dublin, Trinity College, May 2009
Kees van Deemter, Dublin, Trinity College, May 2009
Example by R.Parikh (the book scenario)
• Bob only has to search all blue books
• Ann’s instruction reduces the number of books that Bob can expect to have to check.
• Each calls some books blue that the other does not. But they agree on most books.
Kees van Deemter, Dublin, Trinity College, May 2009
Ann’s books
blue-Ann=250
blue-Bob=300
blue-Ann-Bob=225
Ann’s books =1000
Kees van Deemter, Dublin, Trinity College, May 2009
What’s the utility of “the blue book”?
Compare expected search times1. without this instruction
2. with this instruction
1. Without instruction: ½*1000 = 500
2. With instruction: 9/10*(½*300) + 1/10*(300+(1/2*700)) = 200
Kees van Deemter, Dublin, Trinity College, May 2009
In Parikh’s example, “blue” is crisp.
Scenario can be generalised to situations where each allows boundary cases.
Kees van Deemter, Dublin, Trinity College, May 2009
Why is language vague?
Barton Lipman (in A.Rubinstein, “Economics and Language”, CUP 2000; working paper “Why is Language Vague” (2006))
When/why does vague communication give higher payoff than crisp language?
Kees van Deemter, Dublin, Trinity College, May 2009
One type of answer: conflict
• S and H may have very different utility functions US and UH
• (Crawford & Sobel 1982, Aragones and Neeman 2000): If US and UH are very different, it can be advantageous to hide information– “Our food is healthy!”– “Our burgers are big!”
Kees van Deemter, Dublin, Trinity College, May 2009
Kees van Deemter, Dublin, Trinity College, May 2009
One type of answer: conflict
• S and H may have very different utility functions US and UH
• Crawford & Sobel 1982, Aragones and Neeman 2000: If US and UH are very different, it can be advantageous to hide information– “Our food is healthy!”– “Our burgers are big!”
• Henceforth (following Lipman): US = UH
Kees van Deemter, Dublin, Trinity College, May 2009
Lipman’s questions
When does vague communication give higher payoff than crisp language?
Lipman: the airport scenario
Kees van Deemter, Dublin, Trinity College, May 2009
Kees van Deemter, Dublin, Trinity College, May 2009
Lipman’s scenario
Example: Airport scenario: I describe Mr X to you, so you can pick up X from the airport. All I know is X’s height; heights are distributed across people uniformly on [0,1]. If you identify X right away, you get payoff 1; if you don’t then you get payoff -1
Kees van Deemter, Dublin, Trinity College, May 2009
Lipman: the airport scenario
What description would work best?• Optimal communication: state X’s height
as precisely as possible. If each of us knows X’s exact height then the probability of confusion is close to 0.
Kees van Deemter, Dublin, Trinity College, May 2009
Lipman: the airport scenario
What description would work best?
• Optimal communication: state X’s height as precisely as possible. If each of us knows X’s exact height then the probability of confusion at the airport is close to 0.
Lipman: This is not vague, because there are no boundary cases!
Kees van Deemter, Dublin, Trinity College, May 2009
Some possible answers to Lipman
Kees van Deemter, Dublin, Trinity College, May 2009
1. Necessary vagueness?
“Input may be vague”. E.g.:– verbatim repetition (“hearsay”) – memory may cause details to fade
(e.g., number of casualties in a disaster)– perception may have been inadequate
(e.g., the height of a seated person)
Kees van Deemter, Dublin, Trinity College, May 2009
1. Necessary vagueness?
“Input may be vague”. E.g.:– verbatim repetition (“hearsay”)
This begs the question– memory may cause details to faded
(e.g., number of casualties in a disaster) ?– perception may have been inadequate
(e.g., the height of a seated person)Lipman: Why can’t we convey exactly what our perception/memory is?
E.g. “24 degrees +/- 2 degrees”
Kees van Deemter, Dublin, Trinity College, May 2009
the hazards of measurement 11m 12m
Kees van Deemter, Dublin, Trinity College, May 2009
• Example: One house of 11m height and one house of 12m height
1. “the 12m house needs to be demolished”
2. “the tall house needs to be demolished”
• Comparison is easier and more reliable than measurement prefer utterance 2 (Van Deemter 2006)
Kees van Deemter, Dublin, Trinity College, May 2009
• Example: One house of 11m height and one house of 12m height
1. “the 12m house needs to be demolished”
2. “the tall house needs to be demolished”
• Comparison is easier and more reliable than measurement prefer utterance 2
• But arguably, this utterance is not vague
Its vagueness is merely local
Kees van Deemter, Dublin, Trinity College, May 2009
Apparent vagueness is frequent
• ‘the tall house’ the tallest house
• ‘Physical exercise is good for young and old’ regardless of age
• ‘Bad for bacteria, good for gums’ gums improve as a result of bacterial death
• ‘Fast-flowing rivers are deep’ the faster the deeper (positive correlation between variables)
Kees van Deemter, Dublin, Trinity College, May 2009
3. Production/interpretation Effort
• GTh can reason about the utility of an utterance• Effort needs to be commensurate with utility. In
many cases, more precision adds little benefit (cf. Prashant Parikh 2000, Van Rooij 2003, Jaeger 2008)
• E.g., the feasibility of an outing does not depend on whether it’s 20C or 30C.
• ‘Mild’ takes fewer syllables than ‘twenty three point seven five’.– Vague words tend to be short (Krifka 2002)
Kees van Deemter, Dublin, Trinity College, May 2009
But: Why not round the figure?
“The temperature is 24C”
Kees van Deemter, Dublin, Trinity College, May 2009
4.Evaluation payoff
• Example: You ask the doctor about your blood pressure.– Utterance 1: “Your blood pressure is 150/90.”– Utterance 2: “Your blood pressure is high.”
• U2 offers less detail than U1• But U2 also offers more: an evaluation of
your condition.– A link with actions (cut down on salt, etc.)– Especially useful if metric is “difficult”
Kees van Deemter, Dublin, Trinity College, May 2009
– But why does English not have a (brief) expression that says “Your blood pressure is 150/90 and too high”?
Compare“You are obese”
means
“Your BMI is above 30 and this is dangerous”.
Kees van Deemter, Dublin, Trinity College, May 2009
5.Lack of a good metric
• Maybe areas where there exists a generally accepted measurement are rare– Multidimensional measurements: What’s the
size of a house?– Maths: How difficult is a proof? (“As the
reader may easily verify”)
Kees van Deemter, Dublin, Trinity College, May 2009
5.Lack of a good metric
• Maybe areas where there exists a generally accepted measurement are rare– Multidimensional measurements: What’s the
size of a house?– Maths: How difficult is a proof? (“As the
reader may easily verify”)– How beautiful is a sunset?
Kees van Deemter, Dublin, Trinity College, May 2009
Kees van Deemter, Dublin, Trinity College, May 2009
6.Future contingencies
• Indecent Displays Control Act (1981) forbids public display of indecent matter
Kees van Deemter, Dublin, Trinity College, May 2009
• Indecent Displays Control Act (1981) forbids public display of indecent matter
• Indecent at the time
the law has been parameterised
(Waismann 1968, Hart 1994, Lipman 2006)
Kees van Deemter, Dublin, Trinity College, May 2009
7. Vagueness facilitates search
Kees van Deemter, Dublin, Trinity College, May 2009
7. Vagueness facilitates search
Palace scenario: A diamond has been stolen from the Emperor. The thief must have been one of the 1000 eunuchs. The sole witness gets wounded and says, with his last breath, “the thief is tall”.
What should the emperor do?
Kees van Deemter, Dublin, Trinity College, May 2009
What if ‘tall’ is crisp?
• Separate eunuchs: tall / not tall
tall 500
not tall 500
• Expect to search ½*500=250 tall eunuchs
• What if witness regards more people as tall?
• Seach 500 tall ones + 250 not-tall ones
• No difference between the “non-tall” ones
Kees van Deemter, Dublin, Trinity College, May 2009
What if ‘tall’ is recognised as having borderline cases?
• Separate: tall / not tall / borderline
1. tall 100
2. borderline-tall 400
3. not-tall 500
Assume: probability of being called tall is highest for (1) and lowest for (3).
Kees van Deemter, Dublin, Trinity College, May 2009
What if ‘tall’ is recognised as having borderline cases?
• Separate: tall / not tall / borderline
1. tall 100
2. borderline-tall 400
3. not-tall 500
Assume: probability of being called tall is highest for (1) and lowest for (3).
E.g., assuming the probabilities in the figure: expect to search 0.5*50+0.5(100+200)=175
p = 0.5
p = 0.5
p = 0
Kees van Deemter, Dublin, Trinity College, May 2009
What if ‘tall’ has a continuity of degrees?
• This would allow the emperor to rank eunuchs by their height:
(1) tallest
(2) tallest but 1
(3) tallest but 2
...
(n) shortest
Assume: probability of being called tall is highest for (1), second-highest for (2), etc.
Emperor should search (1) first, then (2), etc.
Kees van Deemter, Dublin, Trinity College, May 2009
First lesson
• Continuous domains invite differences between subjects, regarding their interpretation thresholds
• Stubborn insistence on your own thresholds is foolish: flexibility pays
• Minimal flexibility: tall/not tall/borderline
• Maximal flexibility: continuum of degrees
Kees van Deemter, Dublin, Trinity College, May 2009
Second lesson
• The emperor’s three search strategies correspond with three logics of vagueness:
• Knowledge as ignorance: “tall” is really crisp; we just don’t know where the threshold is.
• Partial Logic: true, false, truth value gap.
• Degree theories (e.g. Fuzzy Logic).
Kees van Deemter, Dublin, Trinity College, May 2009
Second lesson
• The emperor’s three search strategies correspond with three logics of vagueness:
• Knowledge as ignorance: “tall” is really crisp; we just don’t know where the threshold is. Corresponds with 1st strategy
• Partial Logic: true, false, truth value gap. Corresponds with the 2nd strategy
• Degree theories (e.g. Fuzzy Logic). Corresponds with 3rd strategy
Kees van Deemter, Dublin, Trinity College, May 2009
Second lesson
• This suggests that Knowledge as Ignorance is probably “wrong”, and so is Partial Logic.
Kees van Deemter, Dublin, Trinity College, May 2009
Relevance for NLG
• We have focussed on the question “Why is language vague?”
• What does this mean for NLG?– Do our answers translate directly to the
question when NLG systems should use vagueness?
Kees van Deemter, Dublin, Trinity College, May 2009
Vagueness triggers for NLG?
1. Necessary vagueness
2. Apparent vagueness
3. Cost reduction
4. Bias/evaluation
5. Lack of good metric
6. Future contingencies
7. Facilitation of search
Kees van Deemter, Dublin, Trinity College, May 2009
Vagueness triggers for NLG?
1. Necessary vagueness. Yes: When NLG systems take vague information as starting point. (e.g.: Colour(x)=red)
2. Apparent vagueness. Yes: vague descriptions (e.g. “the figure on the left”)
3. Cost reduction: Yes: the fact that English could have had a short expression for “24 degrees +/- 2” is irrelevant.
Kees van Deemter, Dublin, Trinity College, May 2009
Vagueness triggers for NLG?
4. Bias/evaluation. (Similar to 3.) the fact that English could have had an expression for “these shoes cost 200 pounds and I disapprove of that” is irrelevant. The generator can only choose between existing expressions: ”200 pounds”, and “expensive”
Kees van Deemter, Dublin, Trinity College, May 2009
Vagueness triggers for NLG?
5. Lack of good metric. Relevant to NLG in principle, given challenging applications. (E.g. estate adverts)
6. Future contingencies. Similar to (5).7. Facilitation of search. Relevant(?)
unless system defines its terms explicitly, removing differencs between subjects (e.g., Reiter et al. 2005, the word “evening”)
Kees van Deemter, Dublin, Trinity College, May 2009
Towards an algorithm
• In what follows, just a few factors have been taken into account
• Caveat: Any resemblance with empirically supported facts is accidental
Kees van Deemter, Dublin, Trinity College, May 2009
Towards an algorithm
uncertainty precision evaluation cost total penalty payoff benefit
“39.82C” -1 9 0 -4 4
“approx. 40C” 0 8 0 -3 7
“high fever” 0 6 2 -2 8
Kees van Deemter, Dublin, Trinity College, May 2009
Written material
“What Game Theory can do for NLG: the case of vague language”. Proc. 12th European workshop on Natural Language Generation (ENLG-2009)
Kees van Deemter, Dublin, Trinity College, May 2009
Not Exactly:In Praise of Vagueness
Oxford University Press To appear Jan 2010
Part 1: vagueness in science and daily life
Part 2: logic and vaguenessPart 3: vagueness and AI
Kees van Deemter, Dublin, Trinity College, May 2009
Not Exactly:In Praise of Vagueness
Oxford University Press To appear Jan 2010
Chapter 11: “When to be vague?”
Kees van Deemter, Dublin, Trinity College, May 2009
• “Payoff” could become central to NLG• New research by game theorists,
economists, logicians, linguists (e.g., Lipman, Van Rooij, Jäger, de Jaegher, Aragones, Neeman)
Kees van Deemter, Dublin, Trinity College, May 2009
• Lipman: “Why is language vague?”
• Tentative answers are beginning to emerge
• Do not confuse with the question “When should an NLG system (or a human speaker) be vague?”
since this depends on what the language can express
Kees van Deemter, Dublin, Trinity College, May 2009
• Lipman: “Why is language vague?”• Tentative answers are beginning to
emerge• Do not confuse with the question
“When should an NLG system (or a human speaker) be vague?”
since this depends on what the language can express
• Compare De Saussure: langue/parole
Kees van Deemter, Dublin, Trinity College, May 2009
Thanks ...
• to Ehud Reiter and Albert Gatt.