Dark knowledge in Qualitative Reasoning: A call to Arms

Ken Forbus & Dedre Gentner

Northwestern University

Example of the Problem

• Currently CO2 entering the atmosphere exceeds rate of its removal– Therefore it accumulates

– Stopping emissions growth (i.e., freezing emissions at their current level) would not change this

• Steerman (2009) found that 84% of participants drew response curves to CO2 in atmosphere that showed atmospheric CO2 stabilizing. – 3/5ths had degrees in science, engineering, or

mathematics (rest: economics)

– 30% had advanced degrees, 70% of which were in science, engineering, or math

Mental Models Matter

• Climate change– “Wait and see” attitude more prevalent in countries

catching up economically

– Save the planet?

• Financial meltdown– People resist the idea that markets do fall

• Evidence of failures in education– Policy influences economics fog produced by special

interests

– Education as an adversarial game

Hypothesis: QR can help

• Formalisms for causal qualitative models• Thinking tools at natural level of abstraction• Some attempts have been made

– Betty’s brain

– Vmodel

– GARP models of Brazilian ecosystem phenomena.

• Not much headway yet. Why?– Not enough effort expended yet?

– Not enough time to diffuse and have impact?

– Claim: We need to take a more psychologically realistic look at how mental models work

Overview

• How mental models work (we think)– Experience = the dark knowledge that binds our

conceptual universe together

• How to improve human qualitative reasoning– Analogy as mechanism for integrating knowledge

– Example: Bathtubs and climate change

– Example: Bathtubs and credit card debt

• What to do?– Understand better when and how experience is used

– Create tools that can exploit experiential knowledge as well as abstract domain theories

Standard QR model

1st principlesDomain Theory

SituationDescription

Model Builder QualitativeSimulator

ScenarioModel

Predictions ofpossible behaviors

Alternative: Experience-based modeling

• People store and use episodic knowledge– True in all parts of human cognition, why would QR be

different?

– Provides robustness, heuristics for quick decisions, reality checks

• Q: How much folic acid does a pregnant woman need?

• A: Six tons (TREC system)

• Analogy provides means of directly reasoning with episodic knowledge– Similarity-based retrieval works well for within-

domain analogs

– No need to make knowledge into rules before using it

Culture as a Source of Experience

• Many things we understand we’ve never directly experienced– Revolutionary War, molecular structure, plate

tectonics, evolution

• Vygotsky: Much of our knowledge is learned by interactions with other people– Apprenticeship

– Reading, other media

• Implication: Our systems need fluid interaction with human teachers, learners & collaborators

Experience = Dark Knowledge in QR

• In physics,– Dark matter outweighs the matter we can see

– Its gravitational force is what holds the universe together

• In QR,– We’ve focused on abstract, 1st principles knowledge

– Experiential knowledge in human cognition far outweighs the abstract domain theories that we tend to focus on

– Its use in heuristic and analogical reasoning is what holds our conceptual universe together

Experience-based QR (one version)

Experience

1st principles domain theory

Current Situation

Standard QR

Analogical Retrieval

Predictions

Predictions

What Happens

Analogical Generalization

Learning mental models

• Experiences → Protohistories → Causal models → Naïve Physics → Expert Models– Forbus & Gentner, 1986; 1997

– Recall Scott Friedman’s talk

NL

Temporally encodedCycL exemplars

def-EH: participants: … conditions: …

consequences: …

def-EH: participants: … conditions: …

consequences: …

New Scenarios

NL

Reasoning

List of target concepts

{pushing, moving, blocking}

SEQL Generalizations

EncapsulatedHistories

Comic-strip stimuli

Protohistories

Causal models

Analogies

http://throbgoblins.blogspot.com/2009/01/boffin-bathtub-again.html

Analogy as knowledge integration

• Students often “check their intuition at the door”• Comparing their analyses with everyday

experience could improve accuracy• But only if they understand the everyday world

– E.g., Gentner & Gentner (1983), Some participants in water-electricity analogy study didn’t understand basic facts about water pressure

Stocks & Flows• Sweeney & Steerman, 2000: Bathtub Dynamics

The bathtub initially contains 100 liters.

Draw the behavior of the quantity of water on the lower graph

Stocks & Flows• Sweeney & Steerman, 2000: Bathtub Dynamics

A correct student answer

Typical Incorrect Answers

Techniques for using analogy effectively

• Make sure the base domain is well understood• Work through the correspondences explicitly and

carefully– Richland et al 2009, in studying use of math analogies

across instructors in US, Japan, and China

• Having learners compare cases can double transfer effectiveness, including own memory retrieval– Gentner et al 2003, in preparation

Bathtubs, revisited

Bathtubs and Credit Cards

Bathtub Credit Card

Faucet Setting Monthly charges

Drain setting Monthly payment

Level of water Amount of debt

??? Interest rate

Projecting backwards to understand differences

Mr. Will on ice

• George Will, Washington Post, 2/15/09:“As global levels of sea ice declined last year, many experts said this was evidence of man-made global warming. Since September, however, the increase in sea ice has been the fastest change, either up or down, since 1979, when satellite record-keeping began.. According to [authority], global sea ice levels now equal those of 1979.”

Mr. Will on ice

• George Will, Washington Post, 2/15/09:“As global levels of sea ice declined last year, many experts said this was evidence of man-made global warming. Since September, however, the increase in sea ice has been the fastest change, either up or down, since 1979, when satellite record-keeping began.. According to [authority], global sea ice levels now equal those of 1979.”

“No single year marks a trend or holds evidence of long-term

climate change.”-- Andrew Revkin, NYT dot-earth

blog

QR can be subtle

• Jennifer Francis, Rutgers:“At the end of summer each year, the sea ice refreezes and continues to do so until late spring. Thin ice and open water generate new ice faster than thick ice, as heat from the ocean below is able to escape more easily into the atmosphere. In the autumn of 2007 and 2008, the rate of ice production was very large because there was so much open water and thin ice – The rapid growth is completely expected.

What is to be done?

• Better understanding of the nature of experiential knowledge in QR

• Build better tools to support reasoning and learning– QR tools

– Analogy tools that combine QR and episodic knowledge.

EA NLU provides psychologically

plausible language outputs

Building Experience-based Systems

COMLEXlexicon

ResearchCycKB Contents

Allen Parser

Query-based

AbductiveSemantic

Interpreter

DRT-basedPacked

SemanticsBuilder

Formal representation

of story

Then child child13 is playing with

the truck truck13.

CogSketch provides

psychologically plausible visual

and spatial processing

Better tools

• QR systems need to exploit concrete, experiential knowledge more– Garp3, Gizmo, Envision – all require abstract domain

theories, & don’t support concrete knowledge.

– Betty’s Brain, Vmodel – concrete knowledge only, no logical quantification.

– QCM bet: Easier for everyone to encode concrete knowledge first

• Analogy- based tools– Help people work through analogies

– Suggest potential analogs

Summary

• Two of our current crises are caused in part by mental model deficiencies– Even technically trained people do not understand

accumulation of CO2 in the atmosphere

– Our financial crises rely on a variety of faulty models

• Experiential knowledge is the “dark matter” of QR– It far outweighs abstract domain theories, and provides the

forces that hold our conceptual universes together

• QR research can help– Need to better understand how experiential knowledge

works

– Need to build better tools, that incorporate experiential knowledge and support analogical reasoning

Questions, suggestions?