How to Lie with StatisticsChad Orzel

Physics and Astronomy

10/5/04

“There are three kinds of lies:Lies, Damned Lies, and Statistics.” --attributed to Benjamin Disraeli

What’s This All About?Statistics are commonly used to deceive

Technically true, but deceptive

Preys on fear of numbers

“Math is hard!” --Barbie

False impression of accuracy

“Figures never lie, but liars figure.”

Need to know how to lie with statistics, to keep from being lied to with statistics.

Ways to Lie to Voters

0) FabricationJust make things up…

Can be very effective:

Lyndon Johnson:

“Make the son of a bitch deny it.”

Swift Boat Veterans for “Truth”

Not what we’re talking about today

Talking about ways to say things that are true, but misleading…

1) Is Male

2) Plans to Vote for Kerry

3) Has two siblings

4) Is 26 years old

5) Made $18,000 last year

All true statements, based on survey results!

Example:

A typical person in this class:

Ways to Lie to Voters

1) Omission Leave Things Out

Previous slide: What does “typical” mean?

Specify what kind of average you’re using:

Mean: Add ‘em up, divide by total number

Median: value in middle (half higher, half lower)

Not the same

Mean and MedianPhysics Data

Height

186 188 190 192 194

# M

easu

rem

ents

0

2

4

6

8

10

Nearly identical for random variables

“Normal Distribution”

“Bell Curve”

Very different for skewed data:

Mean: 190.1Median: 190

Mean affected by extreme values

Diverse populations

Median less sensitive to extremes

Usually better for economic data

Sibling Distribution

Number of Siblings0 2 4 6 8 10

# R

espo

ndan

ts

0

10

20

30

40

50

60

Example 1: Siblings

Most people have 0,1,2

Mean

Median

Few people with huge families

Pull mean up

Limited range

Can’t have < 0 siblings

Age Distribution

Age

20 30 40 50 60

# of

Res

pond

ant

s

0

5

10

15

20

25

30

35

Example 2: Age

Median

Mean

Diverse Population Problem

Students, mostly 19-22

(Much) older faculty

Nobody at mean age

Very bad description

Income Distribution

Income ($1,000's)

0 20 40 60 80 100

Num

ber

of R

espo

ndan

ts

0

20

40

60

80

100

Example 3: IncomeSort of silly, really…

Usually where this lie comes up:

“The average family willsave $2,000 under my taxplan…”

What kind of average?

Remember: The mean includes Bill Gates…

Median

Mean

Campaign Examples

Facts:

1) 25% receive NO cut

(drops mean to $1,217)

2) Median cut: $470

Half of all taxpayers get $470 or less

Fact:

Based on comparison of broad categories

Lost: Manufacturing jobs

Gained: “Service” jobs

Includes burger flippers

Kerry’s $9,000

“We're told that jobs that pay $9,000 less than the jobs that have been lost is the best that we can do.”

(http://www.factcheck.org/article.aspx?docID=228)

Bush Tax Cut

“111 million taxpayers will save, on average, $1,586 off their taxes.”

(http://www.factcheck.org/article.aspx?docID=145)

Ways to Lie to Voters

1) Omission (Continued)

The Fifth Dentist Problem

“Four out of five dentists surveyed…”

How many dentists total?

5 total: not a good sample

“Four out of five cards drawn from this deck were black!”

Leave out the sample size, and you canprove just about anything…

• Ratio of men registered to women registered: 58.6 to 41.4 percent• Estimated eligible voting population in Afghanistan: 9.8 million• Registered voters in Afghanistan, as of August 21: 10.3 million

• Reported number of registration cards a single Afghan has been able to obtain: from 2 to 40• Percent of the estimated eligible male population that is now registered to vote: 120 percent• Number of provinces that are over-registered: 13 (out of 30)• Number of provinces which registered voters exceed the population by 40% or more: 4

(http://www.tcf.org/afghanistanwatch/main.htm#voterregistrationfraud)

Campaign Example

“And that's what people are seeing now is happening in Afghanistan. Ten million citizens have registered to vote. It's a phenomenal statistic. That if given a chance to be free they will show up at the polls. Forty-one percent of those 10 million are women.”

--G.W. Bush, 1st Presidential Debate

Ways to Lie to Voters

2) Exaggeration Make Something of Nothing

Fear of big numbers:

“My opponent wants to spend $2 million on[something]…”

Sounds bad…

$2 million = 1/1,000,000th of the budget

= chump change

Need to put big numbers in context

Example: Guys Rule!

More Survey Data…

Scale axes to blow upsmall differences

Nothing false in graph

Creates false impression

Gender Distribution

Male Female

% o

f R

esp

on

da

nts

44

46

48

50

52

54

56

Example: Guys Rule!

Honest presentation:

Full scale shown

Bars same width, color

Slightly more male students

Not that big a difference

Gender Distribution

Male Female

% o

f R

esp

on

da

nts

0

25

50

75

100

(http://www.pollkatz.homestead.com/)

Example

What does margin of error really mean?

(http://www.washingtonmonthly.com/archives/individual/2004_08/004536.php)

Campaign Example

“In the first national telephone poll using a fresh sample, NEWSWEEK found the race now statistically tied among all registered voters, 47 percent of whom say they would vote for Kerry and 45 percent for George W. Bush in a three-way race.” --MSNBC

(1,013 voters surveyed, Margin of Error +/- 4%)

“According to the first post-debate poll, from Newsweek, John Kerry leads President Bush by a margin of 49% to 46%. Put Nader in the mix and Kerry's margin drops from 3 to 2.”

--Josh Marshall, Talking Points Memo (weblog)

Other Ways to Lie

3) Misdirection True, but Irrelevant

Quote impressive statistics about side issues

Creates false impression of real support

4) False Correlation Post Hoc Fallacy

Homicide rates peak in summer

Ice cream sales peak in summer

Therefore, ice cream leads to murder?

Correlation is not Causation

What to Do?Questions to ask about any statistic:

1) Who created it?

Do they have an agenda?

2) Why was it created?

Research or politics?

3) How was it created?

Methodology

What to Do? (continued)Questions to ask about any statistic:

4) What’s missing?

Is there hidden context?

5) Is it relevant?

Avoid misdirection

6) Does it make sense?

If it sounds ridiculous, it probably is…