2. Shallow versus deep uncertainties Why many predictions / forecasts fail.
-
Upload
rudolf-harrell -
Category
Documents
-
view
219 -
download
0
Transcript of 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.
![Page 1: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/1.jpg)
2. Shallow versus deep uncertaintiesWhy many predictions / forecasts fail
QuickTime™ and a decompressor
are needed to see this picture. QuickTime™ and a decompressor
are needed to see this picture.
![Page 2: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/2.jpg)
Society is playing a high-stakes game of chance against
nature
We want to
- assess the hazard - how often dangerous events happen - mitigate or reduce the risk - the resulting losses.
Often nature surprises us, when an earthquake, hurricane, or flood is bigger or has greater effects than expected from hazard assessments.
In other cases, nature outsmarts us, doing great damage despite expensive mitigation measures, or causing us to waste resources on what proves a minor hazard.
![Page 3: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/3.jpg)
![Page 4: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/4.jpg)
Hazard assessment failed
2010 map predicts probability of strong shaking in next 30 years
But: 2011 M 9.1 Tohoku, 1995 Kobe M 7.3 & others in areas mapped as low hazard
In contrast: map assumed high hazard in Tokai “gap”
Geller 2011
![Page 5: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/5.jpg)
Hazard model divided trench into segments
Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2(Headquarters for Earthquake Research Promotion)
![Page 6: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/6.jpg)
Giant earthquake broke many segments
2011 Tohoku Earthquake 450 km long fault, M 9.1 (Aftershock map from USGS)
J. Mori
Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2(Headquarters for Earthquake Research Promotion)
![Page 7: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/7.jpg)
Tsunami runup approximately twice
fault slip (Plafker, Okal &
Synolakis 2004)
M9 generates much larger tsunami
Planning assumed maximum magnitude 8 Seawalls 5-10 m high
CNN
NYTStein & Okal, 2011
![Page 8: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/8.jpg)
NY Times 3/31/2011
Mitigation failed
Expensive seawalls - longer than Great Wall of China -proved ineffective
Tsunami overtopped 10m high sea walls, causing more than 15,000 deaths and $210 billion damage.
![Page 9: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/9.jpg)
What’s going wrong?
Shallow uncertainty - we don’t know what will happen, but know the odds (probability
density function). The past is a good predictor of the future. We can make math
models that work well.
Deep uncertainty - we don’t know the odds. The past is a poor predictor of the future. We can make math models, but
they generally won’t work well.
![Page 10: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/10.jpg)
Shallow uncertainty is like estimating the chance that a batter will get a hit. His batting average is a good predictor.
Deep uncertainty is like trying to predict the winner of the World Series five years from now. Teams' past performance give only limited insight into the future.
![Page 11: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/11.jpg)
Due to deep uncertainty
Predicted natural or other disaster probabilities are hard to estimate and thus often very inaccurate
The world is more complicated than we think or admit
Prob(sinking) = 0
![Page 12: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/12.jpg)
1986 – Loss of shuttle Challenger
NASA claimed probability of loss = 1/100,000
Richard Feynman argued for 1/100 – 1000 times higher.
As he pointed out in his report dissenting from the government investigation commission, because this rate implies that "one could put up a shuttle every day for 300 years expecting to lose only one, we could properly ask what is the cause of management's fantastic faith in the machinery... "
![Page 13: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/13.jpg)
1986 – Loss of shuttle Challenger
NASA claimed probability of loss = 1/100,000
Richard Feynman argued for 1/100 – 1000 times higher.
As he pointed out in his report dissenting from the government investigation commission, because this rate implies that "one could put up a shuttle every day for 300 years expecting to lose only one, we could properly ask what is the cause of management's fantastic faith in the machinery... "
In 2003, shuttle Columbia was lost on the 107th shuttle mission
2 lost in 107 missions ≈ 1/50
![Page 14: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/14.jpg)
Activity 2.1:
Given that the shuttle was a new spacecraft, before flights started, how could you assess NASA’s estimated probability of loss = 1/100,000?
![Page 15: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/15.jpg)
Activity 2.1:
Given that the shuttle was a new spacecraft, before flights started, how could you assess NASA’s estimated probability of loss = 1/100,000?
One way: of 11 Apollo missions, one (Apollo 13) suffered near disaster. In addition, Apollo 1 was lost in a launch pad fire.
![Page 16: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/16.jpg)
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Boeing 787 Dreamliner batteries
Boeing “concluded that they were likely to emit smoke less than once in every 10 million flight hours.
Once the planes were placed in service, the batteries overheated and emitted smoke twice, and caused one fire, after about 50,000 hours of commercial flights.” (NYT, 2/7/13)
![Page 17: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/17.jpg)
From 1975 to 2007, U.S. housing prices grew steadily. Neither Washington nor Wall Street recognized that this could not go on forever, or worried that trillions of dollars of risky
mortgages were embedded throughout the financial system
![Page 18: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/18.jpg)
QuickTime™ and a decompressor
are needed to see this picture.
As housing prices and subprime lenders collapsed, Wall Street & government weren’t concerned based
on computer models
![Page 19: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/19.jpg)
The models were wrong
QuickTime™ and a decompressor
are needed to see this picture.
![Page 20: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/20.jpg)
NY Times 3/21/11
![Page 21: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/21.jpg)
Hazard maps are hard to get right: successfully predicting future shaking depends on accuracy of four
assumptions over 500-2500 years
Where will large earthquakes occur?
When will they occur?
How large will they be?
How strong will their shaking be?
Uncertainty & map failure result because these are often hard to assess, given that the earthquakes are much more variable in space and time than the short
earthquake history shows
![Page 22: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/22.jpg)
2008 Wenchuan earthquake (Mw 7.9) was not expected: map showed low hazard based on lack of recent
earthquakes
Didn’t use GPS data showing 1-2 mm/yr (~Wasatch)
Earthquakes prior to the 2008 Wenchuan event
Aftershocks of the Wenchuan event delineating the rupture zone
QuickTime™ and a decompressor
are needed to see this picture.
![Page 23: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/23.jpg)
GSHAP 1999
NUVEL-1Argus, Gordon, DeMets & Stein, 1989
Swafford & Stein, 2007
Slow plate boundary
Africa-Eurasia convergence rate varies smoothly (5 mm/yr)
![Page 24: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/24.jpg)
2004
2003
Swafford & Stein, 2007
GSHAP 1999
NUVEL-1Argus, Gordon, DeMets & Stein, 1989
M 6.4
M 6.3
Slow plate boundary
Africa-Eurasia convergence rate varies smoothly (5 mm/yr)
![Page 25: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/25.jpg)
Familiar pattern
2001 hazard map
http://www.oas.org/cdmp/document/seismap/haiti_dr.htm
2010 M7 earthquake shaking much greater than predicted for next 500 years
![Page 26: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/26.jpg)
Italian hazard maps, which predicted the expected shaking in the next 500 years, forecast some earthquake locations well and others poorly, and so required updating within a decade.
![Page 27: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/27.jpg)
A posteriori changes to a model are "Texas sharpshooting:” shoot
at the barn and then draw
circles around the bullet holes.
![Page 28: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/28.jpg)
Overfitting: a subtle trap
We can fit very complicated models to data like earthquake histories, but we are partly fitting noise
Flipping a coin gives lots of complicated patterns
H H T T H T T T H T H H T H
We could fit a model to those data, but it would do no better than 50% at predicting the next flip
![Page 29: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/29.jpg)
Overfitting: a subtle trap
We can fit very complicated models to data like earthquake histories, but we are partly fitting noise
In such cases, a more complicated model can give worse predictions
![Page 30: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/30.jpg)
QuickTime™ and a decompressor
are needed to see this picture.
![Page 31: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/31.jpg)
Activity 2.2: Time between large earthquakes from paleoseismic record on
southern San Andreas
What’s the mean time between large earthquakes here? When would you expect the next one?
Is one due soon, overdue, or…
DD 9.8
![Page 32: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/32.jpg)
Time dependent predicts lower until ~2/3 mean recurrence
Results depend on both model choice & assumed mean recurrence
Hebden & Stein, 2008
We don’t know whether to assume that probability of a major earthquake is
- constant with time (time-independent) or
- small after a large earthquake and then increases (time-dependent ).
![Page 33: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/33.jpg)
Activity 2.3 Deep uncertainty in earthquake recurrence
Imagine an urn containing e balls labeled "E" for earthquake, and n balls labeled "N" for no earthquake.
We can draw balls in two ways.
![Page 34: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/34.jpg)
Activity 2.3 Deep uncertainty in earthquake recurrence
Option 1: after drawing a ball, we replace it. In successive draws, the probability of an event is constant or time-independent.
Because one event happening does not change the probability of another happening, an event is never overdue.
![Page 35: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/35.jpg)
Activity 2.3 Deep uncertainty in earthquake recurrence
Option 2: We can add a number a of E-balls after a draw when an event does not occur, and remove r E-balls when an event occurs. This makes the probability of an event increase with time until one happens, after which it decreases and then grows again. Events are not independent, because one happening changes the probability of another.
![Page 36: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/36.jpg)
Activity 2.3 Deep uncertainty in earthquake recurrence
Problem: Given a sequence of results, it’s hard or impossible to tell how the urn was sampled. Thus it’s hard to assess the probability of an “earthquake” in the next draw.
![Page 37: 2. Shallow versus deep uncertainties Why many predictions / forecasts fail.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649ec55503460f94bcf7f2/html5/thumbnails/37.jpg)
Italian flag graphic - one way to illustrate uncertainty we can’t
quantify well
50% 30% 20%