Modelling in an imperfect world
-
Upload
luis-willumsen -
Category
Documents
-
view
187 -
download
1
description
Transcript of Modelling in an imperfect world
MODELLING IN AN IMPERFECT WORLD
Luis Willumsen
Planning and policy advice with less-than-rational human beings
CO
NC
ERN
S
Some old concerns
Our track record is not brilliant
J P Morgan found in 14 Case Studies (USA) 2 underestimates (10 to 30% below actual) 4 moderate overestimates (12 to 25% over actual) 8 ‘blue sky’ overestimates (45 to 75% over actual)
• Models are a simplification of reality based on some useful theoretical assumptions and sufficient data to estimate them
• Models are valid insofar the theoretical assumptions remain reasonable; sadly, our theoretical assumptions do not represent human behaviour well
• Therefore, model results are worth little without interpretation and judgement
• So, what do we know about human beings and choices that can help us provide better advice?
CO
NTEN
TS
Contents
1. The key underpinnings of transport demand modelling
2. How travellers really make decisions (the Kahneman model)
Two characters: System 1 and System 2
Two selves: Experiencing and Remembering selves
Two species: Homo Economicus and Homo Sapiens
3. Three contexts for forecasting: Policy Advice
Planning
Forecasting demand and revenue
KEY
REQ
UIR
EMEN
TS
The four pillars of good models
• Good future population synthesis
• System equilibrium
• Consistency of future behaviour
• Behavioural choice modelling
Utility functions and choice models
applied at different levels of aggregation
The parameters in the utility functions
and choice structures remain
the same
Accurate allocation of populations and
activities in the future
Appropriate feed-back through all
relevant submodels to ensure consistent
results
j
m
U
j U
m
eP
e
jq jq jqU V
Forecasting
Tastes and preferences are given and stable, exogenous
to our models
KEY
REQ
UIR
EMEN
TS
The four pillars of good models
From this basis we build a picture of the future that enable us to
compare alternative strategies, projects and policies on a like
with like basis
BEH
AV
IOU
RA
LM
OD
ELS
Utility functions
Modelling choices
The ideal traveller
Is Rational, Selfish and its Tastes do not change, ever.
j
m
U
j U
m
eP
e
j j iU V
A "rational" being that considers opportunities and seeks to optimise
his/her utility by careful choices.
REA
LTR
AV
ELLERS
The real life traveller
A partly rational but also emotional and collaborative being that:
• Cares about changes more than absolute values
• Cannot cope with too many options and uses heuristics
• Has diminishing sensitivity to changes in utility
• Is averse to losses
• Does not react immediately
• But what do we know about this real traveller?
• And how do we adapt our modelling and recommendations to him/her?
REA
LTR
AV
ELLERS
The experiencing self
This is the traveller while travelling
Experience a combination of good and bad aspects of travel
The remembering self
This is what the traveller remembers
Usually salient aspects of the journey
The end of the trips and the results are paramount
Subsequent decisions are more influenced by
what is remembered than the actual journey
itself
TW
OA
SPEC
TSO
FH
UM
AN
DEC
ISION
MA
KIN
GSystem 1 thinking
Intuitive, fast, automatic
Uses heuristics, often answering an easy question rather than a difficult one
Sensitive to changes
Assumes that what you see is all there is WYSIATI
Thoughtful, Logical, requires effort
Lazy, first tendency is to endorse System 1
Can interact with S1 and train it
• iPad and cover cost £550
• iPad costs £500 more than cover
• How much is the Cover?
System 2 thinking
HO
MO
ECO
NO
MIC
US
The Rational Human Being, Homo Economicus
There is strong suspicion that it is a convenient assumption but does not correspond to reality
This mismatch may matter less to develop theory but it does affect the forecasts and advice we provide
• We are not truly Utility Maximisers..
• And we cannot consider all our alternatives..
• We are more affected by changes than by absolute values
• And these changes are based on what we remember from previous experiences, for example delay or price
• Evaluation is relative to a neutral reference point (status quo)
• Diminishing sensitivities to change (Compare £100 to £200 and £1900 to £2000)
• Loss aversion; loses are more onerous than the respective gains
Kahneman-Tversky’s Prospect Theory PR
OSP
ECT
TH
EOR
Y
-16.0
-12.0
-8.0
-4.0
0.0
4.0
8.0
12.0
109876543210-1-2-3-4-5-6-7-8-9-10
Perceivedgain/loss
Generalisedcostchange
Perceivedvaluevscostchange
THERE ARE ALSO OTHER PROBLEMS WITH OUR CURRENT MODELS
ELEC
TRO
NIC
PA
YMEN
T
Fussy prices and money
Santiago tags and gantries
A problem with money...
Separating use from payment crates a different type of money
Ignoring the different kinds of money and prices in our models will lead to wrong forecasts (probably underestimations) and poor advice.
LA
GS
INB
EHA
VIO
UR
AL
RESP
ON
SES
Change job or residenceA problem with time..
Our equilibrium models assume all changes happen at the same time
But people cannot instantly change jobs or homes, and not even time of travel.
We need to recognise the lags in behaviour
But we know and understand little about them
Mode or time of travel
PO
SSIBLE
SOLU
TION
S
Some possible improvements
Hierarchical structure of choices is important for some known biases:
One could give much more important weights to certain attributes in the case of elimination by aspects: Time first, etc..
Nested choices
Choice
PT
Bus
W&R B&B
Metro
W&R B&R P&R
CAR
Owned Club
1 1 1 1 1 1 2 2 2 2 2 2(1 ) (1 ) ..
where and are the parameters for a
loss and a gain respectively, and is equal
1 if the change is a loss and zero otherwise
q
i i
i
V a x a x a x a x
a a
In the case of asymmetric elasticities one can develop a special utility function, even non-linear; but this may create problems for convergence
PO
SSIBLE
SOLU
TION
S
Lags in behavioural change
Hierarchical structure of responses is purpose dependent(?)
For JTW HBShopRoute RouteTime of travel DestinationMode Time of travelDestination ModeFrequency Frequency
Model a time horizon with some of these responses frozen and interpolate
Separate responses
But we know too little about these lags
Cross section data collection is poor at capturing these; this includes SP
We need to learn more from time series and from experimentation
DO
ESIT
MA
TTER?
Does all of this matter?
• Not really, we only want to compare Plans/Schemes/Policies on like-with-like basis that we all agree is good enough
• OK, it is not perfect, but after a little while people do change because of an accumulation of minor disruptions
• There is a trade-off between behavioural accuracy and the equilibrium we need to compare schemes; we vote for consistent comparisons
BUT
o Schemes or plans may affect different responses in different ways
o Sometimes it is important to get the sequence of interventions right
o When forecasting for concessions the right timing and the right response are paramount
IMPROVING INTERPRETATION AND JUDGMENT
IMP
LICA
TION
S
So....
Human nature limits the accuracy of our models
There are implications for Research and for an evolving Best Practice
For Research:
• Develop a better understanding of how uncertainty is affected by the level of disaggregation of our models and data
• Identify lags in behavioural change and develop best ways to deal with them (more social psychology and less mathematics and computational efficiency perhaps?)
• Develop a better relationship between objective (generalised cost) change and perceived loss/gain
• Understand how people switch between System 1 and System 2 modes of thinking in the context of travel
OYSTER
+ GP
SNEW DATA SOURCES WILL HELP
Use of mobile phone, bluetooth, smart card and GPS data
• To monitor performance
• To infer trip matrices
• To study experiments
FO
REC
ASTIN
G
General recommendations on forecasting practice
• Our business is not modelling but forecasting
• We need transport models but our existing tools are less reliable than we pretend; we must acknowledge uncertainty and risk from the outset
• We should start experimenting with the careful adaptation and use of existing techniques to account for more realistic behaviour
• Interpretation and judgement, professional responsibility, should be more open and transparent
• Design and undertake experiments whenever possible, to improve and mediate model results; and this is easier now than in the past
• Document experience more openly
PO
LICY
AD
VIC
E
Policy advice
• Not always depending on modelling
• But our experience should be valuable as it would add analytical rigour to policy discussions
• For example, the issue of Fuel Taxation vs. Road User Charges
• Identification of winners and losers will be more central
• Should experiment with the production of psychological impact evaluation in addition to “objective accounts”
• The role of other “difficulties” of payment, information, familiarity, WYSIATY
• Engage in the discussion of implementation, communication, sequencing and timing (remembering and experiencing self, S1 and S2 thinking modes)
PLA
NN
ING
Transport Planning emerging practice
• Use conventional tools but allow for lags in responses; even with assumed lag rates
• This requires models where certain responses can be switched off at will
• Show and discuss the impact of these lags and if critical look for other approaches to settle the choice of plan/scheme
• Identify winners and losers and by how much
• Account separately for large and small loses/gains
• Acknowledge uncertainty and the risk of over-calibration and spurious precision
FO
REC
ASTIN
G
Forecasting Traffic and Revenue
• Our track record is better than that of bankers and regulators
• But it is still not that good
• Acknowledge uncertainty and risk from the outset: identify sources of risk, estimate their importance and focus on reducing them
• Disaggregate for willingness to pay but do not over-complicate the model
• Careful use of existing techniques, even with the limitations shown, is a reasonable approach. But, support forecasts from different complementary perspectives
• For example, a classic model forecast, a trend extrapolation forecast and benchmarking against similar systems
• Undertake risk analysis
RISK
AN
DD
EMA
ND
CO
NTR
IBU
TOR
S
De-construct contributors to traffic: conceptual LRT forecasts
STO
CH
ASTIC
RISK
AN
ALYSIS
Stochastic risk analysis
Beloved by Financial Institutions
Generally based around Base Case
Take 2-3 key variables : GDP, SVT, etc
..and look into their historical variability (standard deviation )
The model is used to track variability in revenue resulting from variability in key inputs, usually via a simplification in Excel
FutureRevenue = RevenueFactor * Base Case Revenue
A value for RevenueFactor of 1 indicates Base Case
Also presented as the level of revenue that is likely to be exceeded 90 or 95% of the time (P90 and P95)
TYP
ICA
LO
UTP
UT
Example of output
0.85
0.9
0.95
1
1.05
1.1
1.15
2003 2005 2007 2009 2011 2013 2015
Year
Re
ve
nu
e F
ac
tor
GDP variation σ: 0.5%
SVT variation σ : 2.5%*mean VST
FO
REC
ASTS
CA
SE2
P90 and P95 for toll road
'Four Roads- Case 2 WRe P90 sd 1.5
2
7
12
17
22
2008 2010 2012 2014 2016 2018 2020 2022 2024 2026 2028 2030 2032 2034 2036
Mx$ B
illio
ns
RO
UN
DU
P
Round up
1. Some of these risk analysis techniques will also filter through into normal transport planning models and practice Especially for key projects like HS2
2. Fundamental research into real travel behaviour and choices is necessary
3. Improvements to current practice that recognises some limitations of our models are possible and desirable
4. Benchmarking and well documented experience elsewhere will be used more often to support forecasts
5. This will be facilitated by new data sources and electronic trails
6. Modellers should engage more with real issues and develop reliable judgement and interpretation skills; this may require adaptation of training programmes
THANK YOU