Staging Abstraction using Chains of Models

Staging Abstraction using Chains of Models

Bruce EdmondsCentre for Policy Modelling

Manchester Metropolitan University

…and…the Problem of Context-Dependency

Talk Outline

• Introduction: a fundamental difficulty!• Problem 1: needing both relevance and

rigour• Possible strategy: using chains of

models to stage abstraction• Problem 2: context-dependency• :Possible strategies: reducing scope and

including context• Concluding discussion: connecting the

worlds of policy and scienceStaging Abstraction using Chains of Models, Bruce Edmonds, CCSA Seminar, York, March 2012. slide-2

The Anti-Anthropocentric Assumption

• That the universe is not arranged for our benefit (as academics)

• e.g. that assumptions like the following are likely to be wrong:– Our planet is the centre of the universe– Planetary orbits are circles– Risky events follow a normal distribution– Humans act as if they followed a simple utility optimisation

algorithm• The one that I am particularly arguing against here is

that our brains happen to have evolved so as to be able to understand models adequate to the phenomena we observe

Staging Abstraction using Chains of Models, Bruce Edmonds, CCSA Seminar, York, March 2012. slide-3

Versions of this assumption• Whilst other animals have severe limitations

and biases in their cognition, we don’t• That our tools (writing, computers etc.)

allow us to escape our limitations and biases to achieve general intelligence

• That simplicity (that which is easier for us to analyse) is any guide to truth

• If your model is not simple enough to analyse and understand, you are: (1) not clever enough, (2) lazy (have not worked hard enough), (3) premature (don’t yet have the tools to crack it) or (4) mistaken


Living with the AAA• Accepting that that much of the world around us is

fundamentally beyond capturing in a model that is both adequate and sufficiently simple and general for us to cope with

• Acknowledging our (brain+tools) biases and limitations and so considering how we might extend our scientific understanding as much as possible

• Phenomena that are simple enough for us to scientifically understand are the exception – the exception to be sought and struggled for

• Simplicity is the exception – a science of non-simple systems makes no more sense than a science of non-red things


Needing both rigour and relevanceProblem 1:


A Dilemma• KISS: Models that are simple enough to understand

and check (rigour) are difficult to directly relate to both macro data and micro evidence (lack of relevance)

• KIDS: Models that capture the critical aspects of social interaction (relevance) will be too complex and slow to understand and thoroughly check (lack of rigour)

• But we need both rigour and relevance• Mature science connects empirical fit and explanation

from micro-level (explanatory and phenomenological models)


KISS vs. KIDS as a search strategy

Simplest Possible

More Complex in Aspect 2

etc.

More Complex in Aspect 1

KISS

KIDS


The Proposed Approach

• Not to use a single model but rather a closely related “chain” of models

• Starting with narrative and statistical evidence for the micro-level behaviour of individuals etc.

• To build models that are more adequate to the processes that are thought to occur

• Which are checked and assessed against as many kinds of evidence as possible (including macro statistical evidence)

• And only later abstract to simpler simulation, analytic and social network models


Social Complexity of Immigration and Diversity

• A 5 year EPSRC-funded project between:• University of Manchester

– Institute for Social Change• Ed Fieldhouse, Nick Shryane, Nick Crossely, Yaojun Li,

Laurence Lessard-Phillips, Huw Vasey– Theoretical Physics Group

• Alan McKane, Tim Rogers• Manchester Metropolitan University

– Centre for Policy Modelling• Bruce Edmonds, Ruth Meyer, Stefano Picassa

• Aim is to apply complexity methods to social issues with policy relevance


The Modelling Approach

Data-Integration Simulation Model

Micro-Evidence Macro-Data

Abstract Simulation Model 1

Abstract Simulation Model 2

SNA Model Analytic Model


An example of the layering of related models in chemistry from 1990

• Adapted from: Gunsteren, W. F; Berendsen, H. J. C. (1990) Computational Simulation of Molecular Dynamics: Methodology, Applications and Perspectives in Chemistry. Angewandte Chemie - International Edition in English, 29:992-1023.


Roles of each kind of model

• Each is constrained by those “beneath” them, i.e. are consistent with them

• What each component should clearly represent something

• Models “above” analyse, check and explain what is happening in those below

• Models immediately “below” can be used to explore the safety of assumptions

• It might well happen that simpler, more abstract models have validity (w.r.t. a lower model) only under some settings


But why not just jump straight to simple models?• There are many possible models and you don’t know

why to choose one rather than another, this method provides the underlying reasons

• Much social behaviour is context-specific, and this approach allows one to check whether a particular simple model holds when background features/assumptions change

• The chain of reference to the evidence is explicit, allowing one to trace their effect and possibly better criticise/improve the model

• This approach facilitates the mapping onto qualitative stories/evidence


Data Integration Models

• Intended more as a computational description of a particular case than a theory (at least a general theory)

• Its aim is to represent as much of the relevant evidence as possible in one coherent and dynamic simulation

• Provides a precise target for abstraction (which are then checkable against it)

• Stages abstraction from data to theory• Separates representation and abstraction• Preserves chains of reference


Aims and Objectives of DIM

• To develop a simulation that integrates as much as possible of the relevant available evidence, both qualitative and statistical (a Data-Integration Model – a DIM)

• Regardless of how complex this makes it• A description of a specified kind of situation (not

a general theory) that represents the evidence in a single, consistent and dynamic simulation

• This simulation is then a fixed and formal target for later analysis and abstraction


DIM Development Method• A relatively tight interactive “loop” between the social scientists

who are experts in the subject matter and their data and the simulation developers...

• ...trying to give as much ownership and control to social scientists as possible.

• First target: What makes people vote (within the context of a diverse community)?

• Started with developing a fairly complete list of “causal stories” concerning the various processes that might contribute from

• Then initial model iteratively developed in NetLogo to enable maximum responsiveness and transparency

• To be reimplemented in Java/Repast when the target becomes more “settled” for more extensive simulation exploration and analysis


An overview of model structureUnderlying Data from Surveys about Population Composition etc.

Demographics of people in households (both native and immigrant)

Homophily effects the social network and membership of organisations etc.

Social network effects how individuals influence each other, reinforcing and/or changing existing norms/opinions

This effect the behaviours of individuals, which can then be extracted from the simulation as model results and compared with evidence etc.


Demonstration Run

Parametersand

Controls

Pseudo-narrative log of eventshappening to a single agent

SimpleStatistics

concerningOutcomes

Pictureof World

Indicative

Graphsand

Histograms


Example Output – Turnout


Example Output – one agent1945: (person 712) did not vote1946: (person 712) started at (workplace 31)1947: (person 712)(aged 29) moved from (patch 4 2) to (patch 5 3) due to moving to an empty home1947: (person 712) partners with (person 698) at (patch 5 3)1950: (person 712) did not vote1951: (person 712) separates from (person 698) at (patch 5 3)1951: (person 712)(aged 33) moved from (patch 5 3) to (patch 4 2) due to moving back to last household after separation1951: (person 712) did not vote1952: (person 712) partners with (person 189) at (patch 4 2)1954: (person 712)(aged 36) moved from (patch 4 2) to (patch 23 15) due to moving to an empty home1955: (person 712) did not vote1964: (person 712) started at (activity2-place 71)1964: (person 712) voted for the red party1966: (person 712) voted for the red party1970: (person 712) voted for the red party1971: (person 712) started at (workplace 9)1974: (person 712) voted for the red party1979: (person 712) voted for the red party1983: (person 712) died at (patch 23 15)


Social Network at 1950


On-going research and issues

• Suggests that this approach might be able to include and integrate qualitative evidence alongside quantitative evidence, but the method to do this is not well developed

• Creating and maintaining chains of models takes a LOT of time, resources etc.

• Does allow a more principled abstraction to physics type models, since at least some of the assumptions can be tested

• New trade-offs will no doubt be revealed!Staging Abstraction using Chains of Models, Bruce Edmonds, CCSA Seminar, York, March 2012. slide-25

Modelling and Context-DependencyProblem 2:


Another modelling trade-off

• Some desiderata for models: validity, formality, simplicity and generality

• these are difficult to obtain simultaneously (for complex systems)

• there is some sort of complicated trade-off between them (for each modelling exercise)

simplicity

generality

validity

formality


Another Dilemma!

• In order to meaningfully model, communicate or apply knowledge it has to be valid in more than one specific situation

• Yet since the AAA rules out accessible models that have general applicability…

• …we are stuck with models that seem valid and are comprehensible only in specific contexts

• One response is to make fairly simple models that give the perception that they have considerable generality, but in fact are only useful as elaborate analogies


The (direct) modelling relation

Object Systemknown unknown

Modelinput

(parameters, initial conditions etc.)

output(results)

encoding(measurement)

decoding(interpretation)


Analogies• Analogies are usually verbal, but can also be formal

(equations, simulations, etc.)• Their mapping to what is being considered is built “on

the fly” for each situation• Analogies seem to be very basic to the way humans

think and communicate• Their mapping to the situation is different for each

context and each person (in contrast to a model where the mapping is defined)

• This is done automatically and largely unconsciously• This gives the illusion of generality


Modelling a concept of something

Object System

conceptual model

Model


This part of the talk argues for the following strategy: weakening the generality of our formal models to achieve

more validity in the face of the AAA… or, to put it another way, against the following strategy: weakening validity (e.g. to analogy) to preserve

(the illusion of) generality

What is Essential to (empirical) Science?

• Validity: agreement of models to what we observe (the evidence), not science otherwise

• Formality: formal models (maths, simulation) are precise and replicable – essential to being able to build knowledge within a community of researchers

• Simplicity: ability to analyse/understand our models, good to have but unattainable in general (AAA)

• Generality: the extent of the applicability/scope of a single model, there needs to be some small generality to apply models in places other than where developed, but wide generality not necessary


Context

• “Context” is used in many different senses across different fields

• The senses and concepts herein come from discussions and papers presented at the international series of conferences on “Modelling and Using Context”

• Somewhat of a “dustbin” concept resorted to when more immediate explanations fail

• Problematic to talk about, as it is not obvious that “contexts” are identifiably distinct


Situational Context• The situation in which an event takes place• This is indefinitely extensive, it could include

anything relevant or coincident• The time and place specify it, but relevant

details might well not be retrievable • It is almost universal to abstract to what is

relevant about these to a recognised type when communicating about this

• Thus the question “What was the context?” often effectively means “What about the situation do I need to know to understand?


Linguistic Context• This is the set of all language that precede or surround

a focus utterance or phrase (the linguistic subset of the situational context)

• E.g. what pronouns might refer to• Historically the last resort of the linguist when trying to

pin down meaning• Now thought central to natural language production

and understanding• Can be extensive, relying on distant texts or linguistic

norms learnt previously• Sometimes includes common knowledge needed to

distinguish meaningStaging Abstraction using Chains of Models, Bruce Edmonds, CCSA Seminar, York, March 2012. slide-35

Cognitive Context (CC)• Many aspects of human cognition are context-

dependent, including: memory, visual perception, choice making, reasoning, emotion, and language

• The brain somehow deals with situational context effectively, abstracting kinds of situations so relevant information can be easily and preferentially accessed

• The relevant correlate of the situational context will be called the cognitive context

• It is not known how the brain does this, and probably does this in a rich and complex way that might prevent easy labeling of contexts


The Context Heuristic• Divide the world into different sorts of situation

(hereafter simply called a context)• Learn/recognise these in a rich and “fuzzy”

manner• “Crisp” knowledge is “packaged” “within” such

for reasoning, update etc.• Makes within-context reasoning, models,

update etc. more feasible• Whilst each model has limited scope, together

they might cover more ground, albeit in a more “patchy” manner


About Context-Dependency

• Context-dependency is not relativity since contexts can be reliably recognised (and/or corrected if wrongly recognised)

• But since it might be recognised in a “fuzzy” and unconscious manner the bounds of the context may not be reifiable in crisp terms

• This is a heuristic – a strategy that may help push forward the boundaries of formal empirical science

• There is some evidence that our cognition is context-dependent in many ways which means that to a considerable extent it may be unavoidable


Why might the world we study be usefully split into such “contexts”• In some domains, e.g. ecology or social

science contexts might be co-developed over time between the entities (e.g. a niche, or social context like a lecture)

• In some others it may be the only practical way to proceed, as argued above

• In yet others our cognitive, unconscious tendency to deal with the world in terms of contexts might lead us to try and divide the world along less useful lines


Context and Causality• In almost all situations (and all social situations)

there are an unlimited number of things that could be attributed as a cause

• Related to “Causal Spread” (Wheeler); “Wild Disjunction” (Fodor); and “Embeddedness” (Granovetter)

• Without a limitation as to the scope causation makes no sense

• However given a context there are many factors that can be assumed to be insignificantly relevant and/or constant

• Thus causality makes sense given a context, since it excludes most possibilities


Attending to Context• Given that attending to context is not unscientific and is

inevitable (I argue)• Then rather than pretending to generality by using

models as analogies (only)…• …I suggest attending to and incorporating context-

dependency in our investigations (as far as this is possible)…

• …and hence pushing the “envelope” of science a further in the face of complexity

• Fortunately, computers allow us to keep track of a complication of multiple contexts and avoid premature generalisation (we no longer have to weaken validity to get formality)


Kaneko (1990)

• Exhibited a system of parallel chaotic but weakly coupled processes

• Each process seems chaotic and independent• But as system size increases, variance as a

proportion of size does not disappear• Law of large numbers does not apply

Size

Variance(scaled by size)

Globally coupled

Model with random noise


An Illustration of Masked Context-Dependency

Global models are simply uninformative when the phenomena is context-dependent


Cleveland Heart Disease Data Set – the processed sub-set usedIn processed sub-set:• 281 entries• 14 numeric or numerically coded attributes• Attribute 14 is the outcome (0, 1, 2, 3, 4)• Some attributes: age, sex, resting blood

pressure (trestpbs), cholesterol (chol), fasting blood sugar (fbs), maximum heart rate (thalach), number of major vessels (0-3) colored by flourosopy (ca)

• From the Machine Learning Repository Staging Abstraction using Chains of Models, Bruce Edmonds, CCSA Seminar, York, March 2012. slide-44

General Correlations (1% Sig)


Fitting a Global Model (R=56%)

Num = -0.01*age + 0.17*sex + 0.20*cp + 0.00*trestbps + 0.10*restecg + -0.01*thalach + 0.23*exang + 0.18*oldpeak + 0.16*slope + 0.43*ca + 0.14*thal + -0.60 (+/- 0.83)


Looking for Clusters in HD Data Set (Start of Process)


After Solutions Locally Evolve

Speciation of SolutionsIn some areas no solution dominatesSome Solutions Spread over area of applicability


Final Set of Clustered Solutions• Final solution set

after some time.• Still complex but

some structure is revealed

• Note presence of “fbs” despite not being globally correlated and that “chol” helped define the context space


About this approach

• Difficult to quantify the extent to which one is “cherry-picking” (overfitting) models

• This only looks at one aspect of the data – one predicted variable

• A more meaningful correlate of our cognitive contexts would cluster situations as to many different aspects

• But could inform the specification of individual-based models, encoding different behaviours for each detected cluster


A useful context is one that:– includes related models with different

goals/predictions but similar scope

Clusters of Model Scopes suggest a Context

M1 M2

M1

suggests a context


Conclusions• For several reasons context-dependency is

unavoidable• Accepting and thinking about context-dependency

need not have lead to sloppy, relativistic or bad science

• In fact it is essential, since simple systems are the special case

• It may lead to pushing the boundaries of science forward and avoiding some pitfalls

• Science being imperfect or incomplete does not invalidate its project

• We may end up with a “patch work” of locally coherent models of a lot of different kinds – physics may have to be more like zoology


From this…

Physics

Chemistry

Biology

Psychology

EcologySocial Sciences

Geography

Zoology

Reduction

Infe

renc

e


…to this!

Islands of Local Consistency

Clusters of Related Models

Weaker Modelling Relations


This raises some questions…

• Does this make us all relativists?• Does this mean that scientific knowledge is

just the same as other kinds of belief?• Does this mean that we should abandon

formal models?• Does this mean that we cannot attain useful

understanding of complex systems?• Does this mean that interdisciplinary

science is hopeless?


… to which I answer “No”• The picture I paint already represents the reality of

understanding non-simple systems• It just differs from some of the rhetoric of science, and

hence the picture and beliefs many have about science• It does have some consequences for how we do

science • Rather, accepting these realities will help us do better

science by being aware of: – hidden assumptions/context– over-generalisation– reliance on single models– Models as analogies only


Connecting the Policy and Research Worlds

Concluding Discussion:


More modelling trade-offs (different set since simplicity is out)

• Formality matters to researchers – it enables the faithful passing of models to others allowing models to be collectively developed

• What policy people want is very different, they need defensible and explicable policies that have some warrant and work (mostly)

• This is a fundamental conflict

simplicity

generality

validity

formality

micro-validity

macro-validity

AggregateData

SpecificBehavioural

Models AbstractModels

WhatPolicy

AdvisersWant

Squaring the conflict

• Maybe what we should do…• Is simultaneously develop:

1. A “dirty” mainline modelusing global methods, even though pretty näive

2. But also a context-sensitive set of detailed models that are complex but less näive

• Allowing us to deliver:1. A set of “surprise free” predictions but also…2. …A “risk analysis” saying where it will probably

succeed (for targetting of the policy) as well as where and how it might go wrong


Final Conclusions

• An acceptance of complexity• The need for multiple models using multiple

techniques for any one problem• But lacking are well-founded methods for

how to use these well together!• The awareness of context-dependency and

dealing with it appropriately• Whilst developing ways of talking to

stakeholders/policy advisors that meet their needs and let them have some control


The End

Bruce Edmondshttp://bruce.edmonds.name

Centre for Policy Modelling http://cfpm.org

The SCID Projecthttp://scid-project.org

Advertisement!

Staging Abstraction using Chains of Models

Documents

Transcript of Staging Abstraction using Chains of Models