Predictability and Prediction of Social

ProcessesRich Colbaugh*† Kristin Glass*

*New Mexico Institute of Mining and Technology†Sandia National Laboratories

April 2007

Introduction

Objective

Develop formal analytics capability for important social processes, with emphasis on predictive analysis. Sample tasks of interest include: • assessing predictability of

given social process;

• recommending observables upon which to base prediction;

• forming useful predictions even when using noisy/incomplete data. Challenge

Social decision-making is critical aspect of many social processes and is notoriously difficult to predict.

Foundations

Low cognition agent models

Multi-scale representation of social processes via S-HDS framework:

• micro-scale – simple models for individual agent dynamics;

• meso-scale – collective dynamics within social context;

• macro-scale – aggregation of collective dynamics across contexts. stochastic

process

continuoussystem

inputs

inputs

discrete statecontinuous

state

stochasticprocess

continuoussystem

inputs

inputs

discrete statecontinuous

state

Foundations

Predictability assessment

Basic questions

Given a social process and prediction question of interest:

• Is process/question pair predictable? (Does any combination of available knowledge/observations regarding the social process enable desired prediction?)

• Which observables are most useful for making predictions? Can predictions be formed using noisy, incomplete versions of these data?

• If problem is unpredictable, can it be refined to one which is predictable?

Foundations

Predictability assessment (cont’d)

We have developed rigorous, computationally tractable methods for evaluating predictability of a given social process/question pair.

Approach:

• basic idea – assess reachability properties of process of interest and determine if properties are in conflict with prediction goals;

• example – if A, B are both reachable from indistinguishable configurations then process is unpredictable;

• method – one-dimensional abstraction of social process models enables computationally tractable, provably correct reachability assessments without simulations.

B

A

IC

One-dimensional abstraction

Example One: nondeterministic systems.

• System nd: dx/dt = f(x,d), where xX and d(t) D is “disturbance”.

• Theorem: No trajectory from X0 X reaches Xu X if B(x) s.t.

▫ B(x) 0 x X0;

▫ B(x) 0 x Xu;

▫ (B/x) f(x,d) 0 x X, d D.

• Computation: convex relaxation of theorem criteria via SOS decomposition [Parrilo 2000] and semidefinite programming (SDP); for example

relax {B(x) 0 x X0} to {− B(x) − 0T(x) h0(x) = SOS

with 0(x) = SOS}.

Foundations

One-dimensional abstraction (cont’d)

Example Two: stochastic systems.

• System s: dx = f(x) dt + g(x) dw where w(t) is a Wiener process.

• Theorem: is upper bound on probability of reaching Xu X from X0 X while remaining in X if B(x) s.t.

▫ B(x) x X0;

▫ B(x) 1 x Xu;

▫ B(x) 0 x X;

▫ (B/x) f + (1/2) tr [gT (2B/2x) g] 0 x X.

• Computation: convex relaxation of theorem criteria via SOS and SDP.

Foundations

Case studies

On-line markets

Objective: study social process “paradox” – outcomes of social processes are often both unequal and unpredictable – and explore the possibility of prediction in on-line markets.

Empirical data: music and soft-ware markets.

Case studies

On-line markets (cont’d)

Summary of software market study (for description of music market see [Watts et al. 2006]):

• data source: CNET software library (www.download.com) consisting of >30,000 programs with associated news/reviews/prices/technical information, one month data collection period;

• main findings:

▫ daily download market share of item is (statistically significantly) positively correlated with cumulative item downloads, negatively correlated with item age, and not (statistically) affected by other information (e.g., expert reviews, user reviews, technical data);

▫ average quality of “most popular” software is not distinguishable from average quality of all software available on site.

Case studies

On-line markets (cont’d)

Model: “low cognition” agent model in which each agent selects option based upon evaluation of option quality and consideration of choices of other agents.

Predictability assessment: formal analysis of predictability of market share winners/losers

• shows prediction is feasible in both low social influence (SI) and high SI cases;

• identifies observable for which prediction is feasible – limited, very early market share time series.

Predictability assessment

unpredictability at t = 0: 0.63 unpredictability at t = 10: ~10−3

(both for high SI)

unpredictability w/ high SI: 0.63unpredictability w/ low SI: ~10−3

(both for t = 0)

Case studies

On-line markets (cont’d)

Prediction: sample results

• Simulated music market

1. Estimate SI level (low or high) within “multiple models” framework using convergence rate of market share variance.

2. Predict market share using algorithm appropriate for SI level. • Experimental music market: scheme appropriately

classifies market share trajectories as corresponding to low or high SI and enables useful market share predictions in each case.

Movie revenue predictability/prediction

Objective: explore possibility of predicting total box office revenue for a given movie.

Empirical data:

• movie industry – weekly box office receipts, production budget and marketing expenses; personnel data;

• on-line ratings – critic reviews, daily user reviews;

for 40 movies. Studio view: “Nobody knows anything” – screenwriter William

Goldman.

Case studies

Movie revenue predictability/prediction (cont’d)

Sample predictability results

• Classical “input-output” prediction: movie revenues are power law distributed with infinite variance, so classical revenue forecasts have zero precision.

• Dynamics-based prediction: formal analysis of predictability of total box office revenue suggests

▫ prediction is feasible only if some form of time series data is available;

▫ useful predictions may be obtainable using only on-line user reviews (as proxy for “buzz”).

Predictability assessment

unpredictability at t = 0: 0.83 unpredictability at t = 10: ~10−2

(both for high SI)

Case studies

Movie revenue predictability/prediction (cont’d)

Sample prediction results

• Model estimation: use “training” data (for 20 movies) to develop formula that estimates movie appeal and buzz from early on-line reviewer data.

• Algorithm:

▫ estimate {appeal, buzz} for movie from very early (e.g., first week) time series using formula obtained above;

▫ predict total box office revenue for movie by evolving low cognition movie attendance model to equilibrium.

Case studies