Explain emergence of structure in the World Wide Web

Aggregation and competition under informational increasing returns

Presentation by: Petros KavassalisATLANTIS Group, University of Crete &ICS_FORTH, Greece

Contact at: petros@itc.mit.edu

2 BT/January 2003

together with:

Stelios LELIS, ATLANTIS Group, Univ. of Crete, Greece

Charis LINA, ATLANTIS Group, Univ. of Crete, Greece

Manolis PETRAKIS, Dpt of Economics & ATLANTIS Group, Univ. of Crete, Greece

Jakka SAIRAMESH, IBM IAC, USA

Presentation at BT meeting: M. Vavalis, iCities Project Manager

3 BT/January 2003

agenda

A Web Simulated Economy (WSE)…

…To explain agglomeration and fast growth in the Web

Network approach to “Web’s Hidden Order”

Urban explanations of the web sites’ fast growth and differentiated competition

4 BT/January 2003

• Economic frameworks• Bounded rationality• User heterogeneous preferences • Sites with differentiated offerings• Info propagation networks• Sites linked hierarchically• Network externalities

iCities project funded by FET WSE

Design of iCities ?BehaviorLanguage

• Modeling experience• Analysis of existing information cities

• Speed Data-strucuture design Parallel/distributed execution

• Scalability• Configurability (programability)

Multiple models Component-based Data structures/interfaces

• Conceptual framework• Behavioral rules

InternetBehavioral Models

iCitiesproject

SimulationFramework

Economic Geography&

Case studies

5 BT/January 2003

A Web Simulated Economy (iCities WSE) On top of Mozart/Oz (SICS): rigorous simulation environment Capturing essential characteristics of the real web economy:

agglomeration & scale-free state in distribution of population across web sites

Capable to provide insight on empirical regularities: result of the joint action of superposed networks

Able to explain web organization and progressive, fast, web formation: reveal patterns of Internet population clustering into web locations

Reference: New Economic GeographyAgglomeration in the real worldIncreasing returnsP. Krugman, B. Arthur

6 BT/January 2003

What the EconGeo has to say to the Web? P. Krugman, The Self-organizing Economy The geographical space reveals different forms of concentration of population and economic activity. These are not only the result of inherent differences between locations but also of some set of cumulative processes, necessarily involving some forms of increasing returns, whereby concentration can be self-reinforcing.

B. Arthur, Increasing Returns and Path Dependence in the Economy Increasing returns are the tendency for that which is ahead to get further ahead, for that which loses advantage to further lose advantage. They are mechanisms of increasing returns that operate to reinforce that which gains success or aggravate that which suffers loss.

7 BT/January 2003

Towards an economic geography of the Web

H1: Heterogeneous populations of agents

H2: Network structures matter

H3: There are Informational Increasing Returns

8 BT/January 2003

H1: An economy with two populations... Internet Users with partial

information

Web Sites with performance varying over the course

9 BT/January 2003

H2: Decision embedded in nets of interaction

S o c i a l n e t w o r k sS o c i a l n e t w o r k s

L i n k a g e s ( i n c l u d i n gL i n k a g e s ( i n c l u d i n gn a v i g a t i o n h i e r a r c h i e s )n a v i g a t i o n h i e r a r c h i e s )

U n i t s o f a c t i o nU n i t s o f a c t i o np r e f e r e n c e s

i n c r e a s i n g r e t u r n s

i n c r e a s i n g r e t u r n s

Word-of-mouth network or network externalities

Underlying networkPortfolio of sites

10 BT/January 2003

H3: Informational Increasing Returns

Networks carry increasing returns

Word-of-mouth information propagation network (social network with local ties and long distance relationships)

Underlying network linking sites (navigation is hierarchical, produces “linkages”)

Amazon.com-like network externalities (agglomeration benefit)

11 BT/January 2003

The issue: explain power law regularity

A Web Simulated Economy (WSE)…

…To explain agglomeration and fast growth in the Web

Network approach to “Web’s Hidden Order”

Urban explanations of the web sites’ fast growth and differentiated competition

12 BT/January 2003

Huberman’s diagnostic: Web Hidden Order!

The distribution of Internet users per web site follows a universal power law

A power law distribution is a straight line on a log-log scale

Xerox Internet Ecologies ProjectAOL Data,

Prop

ortio

n of

site

sNumber of users

Xerox Internet Ecologies ProjectAOL Data,

13 BT/January 2003

We have reproduced it!

% users volume

%sites

all sitesall sites Our resultsOur results

all sitesall sitesXerox resultsXerox results

0.1 9.28 32.36

1 56.79 55.63

5 85.27 74.81

10 92.77 82.26

50 98.96 94.92

14 BT/January 2003

j

web topology

social network

portfolio of web infohabitant i

inhabitants of the web location j

i

infohabitant, c, lp, uv, d, , s

web locations, ri

Why is this important?

We provide a network-base explanation for the power law regularity!

Internet consumers:Surf the webLearn about web sites by asking other people (word-of-mouth) or by surfing from one site to another along hyperlinksVisit these sites, evaluate and include them in a portfolio of FVS (U = performance + e)Have loyal behavior

Web sites

15 BT/January 2003

What does this imply?

A network approach to the power law issue:

Previous attempts: “random growth” models (from Simon to… Huberman)Question: Where does such a growth come from?Direction: Krugman sees in percolation models, one possible way around the problems with “random growth” modelsWe took that way: online concentration should be the result of a process involving random transport networks

Word-of-mouth information diffuses over a social network structure linking Internet users

Sites link network transport users from one site to another (navigational hierarchies)

16 BT/January 2003

In a nutshell…

Networks carry increasing returns

INFORMATIONAL INCREASING RETURNS

Word-of-mouth network Sites link network

Small world assumptionWatt-Storgatz (WS) beta model with new nodes entering the game

Short path lengthLarge clustering coefficient

1. Small world (WS model)2. Scale free network (Barabasi)

Directed linksNew nodes enter the gameRewiring of existing linksPreferential attachment

17 BT/January 2003

Small world-Small World: findings (I)

Scatter plot: Size versus Age

Scatter plot: Size versus Performance

18 BT/January 2003

Small world-Small World: findings (II)

Evolution of growth rate for site ranked at position 125

Evolution of growth rate for site ranked at position 1

19 BT/January 2003

Small world-Small World: findings (III)

Sites succeeding to be ranked at the higher positions belong to “neighborhoods” of highly visited sites

20 BT/January 2003

Small world-Small World: findings (IV)Word of mouth (Centripetal) Exploration (Centrifugal)

Users loyalty (Centrifugal) Clustering coefficient (Centrifugal)

μ : power law exponent γ : proportion of sites that are visited at least by one user at final timestep

21 BT/January 2003

Small world-Scale free: findings (I)

Most findings are confirmed (slope: 1.4)

22 BT/January 2003

Small world-Scale free: findings (II)

Scatter plot: Size versus Performance and In-degree

23 BT/January 2003

Small world-Scale free-Investments

Sites performance varies over timeSites decide to make investments in predefined time intervals, to improve their performance (affront clutter costs)Accumulated investments depreciate over timeInvestments are made on the basis of

Growth rate Market share (for established sites)

Investments produce a performance increment with a certain probability (there are attention costs)Entry strategies suppose an investment to obtain a good performance and a number of in-links

Out- links are also growing over timeAlgorithm for out-links growth

24 BT/January 2003

Small world-Scale free-Investments: findings (I)

A power law distribution in sites sizes is again obtained (in general and within categories)

25 BT/January 2003

Small world-Scale free-Investments: findings (II)

Sites’ growth rates fluctuate between time intervals in an uncorrelated fashion but about a positive mean value

This is evident in Huberman-Adamic’s data and they use it as an assumption to build their model

Right picture: Fractional fluctuations in the number of users of site ranked at position 60.

26 BT/January 2003

Small world-Scale free-Investments: findings (III)

Web sites’ age and popularity are slightly correlated

This is evident in Huberman-Adamic’s data.

Right picture: Scatter plot of the number of unique visitors versus age.

27 BT/January 2003

Small world-Scale free-Investments: findings (IV)

In- and out-degree distribution of sites follow power-laws.

In-degree distribution Out-degree distribution

28 BT/January 2003

Small world-Scale free-Investments: findings (V)

Slight correlation between the age of sites and their number of in-coming links.

This is evident in Huberman-Adamic’s data.

Right picture: Scatter plot of the number of incoming links versus age.

29 BT/January 2003

Small world-Scale free-Investments: findings (VI)

Again:Relative performance is awarded more than absolute performanceA number of late entrants may survive and prosper (our model spans over Huberman and Barabasi’s models)

But:As economic variables enter directly the model, they are able to break down the power law stabilityOr, a power law distribution survives as long as new sites enter regularly the game (our assumption: exponential entry rate)Then? Instability? What kind of instability?

30 BT/January 2003

The issue: provide directly economic explanations

A Web Simulated Economy (WSE)…

…To explain agglomeration and fast growth in the Web

Network approach to “Web’s Hidden Order”

Urban explanations of the web sites’ fast growth and differentiated competition

31 BT/January 2003

An info-economy for experience goods

externality portfolio of user iUser i

web topologyUsers of the

web location j

• Performance• Vector of products

jSearch engine

Web location j

• Vector of preferences

32 BT/January 2003

Internet users

Have preferences over content/service categories (e.g. Books, Internet communication) and versions (generic/scientific, e-mail/instant messaging/chat rooms etc)

Have a portfolio of frequently visited sites Find new sites to visit through: Search Engine. Users periodically submit queries related to their

preferences to a search engine Exploration. Users surf from one site to another following the links

of sites network Evaluate new sites and include in their portfolio the sites with the

highest utility Users are loyal to their portfolio sites/They include a new site in

their portfolio after number λ visits to that site (stickiness) Users’ utility function depends on

Site performanceMatching of user preferences and site offeringsAgglomeration benefit

33 BT/January 2003

Web sites

Offer a vector of product versions on specific content/service categories

Have a dynamic performance characteristic , rj, that determines their performance in practice.

Periodically make investment to ameliorate their performance May offer services that provide an additional benefit

(“agglomeration” benefit/AB) to their visitors:When agents make choices about web sites, they receive a payoff depending on the number of agents having already visited that site at the time of choice

n versions in 1 category + AB

n versions in m categories

Configuration with 3 types of sites

[1…n] versions in 2 categories + AB with some probability

Highly Differentiated

Specialized

Partially Differentiated

34 BT/January 2003

Model ingredients

Investment StrategyConservative Aggressive

Entry strategyInitial investmentsStrategic use of “in-links” opportunities Strategic use of Search Engines’ promotion opportunities

Continuously updated Sites link networkSites implement a “where to link” strategy (according to categorial relatedness and popularity)Random update

Growing number of out- links

35 BT/January 2003

Principal formal elements

)()()( tMtVtms jj

iiiikjCiCijiijij

Vij tΑtU

1

2211 )...)(()( 2

jcic | x- x| iciccij TS

2/)1(*)/1(*1log* c

cmatchijjj

Aij svnntmstAtU

c

acj

a

jjuncerj tmstmstInvPtA/)1(

1log)'(log*11

User i’s utility from site j (versions’ matching benefit)

User i’s utility from site j (agglomeration benefit)

Site j’s market reach

Site j’s performance

36 BT/January 2003

Results (I)

Evidence of concentration New entrants can enter top ranks

37 BT/January 2003

Results (II)

Fast growth pattern is due to various networks that are present (mostly to the sites link network) and depends also on how search engines are doing their work

Coexistence of Highly Diversified, Partially Diversified & Specialized Sites

The Agglomeration Benefit introduces interesting criticalities

Early entry seems to be related with a higher probability of success (however, late entrants can survive and prosper)

Strategic investment produces instability

Speculation: Instability would evolve to a “cable TV”-like industrial organization model?