NYC Data Science Meetup: Computational Social Science
-
Upload
jakehofman -
Category
Science
-
view
328 -
download
7
Transcript of NYC Data Science Meetup: Computational Social Science
Computational Social Science
Jake Hofman
Microsoft Research
November 6, 2014
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 1 / 62
MSR NYC
http://research.microsoft.com/en-us/labs/newyork/
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 2 / 62
Questions
Many long-standing questions in the social sciences are notoriouslydiοΏ½cult to answer, e.g.:
β’ βWho says what to whom in what channel with what eβ΅ectβ?(Laswell, 1948)
β’ How do ideas and technology spread through cultures?(Rogers, 1962)
β’ How do new forms of communication aβ΅ect society?(Singer, 1970)
β’ . . .
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 3 / 62
Conventional methods
Typically diοΏ½cult to observe the relevant information viaconventional methods
(Katz & Lazarsfeld, 1955)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 4 / 62
Large-scale data
Recently available electronic data provide an unprecedentedopportunity to address these questions at scale
Demographic Behavioral Network
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 5 / 62
Computational social science
An emerging discipline at the intersection of the social sciences,statistics, and computer science
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 6 / 62
Computational social science
An emerging discipline at the intersection of the social sciences,statistics, and computer science
(motivating questions)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 6 / 62
Computational social science
An emerging discipline at the intersection of the social sciences,statistics, and computer science
(fitting large, potentially sparse models)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 6 / 62
Computational social science
An emerging discipline at the intersection of the social sciences,statistics, and computer science
(parallel processing for filtering and aggregating data)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 6 / 62
biogeographic patterns. Their study, too, is
centered on a large database, but in this case it
is entirely of living organisms, the marine
bivalves. Over 28,000 records of bivalve gen-
era and subgenera from 322 locations around
the world have now been compiled by these
authors, giving a global record of some 854
genera and subgenera and 5132 species. No
fossils are included in the database, but
because bivalves have a good fossil record, it is
possible to estimate accurately the age of ori-
gin of almost all extant genera. It is then possi-
ble to plot a backward survivorship curve (8)
for each of the 27 global bivalve provinces (9).
On the basis of these curves, Krug et al. find
that origination rates of marine bivalves in-
creased significantly almost everywhere im-
mediately after the K-Pg mass extinction event.
The highest K-Pg origination rates all occurred
in tropical and warm-temperate regions. A dis-
tinct pulse of bivalve diversification in the early
Cenozoic was concentrated mainly in tropical
and subtropical regions (see the figure).
The steepest part of the global backward
survivorship curve for bivalves lies between 65
and 50 million years ago, pointing to a major
biodiversification event in the Paleogene (65 to
23 million years ago) that is perhaps not yet
captured in Alroy et al.βs database (5, 7). The
jury is still out on what may have caused this
event. But we should not lose sight of the fact
that the steep rise to prominence of many mod-
ern floral and faunal groups in the Cenozoic
may bear no simple relationship to climate or
any other type of environmental change (10, 11).
References
1. G. G. Mittelbach et al., Ecol. Lett. 10, 315 (2007).2. A. Z. Krug, D. Jablonski, J. W. Valentine, Science 323, 767
(2009).3. P. W. Signor, Annu. Rev. Ecol. Syst. 21, 509 (1990).4. R. K. Bambach, Geobios 32, 131 (1999).5. J. Alroy et al., Proc. Natl. Acad. Sci. U.S.A. 98, 6261 (2001).6. A.M. Bush et al., Paleobiology 30, 666 (2004).7. J. Alroy et al., Science 321, 97 (2008).8. M. Foote, in Evolutionary Patterns, J. B. C. Jackson et al.,
Eds. (Univ. of Chicago Press, Chicago, IL, 2001), vol. 245,pp. 245β295.
9. M. D. Spalding et al., Bioscience 57, 573 (2007).10. S. M. Stanley, Paleobiology 33, 1 (2007).11. M. J. Benton, B. C. Emerson, Palaeontology 50, 23 (2007).
10.1126/science.1169410
www.sciencemag.org SCIENCE VOL 323 6 FEBRUARY 2009 721
PERSPECTIVES
We live life in the network. We check
our e-mails regularly, make mobile
phone calls from almost any loca-
tion, swipe transit cards to use public trans-
portation, and make purchases with credit
cards. Our movements in public places may be
captured by video cameras, and our medical
records stored as digital files. We may post blog
entries accessible to anyone, or maintain friend-
ships through online social networks. Each of
these transactions leaves digital traces that can
be compiled into comprehensive pictures of
both individual and group behavior, with the
potential to transform our understanding of our
lives, organizations, and societies.
The capacity to collect and analyze massive
amounts of data has transformed such fields as
biology and physics. But the emergence of a
data-driven βcomputational social scienceβ has
been much slower. Leading journals in eco-
nomics, sociology, and political science show
little evidence of this field. But computational
social science is occurringβin Internet compa-
nies such as Google and Yahoo, and in govern-
ment agencies such as the U.S. National Secur-
ity Agency. Computational social science could
become the exclusive domain of private com-
panies and government agencies. Alternatively,
there might emerge a privileged set of aca-
demic researchers presiding over private data
from which they produce papers that cannot be
critiqued or replicated. Neither scenario will
serve the long-term public interest of accumu-
lating, verifying, and disseminating knowledge.
What value might a computational social
scienceβbased in an open academic environ-
mentβoffer society, by enhancing understand-
ing of individuals and collectives? What are the
A field is emerging that leverages the
capacity to collect and analyze data at a
scale that may reveal patterns of individual
and group behaviors.
Computational Social Science
David Lazer,
1
Alex Pentland,
2
Lada Adamic,
3
Sinan Aral,
2,4
Albert-LΓ‘szlΓ³ BarabΓ‘si,
5
Devon Brewer,
6
Nicholas Christakis,
1
Noshir Contractor,
7
James Fowler,
8
Myron Gutmann,
3
Tony Jebara,
9
Gary King,
1
Michael Macy,
10
Deb Roy,
2
Marshall Van Alstyne
2,11
SOCIAL SCIENCE
1Harvard University, Cambridge, MA, USA. 2MassachusettsInstitute of Technology, Cambridge, MA, USA. 3Universityof Michigan, Ann Arbor, MI, USA. 4New York University,New York, NY, USA. 5Northeastern University, Boston, MA,USA. 6Interdisciplinary Scientific Research, Seattle, WA,USA. 7Northwestern University, Evanston, IL, USA.8University of CaliforniaβSan Diego, La Jolla, CA, USA.9Columbia University, New York, NY, USA 10CornellUniversity, Ithaca, NY, USA. 11Boston University, Boston,MA, USA. E-mail: [email protected]. Completeaffiliations are listed in the supporting online material.
Data from the blogosphere. Shown is a link structure within a community of political blogs (from 2004),where red nodes indicate conservative blogs, and blue liberal. Orange links go from liberal to conservative,and purple ones from conservative to liberal. The size of each blog reflects the number of other blogs thatlink to it. [Reproduced from (8) with permission from the Association for Computing Machinery]
Published by AAAS
β... a computational social science is emerging that
leverages the capacity to collect and analyze data with an
unprecedented breadth and depth and scale ...β
http://sciencemag.org/content/323/5915/721
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 7 / 62
biogeographic patterns. Their study, too, is
centered on a large database, but in this case it
is entirely of living organisms, the marine
bivalves. Over 28,000 records of bivalve gen-
era and subgenera from 322 locations around
the world have now been compiled by these
authors, giving a global record of some 854
genera and subgenera and 5132 species. No
fossils are included in the database, but
because bivalves have a good fossil record, it is
possible to estimate accurately the age of ori-
gin of almost all extant genera. It is then possi-
ble to plot a backward survivorship curve (8)
for each of the 27 global bivalve provinces (9).
On the basis of these curves, Krug et al. find
that origination rates of marine bivalves in-
creased significantly almost everywhere im-
mediately after the K-Pg mass extinction event.
The highest K-Pg origination rates all occurred
in tropical and warm-temperate regions. A dis-
tinct pulse of bivalve diversification in the early
Cenozoic was concentrated mainly in tropical
and subtropical regions (see the figure).
The steepest part of the global backward
survivorship curve for bivalves lies between 65
and 50 million years ago, pointing to a major
biodiversification event in the Paleogene (65 to
23 million years ago) that is perhaps not yet
captured in Alroy et al.βs database (5, 7). The
jury is still out on what may have caused this
event. But we should not lose sight of the fact
that the steep rise to prominence of many mod-
ern floral and faunal groups in the Cenozoic
may bear no simple relationship to climate or
any other type of environmental change (10, 11).
References
1. G. G. Mittelbach et al., Ecol. Lett. 10, 315 (2007).2. A. Z. Krug, D. Jablonski, J. W. Valentine, Science 323, 767
(2009).3. P. W. Signor, Annu. Rev. Ecol. Syst. 21, 509 (1990).4. R. K. Bambach, Geobios 32, 131 (1999).5. J. Alroy et al., Proc. Natl. Acad. Sci. U.S.A. 98, 6261 (2001).6. A.M. Bush et al., Paleobiology 30, 666 (2004).7. J. Alroy et al., Science 321, 97 (2008).8. M. Foote, in Evolutionary Patterns, J. B. C. Jackson et al.,
Eds. (Univ. of Chicago Press, Chicago, IL, 2001), vol. 245,pp. 245β295.
9. M. D. Spalding et al., Bioscience 57, 573 (2007).10. S. M. Stanley, Paleobiology 33, 1 (2007).11. M. J. Benton, B. C. Emerson, Palaeontology 50, 23 (2007).
10.1126/science.1169410
www.sciencemag.org SCIENCE VOL 323 6 FEBRUARY 2009 721
PERSPECTIVES
We live life in the network. We check
our e-mails regularly, make mobile
phone calls from almost any loca-
tion, swipe transit cards to use public trans-
portation, and make purchases with credit
cards. Our movements in public places may be
captured by video cameras, and our medical
records stored as digital files. We may post blog
entries accessible to anyone, or maintain friend-
ships through online social networks. Each of
these transactions leaves digital traces that can
be compiled into comprehensive pictures of
both individual and group behavior, with the
potential to transform our understanding of our
lives, organizations, and societies.
The capacity to collect and analyze massive
amounts of data has transformed such fields as
biology and physics. But the emergence of a
data-driven βcomputational social scienceβ has
been much slower. Leading journals in eco-
nomics, sociology, and political science show
little evidence of this field. But computational
social science is occurringβin Internet compa-
nies such as Google and Yahoo, and in govern-
ment agencies such as the U.S. National Secur-
ity Agency. Computational social science could
become the exclusive domain of private com-
panies and government agencies. Alternatively,
there might emerge a privileged set of aca-
demic researchers presiding over private data
from which they produce papers that cannot be
critiqued or replicated. Neither scenario will
serve the long-term public interest of accumu-
lating, verifying, and disseminating knowledge.
What value might a computational social
scienceβbased in an open academic environ-
mentβoffer society, by enhancing understand-
ing of individuals and collectives? What are the
A field is emerging that leverages the
capacity to collect and analyze data at a
scale that may reveal patterns of individual
and group behaviors.
Computational Social Science
David Lazer,
1
Alex Pentland,
2
Lada Adamic,
3
Sinan Aral,
2,4
Albert-LΓ‘szlΓ³ BarabΓ‘si,
5
Devon Brewer,
6
Nicholas Christakis,
1
Noshir Contractor,
7
James Fowler,
8
Myron Gutmann,
3
Tony Jebara,
9
Gary King,
1
Michael Macy,
10
Deb Roy,
2
Marshall Van Alstyne
2,11
SOCIAL SCIENCE
1Harvard University, Cambridge, MA, USA. 2MassachusettsInstitute of Technology, Cambridge, MA, USA. 3Universityof Michigan, Ann Arbor, MI, USA. 4New York University,New York, NY, USA. 5Northeastern University, Boston, MA,USA. 6Interdisciplinary Scientific Research, Seattle, WA,USA. 7Northwestern University, Evanston, IL, USA.8University of CaliforniaβSan Diego, La Jolla, CA, USA.9Columbia University, New York, NY, USA 10CornellUniversity, Ithaca, NY, USA. 11Boston University, Boston,MA, USA. E-mail: [email protected]. Completeaffiliations are listed in the supporting online material.
Data from the blogosphere. Shown is a link structure within a community of political blogs (from 2004),where red nodes indicate conservative blogs, and blue liberal. Orange links go from liberal to conservative,and purple ones from conservative to liberal. The size of each blog reflects the number of other blogs thatlink to it. [Reproduced from (8) with permission from the Association for Computing Machinery]
Published by AAAS
β... shares with other nascent interdisciplinary fields
(e.g., sustainability science) the need to develop a
paradigm for training new scholars ...β
http://sciencemag.org/content/323/5915/721
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 7 / 62
The clean real story
βWe have a habit in writing articles published in
scientific journals to make the work as finished as
possible, to cover all the tracks, to not worry about the
blind alleys or to describe how you had the wrong idea
first, and so on. So there isnβt any place to publish, in
a dignified manner, what you actually did in order to
get to do the work ...β
-Richard FeynmanNobel Lecture
1, 1965
1
http://bit.ly/feynmannobel
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 8 / 62
Outline
Search predictions"Right Round"
Week
Ran
k
40
30
20
10
cccccccccccccccccccccccccccccccccccccccccc
Marβ09 Aprβ09 Mayβ09 Junβ09 Julβ09 Augβ09
BillboardSearch
Web diversity
Dai
ly P
erβC
apita
Pag
evie
ws
0
10
20
30
40
50
60
70
β
β
ββ
β
Over $25k
Under $25k
Black&
Hispanic
White
No College
Some College
Over 65
Under 65
Female
Male
Income Race Education Age Sex
Information diβ΅usion
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 9 / 62
Predicting consumer activity with Web searchwith Sharad Goel, Sebastien Lahaie, David Pennock, Duncan Watts
"Right Round"
Week
Ran
k
40
30
20
10
cccccccccccccccccccccccccccccccccccccccccc
Marβ09 Aprβ09 Mayβ09 Junβ09 Julβ09 Augβ09
BillboardSearch
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 10 / 62
Search predictionsMotivation
Does collective search activityprovide useful predictive signalabout real-world outcomes?
"Right Round"
Week
Ran
k
40
30
20
10
cccccccccccccccccccccccccccccccccccccccccc
Marβ09 Aprβ09 Mayβ09 Junβ09 Julβ09 Augβ09
BillboardSearch
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 11 / 62
Search predictionsMotivation
Past work mainly focuses on predicting the present2 and ignoresbaseline models trained on publicly available data
Date
Flu
Leve
l (Pe
rcen
t)
1
2
3
4
5
6
7
8
2004 2005 2006 2007 2008 2009 2010
ActualSearchAutoregressive
2
Varian, 2009
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 12 / 62
Search predictionsMotivation
We predict future sales for movies, video games, and music
"Transformers 2"
Time to Release (Days)
Sear
ch V
olum
e
a
β30 β20 β10 0 10 20 30
"Tom Clancy's HAWX"
Time to Release (Days)
Sear
ch V
olum
e
b
β30 β20 β10 0 10 20 30
"Right Round"
Week
Rank
40
30
20
10
cccccccccccccccccccccccccccccccccccccccccc
Marβ09 Aprβ09 Mayβ09 Junβ09 Julβ09 Augβ09
BillboardSearch
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 13 / 62
Search predictionsSearch models
For movies and video games, predict opening weekend box oοΏ½ceand first month sales, respectively:
log(revenue) = οΏ½0
+ οΏ½1
log(search) + β
For music, predict following weekβs Billboard Hot 100 rank:
billboardt+1
= οΏ½0
+ οΏ½1
searcht + οΏ½2
searchtοΏ½1
+ β
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 14 / 62
Search predictionsSearch volume
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 15 / 62
Search predictionsSearch models
Search activity is predictive for movies, video games, and musicweeks to months in advance
Movies
Predicted Revenue (Dollars)
Actu
al Re
venu
e (D
ollar
s)
103
104
105
106
107
108
109
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
ββ
β
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β β
β
β
βββ
β
β
β
β
βββ
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
103 104 105 106 107 108 109
Video Games
Predicted Revenue (Dollars)
Actu
al Re
venu
e (D
ollar
s)103
104
105
106
107
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
ββ
β β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
103 104 105 106 107
β NonβSequelSequel
Music
Predicted Billboard Rank
Actu
al Bi
llboa
rd R
ank
0
20
40
60
80
100
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
c
0 20 40 60 80 100
Movies
Time to Release (Weeks)
Mod
el Fi
t
0.4
0.5
0.6
0.7
0.8
0.9 ddddddd
β6 β5 β4 β3 β2 β1 0
Video Games
Time to Release (Weeks)
Mod
el Fi
t
0.4
0.5
0.6
0.7
0.8
0.9 eeeeeee
β6 β5 β4 β3 β2 β1 0
Music
Time to Release (Weeks)M
odel
Fit
0.4
0.5
0.6
0.7
0.8
0.9 fffffff
β6 β5 β4 β3 β2 β1 0
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 16 / 62
Search predictionsBaseline models
For movies, use budget, number of opening screens and HollywoodStock Exchange:
log(revenue) = οΏ½0
+ οΏ½1
log(budget) + οΏ½2
log(screens) +
οΏ½3
log(hsx) + β
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 17 / 62
Search predictionsBaseline models
For video games, use critic ratings and predecessor sales (sequelsonly):
log(revenue) = οΏ½0
+ οΏ½1
rating + οΏ½2
log(predecessor) + β
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 17 / 62
Search predictionsBaseline models
For music, use an autoregressive model with the previouslyavailable rank:
billboardt+1
= οΏ½0
+ οΏ½1
billboardtοΏ½1
+ β
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 17 / 62
Search predictionsBaseline + combined models
Baseline models are often surprisingly good
Movies (Baseline)
Predicted Revenue (Dollars)
Actu
al Re
venu
e (D
ollar
s)
103
104
105
106
107
108
109
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
βββ
β
β
β
β
β
β β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β β
β
β
ββββ
β
β
β
βββ
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
103 104 105 106 107 108 109
Video Games (Baseline)
Predicted Revenue (Dollars)
Actu
al Re
venu
e (D
ollar
s)103
104
105
106
107
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
ββ
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
103 104 105 106 107
β NonβSequelSequel
Music (Baseline)
Predicted Billboard Rank
Actu
al Bi
llboa
rd R
ank
0
20
40
60
80
100
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
c
0 20 40 60 80 100
Movies (Combined)
Predicted Revenue (Dollars)
Actu
al Re
venu
e (D
ollar
s)
103
104
105
106
107
108
109
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β β
β
β
ββββ
β
β
β
βββ
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
103 104 105 106 107 108 109
Video Games (Combined)
Predicted Revenue (Dollars)
Actu
al Re
venu
e (D
ollar
s)
103
104
105
106
107
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
ββ
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
103 104 105 106 107
β NonβSequelSequel
Music (Combined)
Predicted Billboard Rank
Actu
al Bi
llboa
rd R
ank
0
20
40
60
80
100
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
f
0 20 40 60 80 100
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 18 / 62
Search predictionsModel comparison
For movies, search is outperformed by the baseline and of littlemarginal value
M
odel
Fit
0.4
0.5
0.6
0.7
0.8
0.9
1.0
CombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombined
SearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearch
BaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaseline
Nonse
quel
Games
Seque
l Gam
esMus
ic
Movies Flu
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 19 / 62
Search predictionsModel comparison
For video games, search helps substantially for non-sequels, less sofor sequels
M
odel
Fit
0.4
0.5
0.6
0.7
0.8
0.9
1.0
CombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombined
SearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearch
BaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaseline
Nonse
quel
Games
Seque
l Gam
esMus
ic
Movies Flu
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 19 / 62
Search predictionsModel comparison
For music, the addition of search yields a substantially bettercombined model
M
odel
Fit
0.4
0.5
0.6
0.7
0.8
0.9
1.0
CombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombinedCombined
SearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearchSearch
BaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaselineBaseline
Nonse
quel
Games
Seque
l Gam
esMus
ic
Movies Flu
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 19 / 62
Search predictionsSummary
β’ Relative performance and value of search varies acrossdomains
β’ Search provides a fast, convenient, and flexible signal acrossdomains
β’ βPredicting consumer activity with Web searchβGoel, Hofman, Lahaie, Pennock & Watts, PNAS 2010
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 20 / 62
Outline
Search predictions"Right Round"
Week
Ran
k
40
30
20
10
cccccccccccccccccccccccccccccccccccccccccc
Marβ09 Aprβ09 Mayβ09 Junβ09 Julβ09 Augβ09
BillboardSearch
Web diversity
Dai
ly P
erβC
apita
Pag
evie
ws
0
10
20
30
40
50
60
70
β
β
ββ
β
Over $25k
Under $25k
Black&
Hispanic
White
No College
Some College
Over 65
Under 65
Female
Male
Income Race Education Age Sex
Information diβ΅usion
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 21 / 62
Demographic diversity on the Webwith Irmak Sirer and Sharad Goel (ICWSM 2012)
Dai
ly P
erβC
apita
Pag
evie
ws
0
10
20
30
40
50
60
70
β
β
ββ
β
Over $25k
Under $25k
Black&
Hispanic
White
No College
Some College
Over 65
Under 65
Female
Male
Income Race Education Age Sex
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 22 / 62
Motivation
Previous work is largely survey-based and focuses and group-leveldiβ΅erences in online access
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 23 / 62
Motivation
βAs of January 1997, we estimate that 5.2 million
African Americans and 40.8 million whites have ever used
the Web, and that 1.4 million African Americans and
20.3 million whites used the Web in the past week.β
-Hoβ΅man & Novak (1998)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 23 / 62
Motivation
Focus on activity instead of access
How diverse is the Web?
To what extent do online experiences vary across demographicgroups?
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 24 / 62
Data
β’ Representative sample of 265,000 individuals in the US, paidvia the Nielsen MegaPanel3
β’ Log of anonymized, complete browsing activity from June2009 through May 2010 (URLs viewed, timestamps, etc.)
β’ Detailed individual and household demographic information(age, education, income, race, sex, etc.)
3
Special thanks to Mainak Mazumdar
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 25 / 62
Data
# ls -alh nielsen_megapanel.tar
-rw-r--r-- 100G Jul 17 13:00 nielsen_megapanel.tar
β’ Normalize pageviews to at most three domain levels, sans wwwe.g. www.yahoo.com ! yahoo.com,us.mg2.mail.yahoo.com/neo/launch ! mail.yahoo.com
β’ Restrict to top 100k (out of 9M+ total) most popular sites(by unique visitors)
β’ Aggregate activity at the site, group, and user levels
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 26 / 62
Data
# ls -alh nielsen_megapanel.tar
-rw-r--r-- 100G Jul 17 13:00 nielsen_megapanel.tar
β’ Normalize pageviews to at most three domain levels, sans wwwe.g. www.yahoo.com ! yahoo.com,us.mg2.mail.yahoo.com/neo/launch ! mail.yahoo.com
β’ Restrict to top 100k (out of 9M+ total) most popular sites(by unique visitors)
β’ Aggregate activity at the site, group, and user levels
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 26 / 62
Data
# ls -alh nielsen_megapanel.tar
-rw-r--r-- 100G Jul 17 13:00 nielsen_megapanel.tar
β’ Normalize pageviews to at most three domain levels, sans wwwe.g. www.yahoo.com ! yahoo.com,us.mg2.mail.yahoo.com/neo/launch ! mail.yahoo.com
β’ Restrict to top 100k (out of 9M+ total) most popular sites(by unique visitors)
β’ Aggregate activity at the site, group, and user levels
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 26 / 62
Data
# ls -alh nielsen_megapanel.tar
-rw-r--r-- 100G Jul 17 13:00 nielsen_megapanel.tar
β’ Normalize pageviews to at most three domain levels, sans wwwe.g. www.yahoo.com ! yahoo.com,us.mg2.mail.yahoo.com/neo/launch ! mail.yahoo.com
β’ Restrict to top 100k (out of 9M+ total) most popular sites(by unique visitors)
β’ Aggregate activity at the site, group, and user levels
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 26 / 62
Aggregate usage patterns
How do users distribute their time across diβ΅erent categories?
Frac
tion
of to
tal p
agev
iew
s
0.05
0.10
0.15
0.20
0.25β
β
β
β β
Social
Media
Eβmail
Games
Portals
Search
All groups spend the majority of their time in the top five mostpopular categories
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 27 / 62
Aggregate usage patterns
How do users distribute their time across diβ΅erent categories?
User Rank by Daily Activity
Frac
tion
of P
agev
iew
s in
Cat
egor
y
0.05
0.10
0.15
0.20
0.25
0.30
β
β β β ββ
β
β
β
β
10% 30% 50% 70% 90%
β Social MediaEβmailGamesPortalsSearch
Highly active users devote nearly twice as much of their time tosocial media relative to typical individuals
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 27 / 62
Group-level activity
How does browsing activity vary at the group level?
Dai
ly P
erβC
apita
Pag
evie
ws
0
10
20
30
40
50
60
70
β
β
ββ
β
Over $25k
Under $25k
Black&
Hispanic
White
No College
Some College
Over 65
Under 65
Female
Male
Income Race Education Age Sex
Large diβ΅erences exist even at the aggregate level(e.g. women on average generate 40% more pageviews than men)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 28 / 62
Group-level activity
How does browsing activity vary at the group level?
Dai
ly P
erβC
apita
Pag
evie
ws
0
10
20
30
40
50
60
70
β
β
ββ
β
Over $25k
Under $25k
Black&
Hispanic
White
No College
Some College
Over 65
Under 65
Female
Male
Income Race Education Age Sex
Younger and more educated individuals are both more likely toaccess the Web and more active once they do
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 28 / 62
Group-level activity
All demographic groups spend the majority of their time in thesame categories
Age
Frac
tion
of to
tal p
agev
iew
s
0.0
0.1
0.2
0.3
0.4
0.5
β
β
β
β
ββ
β β
β
β
β
β
ββ
β β
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80
β Social MediaEβmailGamesPortalsSearch
Fr
actio
n of
tota
l pag
evie
ws
0.0
0.1
0.2
0.3
0.4Education
β β
ββ
β
β
β
Grammar
Schoo
l
Some H
igh Sch
ool
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
Sex
β
β
Female Male
Income
ββ β
ββ
β
$0β25k
$25β50k
$50β75k
$75β100k
$100β150k
$150k+
Race
β ββ β
β
Other
Hispan
icBlack
White
Asian
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 29 / 62
Group-level activity
Older, more educated, male, wealthier, and Asian Internet usersspend a smaller fraction of their time on social media
Age
Frac
tion
of to
tal p
agev
iew
s
0.0
0.1
0.2
0.3
0.4
0.5
β
β
β
β
ββ
β β
β
β
β
β
ββ
β β
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80
β Social MediaEβmailGamesPortalsSearch
Fr
actio
n of
tota
l pag
evie
ws
0.0
0.1
0.2
0.3
0.4Education
β β
ββ
β
β
β
Grammar
Schoo
l
Some H
igh Sch
ool
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
Sex
β
β
Female Male
Income
ββ β
ββ
β
$0β25k
$25β50k
$50β75k
$75β100k
$100β150k
$150k+
Race
β ββ β
β
Other
Hispan
icBlack
White
Asian
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 29 / 62
Group-level activity
Lower social media use by these groups is often accompanied byhigher e-mail volume
Age
Frac
tion
of to
tal p
agev
iew
s
0.0
0.1
0.2
0.3
0.4
0.5
β
β
β
β
ββ
β β
β
β
β
β
ββ
β β
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80
β Social MediaEβmailGamesPortalsSearch
Fr
actio
n of
tota
l pag
evie
ws
0.0
0.1
0.2
0.3
0.4Education
β β
ββ
β
β
β
Grammar
Schoo
l
Some H
igh Sch
ool
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
Sex
β
β
Female Male
Income
ββ β
ββ
β
$0β25k
$25β50k
$50β75k
$75β100k
$100β150k
$150k+
Race
β ββ β
β
Other
Hispan
icBlack
White
Asian
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 29 / 62
Group-level activity
Fem
aleβ
toβm
ale
page
view
ratio
0.5
1
2
β β
ββ
β
β β β β β
β β β
ββ β
β β β ββ
β ββββββ β β β β β ββ β β β β βββ ββ ββ β
β β β β ββββ β
ββ β β β β β β β
ββ ββ β
β β β β ββ
β β β
β
β
β
β
β
ββ
Appare
l/Bea
uty
Family
Resou
rces
Multiβc
atego
ry Hom
e & Fa
shionPets
Holiday
s & Spe
cial E
vents
Health
, Fitn
ess &
Nutritio
n
Food
& Cookin
g
Photog
raphy
NonβP
rofit
Multiβc
atego
ry Spe
cial O
ccasio
ns
Home &
Gard
en
Multiβc
atego
ry Fa
mily & Li
festyle
sBoo
ks
Membe
r Com
munitie
s
Mass M
ercha
ndise
r
Greetin
g Card
s
Genea
logy
Univers
ities
Shopp
ing Dire
ctorie
s & G
uides
Educa
tiona
l Res
ource
s
Gifts & Flow
ers
Corpora
te Inf
ormati
on
Real E
state/
Apartm
ents
Eβmail
Kids, G
ames
, Toy
s
Govern
ment
Online G
ames
Directo
ries/L
ocal
Guides
Coupo
ns/Rew
ards
Cellular
/Paging
Multiβc
atego
ry Te
lecom
/Inter
net S
ervice
s
Cruise
Line
s
Insura
nce
Full Serv
ice Ban
ks & Cred
it Unio
ns
Full Serv
ice Com
mercial
Banks
& Credit U
nionsLo
ans
Religion
& Spiritu
ality
Broadc
ast M
edia
Destin
ation
s
Multiβc
atego
ry Tra
vel
Genera
l Inter
est P
ortals
& Commun
ities
Software
Man
ufactu
rers
Delivery
/Stamps
Arts/G
raphic
s
Credit C
ard
Search
Hotels/H
otel D
irecto
ries
Maps/T
ravel
Info
Multiβc
atego
ry Ente
rtainm
ent
Long
Distanc
e/Loc
al Carr
ier
Airline
s
Career
Develop
ment
Financ
ial To
ols
Classifi
eds/A
uctio
ns
Free M
ercha
ndiseEve
nts
Multiβc
atego
ry New
s & In
formati
onISP
Instan
t Mes
sagin
g
Ground
Tran
sport
ation
Multiβc
atego
ry Fina
nce/I
nsura
nce/I
nvestm
ents
Curren
t Eve
nts & G
lobal
News
Music
Specia
l Inter
est N
ews
Weathe
r
Intern
et To
ols/W
eb Serv
ices
Gamblin
g/Swee
pstak
es
Resea
rch To
ols
Military
Hardware
Man
ufactu
rers
Targe
ted Port
als & Com
munitie
s
Multiβc
atego
ry Com
puter
s & Con
sumer
Electro
nics
Automoti
ve M
anufa
cturer
Videos
/Mov
ies
Web Hos
ting
Compu
ter & Con
sumer
Electro
nics N
ews
Multiβc
atego
ry Auto
motive
Automoti
ve In
formati
on
MultiβC
atego
ry Edu
catio
n & Care
ers
Parts &
Accesso
ries
Financ
ial New
s & In
formati
onHum
or
Person
als
Online T
radingSpo
rtsAdu
lt
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 30 / 62
Revisiting the digital divide
How does usage of news, health, and reference vary withdemographics?
A
vera
ge p
agev
iews
per
mon
th
0
2
4
6
8
10
12Education
β
β
β
β β
β
β
Grammar
Schoo
l
Some H
igh Sch
ool
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
Sex
β
β
Female Male
Income
β β ββ
β
β
$0β25k
$25β50k
$50β75k
$75β100k
$100β150k
$150k+
Race
β ββ
β
β
Other
Hispan
icBlack
White
Asian
β NewsHealthReference
Post-graduates spend three times as much time on health sitesthan adults with only some high school education
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 31 / 62
Revisiting the digital divide
How does usage of news, health, and reference vary withdemographics?
A
vera
ge p
agev
iews
per
mon
th
0
2
4
6
8
10
12Education
β
β
β
β β
β
β
Grammar
Schoo
l
Some H
igh Sch
ool
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
Sex
β
β
Female Male
Income
β β ββ
β
β
$0β25k
$25β50k
$50β75k
$75β100k
$100β150k
$150k+
Race
β ββ
β
β
Other
Hispan
icBlack
White
Asian
β NewsHealthReference
Asians spend more than 50% more time browsing online news thando other race groups
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 31 / 62
Revisiting the digital divide
How does usage of news, health, and reference vary withdemographics?
A
vera
ge p
agev
iews
per
mon
th
0
2
4
6
8
10
12Education
β
β
β
β β
β
β
Grammar
Schoo
l
Some H
igh Sch
ool
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
Sex
β
β
Female Male
Income
β β ββ
β
β
$0β25k
$25β50k
$50β75k
$75β100k
$100β150k
$150k+
Race
β ββ
β
β
Other
Hispan
icBlack
White
Asian
β NewsHealthReference
Even when less educated and less wealthy groups gain access tothe Web, they utilize these resources relatively infrequently
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 31 / 62
Revisiting the digital divide
How does usage of news, health, and reference vary withdemographics?
A
vera
ge p
agev
iew
s pe
r mon
th
0
2
4
6
8
10
12News
β
β β
β
β
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
Health
ββ β
ββ
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
Reference
ββ β
β β
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
AsianBlackHispanicWhite
Controlling for other variables, eβ΅ects of race and gender largelydisappear, while education continues to have large eβ΅ect
pi =X
j
β΅jxij +X
j
X
k
οΏ½jkxijxik +X
j
οΏ½jx2
ij + βi
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 32 / 62
Revisiting the digital divide
How does usage of news, health, and reference vary withdemographics?
A
vera
ge p
agev
iew
s pe
r mon
th
0
2
4
6
8
10
12Health
ββ β
β β
High Sch
ool G
radua
te
Some C
ollege
Associa
te Deg
ree
Bache
lor's D
egree
Post G
radua
te Deg
ree
FemaleMale
However, women spend considerably more time on health sitescompared to men
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 33 / 62
Revisiting the digital divide
How does usage of news, health, and reference vary withdemographics?
Monthly pageviews on health sites
20 40 60 80 100
FemaleMale
However, women spend considerably more time on health sitescompared to men, although means can be misleading
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 33 / 62
Individual-level prediction
How well can one predict an individualβs demographics from theirbrowsing activity?
β’ Represent each user by the set of sites visited
β’ Fit linear models4 to predict majority/minority for eachattribute on 80% of users
β’ Tune model parameters using a 10% validation set
β’ Evaluate final performance on held-out 10% test set
4
http://bit.ly/svmperf
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 34 / 62
Individual-level prediction
Reasonable (β 70-85%) accuracy and AUC across all attributes
College/No College
Under/Over $50,000Household Income
White/NonβWhite
Female/Male
Over/Under 25Years Old
Accuracyβ
β
β
β
β
.5 .6 .7 .8 .9 1
AUCβ
β
β
β
β
.5 .6 .7 .8 .9 1
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 35 / 62
Individual-level prediction
Highly-weighted sites under the fitted models
Large positive weight Large negative weight
Femalewinster.com
lancome-usa.com
sports.yahoo.com
espn.go.com
Whitemarlboro.com
cmt.com
mediatakeout.com
bet.com
College Educatednews.yahoo.com
linkedin.com
youtube.com
myspace.com
Over 25 Years Oldevite.com
classmates.com
addictinggames.com
youtube.com
Household IncomeUnder $50,000
eharmony.com
tracfone.com
rownine.com
matrixdirect.com
Table 2: A selection of the most predictive (i.e., most highly weighted) sites for each classification task.
College/No College
Under/Over $50,000Household Income
White/NonβWhite
Female/Male
Over/Under 25Years Old
AUC!
!
!
!
!
.5 .6 .7 .8 .9 1
Accuracy!
!
!
!
!
.5 .6 .7 .8 .9 1
Figure 7: Summary of model performance, indicatedby solid circles, for all demographic attributes. Pop-ulation skew is given by xβs for comparison. Notethat higher AUC closely corresponds to lower Jac-card similarity, as shown in Figure 6.
ear SVMs generate predictions of the form
y(xi) = w Β· xi + b
where the predicted class is defined by the sign of y(xi). Toguard against overfitting, SVMs seek the weight vector wthat maximally separates the positive and negative examplesin the training set. Specifically, SVMs optimize the lossfunction
L(y, y) = CX
i
[1 οΏ½ yiy(xi)]+ + ||w||2
where [x]+ = (|x|+x)/2 indicates the positive part, and C isa tunable parameter that balances model fit against gener-alization. Users are randomly divided into an 80% trainingset on which models are fit, a 10% validation set used toselect the optimal parameter C for each demographic at-tribute, and a 10% held-out test set on which we evaluateand report final performance.
Figure 7 summarizes our results for all five classificationtasks. The right panel displays the accuracy of predictions,showing reasonable performance across all demographic di-mensions, with slightly higher accuracies for age, sex, andraceβ80%, 76%, and 82%, respectivelyβthan for educationand incomeβ70% and 68%. To help put these numbers inperspective, Figure 7 also includes the overall populationskew for each demographic attribute, indicated by xβs (e.g.,57% of the online population is female, while 76% is com-prised of adults).
Given the substantial demographic skew, we also presentAUCβor area under the ROC curveβin the left panel of
Figure 7, a measure that eοΏ½ectively re-normalizes the ma-jority and minority classes to have equal size. Intuitively,AUC is the probability that a model scores a randomly se-lected positive example higher than a randomly selected neg-ative one (e.g., the probability that the model correctly dis-tinguishes between a randomly selected female and male).Though an uninformative rule would correctly discriminatebetween such pairs 50% of the time, predictions based onbrowsing histories are relatively reliable, ranging from 74%to 85%. Thus, whether we measure performance in terms ofaccuracy or AUC, we find that browsing activity provides astrong signal for inferring individual-level demographic at-tributes.
A benefit of linear models is the interpretability of theweight vector w. In Table 2, we report a sample of the mostpredictive (i.e., largest positively and negatively weighted)sites for each attribute. For example, visiting the popu-lar cosmetics company lancome-usa.com strongly indicatesthat a user is female, while visits to the sports sites sports.yahoo.com or espn.go.com are highly predictive of beingmale. Interestingly, and perhaps less apparent, the collab-orative gaming community site winster.com is also amongthe highest weighted female-predicitive sites; closer inspec-tion reveals that the site was created by a northern Cal-ifornia housewife as an alternative to gaming destinationsthat cater to young males. Analogously, visits to Coun-try Music Television (cmt.com) are a strong indicator of be-ing White, while visits to Black Entertainment Television(bet.com) are a strong non-White indicator. Though visitsto highly weighted sites provide strong cues, we note thatmany such sites are frequented by a relatively small frac-tion of the population. Thus, model performance is likelyenhanced by the many weak signals from visits to popularbut less discriminating sites.
We next examine whether demographic diοΏ½erences in on-line activityβas measured by predictive qualityβpersist aswe restrict to increasingly popular sites. As shown in Figure8, models fit on as few as the top 1,000 sites perform onlymarginally worse than those fit on all 114,000 domains (farright)βin other words, even on these top sites, demographicdiοΏ½erences are relatively large. For example, in predictingsex using the top 1,000 sites, AUC decreases only four per-centage points, from 75% to 71%. That visits to popularβand relatively heterogenousβsites are quite informative is atestament to the aggregate strength of weak signals.
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 36 / 62
Individual-level prediction
Substantially better performance when restricted to βstereotypicalβusers (β 80-90%)
Fraction of Users
AUC
0.70
0.75
0.80
0.85
0.90
0.95
βββββ
β
β
β
β
0.0 0.2 0.4 0.6 0.8 1.0
β AgeSexRaceEducationIncome
Fraction of Users
Accu
racy
0.70
0.75
0.80
0.85
0.90
0.95
ββββββ
β
β
β
0.0 0.2 0.4 0.6 0.8 1.0
β AgeSexRaceEducationIncome
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 37 / 62
Individual-level prediction
Similar performance even when restricted to top 1k sites
Number of Domains
AUC
0.5
0.6
0.7
0.8
0.9
β
β
β β
102 102.5 103 103.5 104 104.5 105
β AgeSexRaceEducationIncome
Number of Domains
Accu
racy
0.5
0.6
0.7
0.8
0.9
β
ββ β
102 102.5 103 103.5 104 104.5 105
β AgeSexRaceEducationIncome
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 38 / 62
Site-level skew
Proportion Female Visitors
Den
sity
0.0 0.2 0.4 0.6 0.8 1.0Proportion White Visitors
Den
sity
0.0 0.2 0.4 0.6 0.8 1.0Proportion College Educated Visitors
Den
sity
0.0 0.2 0.4 0.6 0.8 1.0
Proportion Adult Visitors
Den
sity
0.0 0.2 0.4 0.6 0.8 1.0 Proportion of Visitors WithHousehold Incomes Under $50,000
Den
sity
0.0 0.2 0.4 0.6 0.8 1.0
Many sites have skew close the overall mean, but there alsopopular, highly-skewed sites
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 39 / 62
Individual-level prediction
Proof of concept browser demo
http://bit.ly/surfpreds
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 40 / 62
Summary
β’ Highly active users spend disproportionately more of theirtime on social media and less on e-mail relative to the overallpopulation
β’ Access to research, news, and healthcare is strongly related toeducation, not as closely to ethnicity
β’ User demographics can be inferred from browsing activity withreasonable accuracy
β’ βWho Does What on the Webβ, Goel, Hofman & Sirer,ICWSM 2012
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 41 / 62
Outline
Search predictions"Right Round"
Week
Ran
k
40
30
20
10
cccccccccccccccccccccccccccccccccccccccccc
Marβ09 Aprβ09 Mayβ09 Junβ09 Julβ09 Augβ09
BillboardSearch
Web diversity
Dai
ly P
erβC
apita
Pag
evie
ws
0
10
20
30
40
50
60
70
β
β
ββ
β
Over $25k
Under $25k
Black&
Hispanic
White
No College
Some College
Over 65
Under 65
Female
Male
Income Race Education Age Sex
Information diβ΅usion
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 42 / 62
The structual virality of online diβ΅usionwith Ashton Anderson, Sharad Goel, Duncan Watts (201?)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 43 / 62
βGoing Viralβ?
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 44 / 62
βGoing Viralβ?
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 45 / 62
βGoing Viralβ?
βTherefore we ... wish to proceed with great care as is
proper, and to cut oβ΅ the advance of this plague and
cancerous disease so it will not spread any further ...β
5
-Pope Leo XExsurge Domine (1520)
5
http://www.economist.com/node/21541719
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 45 / 62
βGoing Viralβ?
Rogers (1962), Bass (1969)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 46 / 62
βGoing viralβ?
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 47 / 62
βGoing viralβ?
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 47 / 62
Data
β’ Examined one year of tweets from July 2011 to July 2012
β’ Restricted to 1.4 billion tweets containing links to top news,videos, images, and petitions sites
β’ Aggregated tweets by URL, resulting in 1 billion distinctβeventsβ
β’ Crawled friend list of each adopter
β’ Inferred βwho got what from whomβ to construct diβ΅usiontrees
β’ Characterized size and structure of trees
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 48 / 62
Data
β’ Examined one year of tweets from July 2011 to July 2012
β’ Restricted to 1.4 billion tweets containing links to top news,videos, images, and petitions sites
β’ Aggregated tweets by URL, resulting in 1 billion distinctβeventsβ
β’ Crawled friend list of each adopter
β’ Inferred βwho got what from whomβ to construct diβ΅usiontrees
β’ Characterized size and structure of trees
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 48 / 62
Data
β’ Examined one year of tweets from July 2011 to July 2012
β’ Restricted to 1.4 billion tweets containing links to top news,videos, images, and petitions sites
β’ Aggregated tweets by URL, resulting in 1 billion distinctβeventsβ
β’ Crawled friend list of each adopter
β’ Inferred βwho got what from whomβ to construct diβ΅usiontrees
β’ Characterized size and structure of trees
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 48 / 62
Data
β’ Examined one year of tweets from July 2011 to July 2012
β’ Restricted to 1.4 billion tweets containing links to top news,videos, images, and petitions sites
β’ Aggregated tweets by URL, resulting in 1 billion distinctβeventsβ
β’ Crawled friend list of each adopter
β’ Inferred βwho got what from whomβ to construct diβ΅usiontrees
β’ Characterized size and structure of trees
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 48 / 62
Data
β’ Examined one year of tweets from July 2011 to July 2012
β’ Restricted to 1.4 billion tweets containing links to top news,videos, images, and petitions sites
β’ Aggregated tweets by URL, resulting in 1 billion distinctβeventsβ
β’ Crawled friend list of each adopter
β’ Inferred βwho got what from whomβ to construct diβ΅usiontrees
β’ Characterized size and structure of trees
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 48 / 62
Data
β’ Examined one year of tweets from July 2011 to July 2012
β’ Restricted to 1.4 billion tweets containing links to top news,videos, images, and petitions sites
β’ Aggregated tweets by URL, resulting in 1 billion distinctβeventsβ
β’ Crawled friend list of each adopter
β’ Inferred βwho got what from whomβ to construct diβ΅usiontrees
β’ Characterized size and structure of trees
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 48 / 62
The Structural Virality of Online Diβ΅usion
A
B
D
C
E
Tim
e
Group posts by URL
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 49 / 62
The Structural Virality of Online Diβ΅usion
A
B
D
C
E
Tim
e
Label each friend who previously adopted as a potential parent
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 49 / 62
The Structural Virality of Online Diβ΅usion
A
B
D
C
E
Tim
e
Select each nodeβs most recent adopting friend as its parent
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 49 / 62
The Structural Virality of Online Diβ΅usion
A
B
D
C
E
Gene
ratio
ns
Characterize size and structure of trees
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 49 / 62
Information diβ΅usionCascade size distribution
0.00001%
0.0001%
0.001%
0.01%
0.1%
1%
10%
1 10 100 1,000 10,000
Cascade Size
CC
DF
Focus on the rare hits that get at least 100 adoptions
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 50 / 62
Quantifying structure
Measure the average distance between all pairs of nodes6
β«(T ) =1
n(n οΏ½ 1)
nX
i=1
nX
j=1
dij
6
Weiner (1947); correlated with other possible metrics
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 51 / 62
Quantifying structure
Measure the average distance between all pairs of nodes6
β«(T ) =2n
n οΏ½ 1
"1
n
X
S2S|S |οΏ½ 1
n
2
X
S2S|S |2
#
6
Weiner (1947); correlated with other possible metrics
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 51 / 62
Information diβ΅usionSize and virality by category
Remarkable structural diversity across across categories
0.001%
0.01%
0.1%
1%
10%
100%
100 1,000 10,000
Cascade Size
CC
DF
VideosPicturesNewsPetitions
0.001%
0.01%
0.1%
1%
10%
100%
3 10 30
Structural Virality
CC
DF
VideosPicturesNewsPetitions
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 52 / 62
Information diβ΅usionStructural diversity
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 53 / 62
Information diβ΅usionStructural diversity
Size is relatively poor predictive of structure
Petitions News Pictures Videos
βββ
ββ
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
ββ
ββ
β
ββββ
β
ββ
β
ββ
β
ββββ
β
β
ββ
β
βββββ
ββ
β
β
β
ββββ
β
β
ββ
β
β
ββ
β
β
ββ
β
βββ
β
β
β
β
β
βββββββ
ββ
β
β
βββββ
βββ
β
β
β
β
βββ
β
ββ
β
β
β
β
β
βββββ
β
β
β
β
β
βββ
β
β
β
βββ
βββ
β
β
β
βββ
ββ
β
β
ββββ
β
β
β
β
ββ
β
β
ββββββ
β
β
β
β
ββ
β
ββ
β
β
β
β
β
ββ
ββ
β
β
βββ
β
ββ
β
βββ
β
β
β
ββββ
β
ββ
ββ
β
β
β
β
β
β
β
β
β
β
β
βββββββ
β
βββ
β
β
β
β
βββ
β
βββββ
β
ββ
β
β
β
ββββ
β
β
β
β
β
β
β
β
β
βββ
ββ
β
β
ββ
ββββ
ββ
β
β
β
β
β
β
β
β
β
ββββ
β
ββ
β
βββ
β
ββ
β
βββ
β
β
β
βββ
β
β
β
β
β
β
β
βββ
β
βββ
ββ
ββ
β
β
β
β
β
ββ
β
β
β
ββ
βββ
β
β
ββββββ
β
β
ββ
β
ββββ
ββ
β
β
β
ββ
β
ββ
β
ββββ
β
β
β
βββ
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
βββ
β
βββ
β
ββ
ββ
β
β
β
β
β
ββ
ββ
β
β
ββββ
β
β
β
βββ
β
βββ
β
ββ
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
βββ
β
ββββ
β
β
β
β
ββββ
ββ
βββ
ββ
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
βββ
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
β
ββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
βββ
ββ
ββ
β
β
β
β
β
βββ
βββββ
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
βββ
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
βββ
β
β
β
β
βββ
βββ
β
β
β
β
βββ
βββ
β
β
ββ
β
β
β
ββ
β
ββ
β
βββββ
β
ββ
ββββ
ββ
β
ββ
ββββββ
β
β
β
β
β
ββββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββββββ
β
ββ
β
β
β
ββ
β
β
β
ββ
ββ
β
β
ββ
β
ββ
β
βββ
β
ββ
βββββ
β
β
β
β
β
βββ
ββ
ββ
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
ββββ
βββ
β
β
β
ββ
β
β
β
βββ
β
β
β
β
β
β
β
βββ
β
β
β
β
β
βββ
β
ββ
β
β
β
β
β
β
ββ
ββ
ββ
β
β
β
β
β
β
ββ
β
β
ββ
β
β
βββ
ββ
ββ
βββββ
β
β
β
βββ
ββ
β
ββ
β
β
ββββ
β
ββ
β
β
βββββ
β
β
ββ
ββ
β
β
β
β
ββββββ
β
β
ββ
β
β
β
β
β
ββ
β
ββ
β
β
β
ββ
β
β
β
ββ
β
βββ
β
β
β
β
β
β
β
β
β
β
β
βββββ
ββ
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
ββ
β
ββ
ββ
β
ββββ
β
β
ββ
β
β
ββ
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
ββ
βββ
β
ββ
β
β
β
β
β
ββ
β
ββββ
ββ
β
β
β
β
β
ββ
β
β
ββ
β
ββββ
β
ββββ
β
βββ
β
β
ββββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
ββββ
β
βββ
β
β
β
β
ββ
β
β
βββ
β
ββ
βββ
β
ββββββ
ββ
ββ
ββ
β
ββ
ββ
β
ββ
β
β
β
β
β
β
βββ
ββ
β
β
β
β
ββ
β
β
ββ
ββ
ββ
βββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
ββ
β
β
β
β
β
β
βββββ
β
β
β
ββ
β
β
β
β
ββ
β
ββ
β
β
ββ
ββ
ββ
β
β
β
β
β
βββ
β
β
ββ
β
βββ
β
ββ
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
βββββ
β
ββββββ
βββ
ββββ
β
β
ββ
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
β
β
β
β
β
β
ββ
β
β
ββ
β
β
β
β
β
β
β
ββ
β
ββ
β
β
βββ
β
βββ
β
β
β
β
βββ
β
ββ
β
ββ
β
βββ
β
ββ
ββ
β
β
β
β
β
β
β
β
βββββ
β
βββ
β
βββ
β
β
ββ
β
β
β
ββ
β
ββ
β
β
ββ
ββ
β
ββ
ββ
β
β
β
ββ
β
ββ
β
ββ
β
β
β
β
β
β
β
β
ββ
β
ββββ
β
β
β
β
β
β
ββ
β
β
β
ββ
β
ββ
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
ββ
β
β
β
β
β
β
β
ββ
ββ
β
β
ββ
β
ββββ
β
ββ
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
β
ββββββ
β
β
β
βββ
β
β
β
ββ
βββ
β
β
β
β
β
β
β
ββββ
β
β
β
β
βββββ
ββ
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
ββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
ββ
βββ
ββ
ββ
β
βββ
β
β
β
β
β
β
β
β
β
β
ββ
ββ
β
β
β
ββ
β
β
β
β
β
ββββ
β
ββ
β
β
β
β
β
ββ
β
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
ββ
β
β
ββ
βββ
β
ββββ
β
βββ
β
βββ
β
βββ
β
ββ
β
β
β
β
β
β
β
ββ
βββ
ββ
β
βββ
βββ
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββ
β
β
ββββ
β
ββββ
β
ββ
β
ββ
β
βββ
βββββ
β
βββ
β
ββ
ββββ
β
β
ββ
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββ
β
β
β
β
β
β
β
βββ
β
ββββ
β
β
β
ββ
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
βββββ
β
ββ
β
β
β
ββ
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
βββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
βββ
β
β
β
β
β
β
β
β
ββββ
ββ
β
ββββββ
βββ
ββ
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
βββ
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
ββββ
β
β
β
ββ
ββ
β
ββ
β
β
βββ
ββ
β
ββ
ββ
ββ
ββ
β
β
β
ββ
β
β
β
ββββ
β
ββ
β
β
βββ
β
ββ
β
βββ
β
β
β
β
β
β
β
βββ
β
β
β
β
β
βββββ
β
β
β
βββ
β
βββ
βββ
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
βββ
β
β
β
β
β
ββ
β
β
ββ
β
β
β
β
β
ββ
β
β
ββ
β
ββ
β
β
βββ
β
β
ββ
ββ
β
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
βββ
β
β
β
β
β
β
β
β
β
ββββββ
β
β
β
βββ
β
β
ββ
β
β
β
β
ββ
βββ
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
ββ
β
ββ
β
β
β
β
ββ
β
β
ββ
β
ββ
ββ
β
β
ββ
β
β
βββ
β
β
β
β
β
β
βββ
ββ
βββ
β
β
ββββ
β
βββ
ββ
ββ
β
ββββ
ββ
βββ
β
ββββ
β
ββ
βββ
β
β
βββ
β
β
β
β
β
ββ
β
β
β
ββ
β
ββ
β
βββ
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
ββ
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
βββ
β
β
ββββ
ββ
β
β
βββ
β
ββββ
β
β
βββ
β
β
β
β
β
βββ
ββ
β
βββ
β
ββ
ββββ
β
β
βββ
ββ
β
β
ββ
β
β
βββ
β
ββ
β
β
β
ββ
β
β
βββ
β
β
β
β
β
ββ
β
β
β
β
β
β
ββ
ββ
β
β
β
β
β
β
ββ
β
ββ
β
ββ
β
β
βββ
β
β
ββ
β
ββ
βββ
β
β
ββ
β
β
β
β
β
β
ββ
β
β
ββ
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
βββ
β
βββ
β
β
β
ββ
β
β
β
βββ
β
β
β
β
β
β
β
ββββ
β
β
ββ
ββββ
β
β
β
βββββ
β
ββ
β
βββ
β
β
β
β
βββ
β
ββ
ββββ
β
β
β
ββ
β
β
β
β
ββ
β
β
βββ
ββ
β
ββ
β
β
ββ
β
βββ
ββ
ββ
ββ
β
ββ
β
βββ
β
β
ββββββ
ββ
β
β
β
ββ
β
β
β
ββ
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
ββ
βββ
βββ
β
ββ
β
β
βββ
β
β
β
β
β
βββββ
ββ
β
βββ
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
β
ββ
ββ
ββ
ββ
β
β
β
β
β
β
β
β
β
βββ
ββββ
β
β
ββ
βββββ
β
ββ
β
ββ
β
βββ
β
ββββββ
β
β
β
β
β
ββ
β
βββββ
ββ
β
β
β
β
β
β
β
β
β
β
ββ
β
βββββββ
β
β
β
β
β
ββ
ββ
β
β
β
β
β
ββ
ββ
β
β
ββ
β
β
β
β
βββββ
ββ
β
β
ββββ
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
βββ
β
β
β
ββ
β
β
β
β
βββ
ββ
β
βββ
ββ
β
β
βββ
β
β
β
ββ
β
ββ
βββββ
βββ
β
β
ββββ
β
ββ
β
β
ββ
β
β
β
ββ
β
β
β
β
β
β
β
ββ
βββ
β
β
βββββ
β
β
βββ
β
β
βββββ
β
ββ
β
β
β
β
β
ββ
β
βββ
β
β
β
β
βββ
ββ
ββ
β
β
β
β
β
β
ββ
β
β
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
ββ
ββ
β
β
β
β
β
β
βββ
β
β
βββ
β
β
β
βββ
β
β
ββ
β
β
β
ββ
β
β
β
βββββ
β
β
β
β
β
β
β
β
β
ββ
ββ
β
β
ββββββ
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
βββ
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
βββ
βββ
β
β
β
β
β
ββ
ββ
β
β
ββ
β
β
β
β
β
β
β
β
β
β
ββββ
ββ
ββ
β
β
ββ
β
βββ
β
β
β
β
β
β
β
β
β
β
βββ
β
ββ
β
β
ββ
βββ
ββ
ββ
β
β
β
β
β
β
β
β
ββ
β
ββββ
β
β
β
β
β
β
β
β
β
β
β
β
βββ
ββ
β
β
β
β
β
β
β
β
βββ
β
ββ
β
β
ββ
ββ
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
βββ
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
ββ
β
β
β
β
βββ
β
β
β
β
β
β
ββ
β
β
β
βββββ
β
β
β
ββ
β
β
ββ
β
β
β
ββ
β
β
ββ
β
β
β
β
β
β
ββ
β
β
β
β
β
ββββββ
β
β
β
ββ
ββ
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
βββ
β
β
β
ββ
β
β
β
β
β
β
ββ
βββ
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
ββ
ββ
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
ββ
β
β
ββ
β
β
β
β
β
ββ
β
β
ββ
β
β
ββ
βββ
ββ
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
βββ
ββββ
ββββ
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
ββ
β
βββ
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
βββ
β
ββ
β
β
β
β
β
ββ
β
ββ
β
β
ββ
β
β
ββ
ββββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
βββ
β
β
β
β
β
βββββ
ββ
β
ββ
β
β
β
βββ
ββ
β
β
βββββ
βββ
β
β
β
β
β
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
ββ
βββ
β
β
β
β
ββ
β
β
ββ
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
ββ
β
β
ββ
βββ
β
ββ
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
βββ
β
β
β
β
β
ββ
ββββ
β
β
β
β
ββ
β
β
β
β
βββ
β
β
β
β
ββ
β
ββ
β
β
β
β
β
βββ
β
β
β
β
β
β
βββ
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
β
β
ββ
β
β
β
β
β
β
β
β
ββββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
ββ
ββ
β
β
β
β
β
β
β
ββ
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
β
β
β
βββ
β
β
β
β
β
β
β
ββββ
β
β
β
β
β
βββ
β
ββ
β
β
β
ββ
β
β
ββ
β
β
β
β
β
ββ
β
β
β
βββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
β
β
β
β
ββ
β
ββ
β
β
β
β
ββ
β
β
β
ββ
βββ
β
β
β
β
β
ββ
β
β
βββ
ββ
β
β
β
β
ββ
ββ
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
βββ
βββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
ββββββ
β
β
β
βββ
β
ββ
β
β
β
β
ββ
β
β
β
β
β
β
ββ
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
ββ
ββ
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
3
10
30
100
300
1,000 10
030
01,0
003,0
00 100
300
1,000
3,000
10,00
010
030
01,0
003,0
0010
,000
Cascade size
Stru
ctur
al v
iralit
y
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 54 / 62
Simulations
Simulate cascades with a simple SIR model7,varying infectivity and degree skew
662 CHAPTER 21. EPIDEMICS
y
x z
t
r v
u
w
s
(a)
y
x z
t
r v
u
w
s
(b)
y
x z
t
r v
u
w
s
(c)
y
x z
t
r v
u
w
s
(d)
Figure 21.2: The course of an SIR epidemic in which each node remains infectious for anumber of steps equal to tI = 1. Starting with nodes y and z initially infected, the epidemicspreads to some but not all of the remaining nodes. In each step, shaded nodes with darkborders are in the Infectious (I) state and shaded nodes with thin borders are in the Removed(R) state.
Extensions to the SIR model. Although the contact network in the general SIR model
can be arbitrarily complex, the disease dynamics are still being modeled in a simple way.
Contagion probabilities are set to a uniform value p, and contagiousness has a kind of βon-oοΏ½β
property: a node is equally contagious for each of the tI steps while it has the disease.
However, it is not diοΏ½cult to extend the model to handle more complex assumptions.
First, we can easily capture the idea that contagion is more likely between certain pairs of
nodes by assigning a separate probability pv,w to each pair of nodes v and w for which v
links to w in the directed contact network. Here, higher values of pv,w correspond to closer
contact and more likely contagion, while lower values indicate less intensive contact. We
can also choose to model the infectious period as random in length, by assuming that an
infected node has a probability q of recovering in each step while it is infected, while leaving
8
7
Kermack & McKendrick (1927)
8
Easley & Kleinberg (2010)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 55 / 62
Simulations
This reproduces the observed marginal distributions of size andstructure
3
10
30
100
βββ
ββββ
β
ββββββββββββββββββββββββ
ββ
β
ββ
β
β
βββββ
ββββ
ββ
β
βββ
β
βββ
β
β
βββββββββββββ
ββββββ
ββ
ββββββββββ
β
ββββββ
β
β
ββββββββββββ
ββββ
β
βββ
β
β
β
ββ
β
ββ
ββββββββ
β
ββ
β
βββββ
ββ
βββββββ
β
β
ββββββ
ββββββββ
β
β
β
ββ
β
ββ
β
β
ββ
ββ
ββββ
β
βββββββββββββββ
ββββββ
ββ
β
βββ
β
βββββ
β
β
ββββ
β
ββ
ββββββββββ
β
ββββ
β
βββ
β
ββββ
β
β
β
ββ
βββββ
β
β
β
β
ββ
βββ
β
ββββββββββββββ
β
βββ
β
βββ
βββ
βββββββββββ
ββ
β
ββββββββ
ββ
β
ββ
βββ
βββ
β
β
β
β
ββ
β
βββββββββββββββββββ
ββ
βββ
ββββββββ
βββ
β
ββββββ
βββββββ
β
βββ
ββββββββββ
ββ
β
β
β
β
βββββ
β
ββββ
β
βββββ
β
ββββββ
βββ
ββ
ββ
β
ββ
β
ββ
ββ
ββ
β
ββ
ββββ
β
β
β
β
β
ββββ
ββββ
β
β
ββ
βββββ
β
β
βββ
β
ββββββ
β
β
β
β
β
β
β
βββββ
β
β
ββββββββββ
ββββββββββ
β
β
β
ββββ
β
β
ββββ
β
β
β
β
βββ
ββ
ββββββββ
ββ
βββββββ
ββ
β
ββββ
β
βββββ
β
βββ
β
β
β
β
βββββββββββββββ
β
βββ
ββ
ββββ
β
ββ
β
ββ
β
β
ββ
ββ
ββββββββ
β
β
βββ
βββ
ββββββββ
ββ
β
ββ
β
βββ
β
βββ
ββ
β
ββ
β
βββββββ
ββ
β
ββββ
ββββββ
β
βββββ
β
β
β
β
ββ
β
β
ββ
βββββββββββββββββββββββββββ
ββββββββββββββββββββ
β
β
β
ββββ
β
β
β
β
β
βββββββ
β
βββ
β
βββββ
ββββββ
β
ββββββββ
β
ββββ
βββββββ
β
β
ββ
ββ
β
β
β
β
βββ
ββ
β
β
β
βββββ
β
β
βββ
β
βββ
β
ββ
β
β
βββββ
β
β
ββββ
β
ββββββ
β
β
β
ββββββββ
β
ββββββββ
β
βββ
β
βββ
β
β
βββ
β
ββββ
βββ
βββββββ
β
ββββ
βββ
β
βββββ
β
ββ
ββββ
β
ββββ
βββ
β
ββββββββ
βββ
β
ββββββββββ
βββ
β
βββββ
ββ
ββ
β
βββββ
β
βββ
ββββββ
ββ
β
β
β
ββ
βββ
β
β
ββββββ
βββ
β
βββ
β
ββββ
β
βββ
β
ββββββ
β
βββββββ
β
βββββ
ββββ
β
β
β
β
β
ββββ
β
β
β
ββββββββ
ββββ
β
ββββββ
ββ
β
βββ
β
ββ
ββββ
β
ββββββ
β
ββββββββ
ββββ
βββ
β
β
β
βββ
β
ββββββββ
β
ββββββββ
ββ
βββ
βββ
βββ
ββββ
βββ
β
β
β
β
β
ββββ
βββ
β
β
βββββββββββ
ββββββ
β
β
ββ
β
β
β
β
ββββ
β
β
β
βββββββ
βββ
β
ββ
β
βββββ
β
βββ
ββ
β
βββββ
ββ
β
β
β
βββββ
β
βββ
β
ββββββ
ββ
ββββββββββββ
βββ
ββββββββββββ
βββ
βββββ
ββββββββββ
β
β
βββββββββββ
β
βββ
ββββββββ
ββ
β
β
β
β
ββ
β
βββ
β
β
β
ββ
βββββββ
β
β
ββ
βββ
β
ββββ
ββββββββ
ββββ
ββββββ
β
β
β
ββββ
β
ββ
β
βββββββ
ββ
β
ββββββββββββββ
ββ
ββ
ββββ
β
ββββ
β
β
ββββββ
β
βββββββ
β
β
β
ββββ
ββ
β
ββββββ
β
β
β
β
ββ
β
β
ββ
ββ
β
βββ
β
ββ
β
β
ββββ
β
ββββ
β
βββ
β
β
β
β
βββ
β
β
βββ
βββββ
β
ββ
β
βββ
β
ββ
ββ
ββ
β
ββββ
ββ
ββββββββββββββββ
ββββββββ
β
β
βββββ
ββ
ββ
β
ββββββ
β
ββ
β
β
ββ
ββββββββββββ
β
β
βββββββ
ββββ
ββ
β
β
βββββ
βββββββββββββββββββββββ
β
β
β
β
ββββ
βββββββββββ
β
β
βββ
β
βββββββββ
ββ
β
ββββ
ββββ
β
ββ
β
ββββββ
ββ
β
ββββββ
βββ
β
β
ββ
βββββ
β
βββ
ββ
βββββ
ββββββββββββββ
β
ββββ
β
ββββββ
β
βββ
β
ββββ
βββ
ββ
β
β
β
β
β
ββ
β
βββββββ
βββ
β
ββ
β
βββ
ββ
β
ββββββ
β
ββ
β
β
β
β
ββ
βββββββββ
β
ββββ
ββ
β
βββ
ββββββββ
β
βββ
β
β
βββββββ
β
β
β
ββββ
ββ
ββ
ββ
β
β
ββββ
ββ
βββββ
β
ββββββ
ββββββββ
ββ
ββββββ
β
ββββββββ
ββ
β
βββ
β
ββββββββββββ
β
β
βββ
ββ
ββββ
β
ββ
βββββββββ
βββββββββββ
ββ
βββββ
β
β
β
βββββββ
β
βββ
β
βββ
β
ββ
ββ
β
β
ββββ
β
β
β
β
βββββββββββ
β
β
βββ
βββββββ
ββ
βββββββ
ββββββ
β
ββββββββ
β
β
β
βββββ
βββββββββββββββ
ββ
ββββββββ
β
ββββββ
β
β
β
β
βββ
βββββββ
β
βββββ
β
ββββββββββββ
β
βββ
β
ββ
ββββ
β
β
βββββ
β
βββ
β
β
β
β
β
ββββββββββββ
β
ββββββββ
β
ββ
β
ββββ
β
β
β
ββββββββββββββββ
ββ
βββ
βββββββ
β
βββ
βββ
β
ββββββ
β
ββ
β
βββββββββββββββββββββββ
β
ββ
β
ββ
β
β
β
ββ
β
ββββββββββ
β
β
β
ββββ
β
ββββ
β
βββ
β
β
ββ
β
ββββ
β
ββ
ββββββ
βββ
β
β
β
βββ
β
βββββ
β
βββββ
βββ
β
βββββββββ
β
βββββ
β
ββ
β
βββββββ
βββββββ
βββ
β
ββββ
β
β
βββ
ββ
β
βββββ
β
ββ
ββ
ββββββββββ
ββββ
β
β
β
β
β
βββββ
β
ββββββββββ
β
β
ββββ
ββββββ
ββ
βββββββββββββββββββ
βββ
ββ
ββββ
β
βββββ
βββ
ββββββββ
β
ββββ
β
βββββββ
β
β
βββββββ
ββ
β
β
ββ
ββ
ββββ
β
β
βββββ
βββββββββββ
βββββ
βββββββββ
β
βββ
β
ββ
ββ
β
β
β
β
β
β
ββ
βββ
β
ββββββββ
β
βββ
β
ββ
β
β
βββββββββ
β
ββββ
β
ββ
ββββ
βββ
βββ
β
ββββββ
ββ
βββ
β
ββββ
ββ
β
ββββββ
βββββββββ
βββββββ
ββ
β
ββββββ
β
ββ
ββ
ββ
β
ββββ
β
β
β
ββββ
ββ
βββββ
β
βββββββββββββββββββββββ
βββββ
ββ
β
β
βββββββ
β
β
β
βββ
β
ββββ
ββββββββββββ
βββββ
β
ββββββ
β
ββββββ
β
β
ββββ
β
β
ββ
β
βββ
β
β
ββββ
βββ
β
βββ
ββ
β
β
β
βββββ
β
βββ
βββββ
β
ββ
β
ββββββββββββββ
β
β
βββββ
β
β
β
β
ββ
ββββ
β
β
β
β
β
ββββββββββ
ββ
βββ
ββ
β
ββ
β
βββ
β
βββββββ
βββ
β
β
β
β
βββ
βββββ
β
βββ
β
βββ
β
ββ
ββ
β
β
ββββ
β
ββββββ
ββ
ββββββ
ββ
β
βββ
βββββ
β
β
β
ββββββ
β
βββββ
β
ββ
βββββ
β
ββ
β
ββββββ
βββββββββ
β
ββββ
β
β
β
ββββββββ
ββ
βββββββββββββββ
β
ββββββ
β
βββββ
βββββ
β
ββ
β
β
ββ
ββββ
βββ
ββ
β
β
βββββββββββ
β
ββββββ
ββ
βββ
ββββββ
β
ββββββ
β
βββ
β
β
βββ
ββ
ββ
β
β
ββββ
β
ββ
β
βββ
β
ββ
β
βββββ
βββββ
β
β
βββββββββ
β
β
ββββββββββ
ββ
β
βββ
ββ
ββ
β
ββ
β
ββββββ
ββββββββ
ββ
ββββ
β
β
β
β
β
ββββ
βββ
β
ββββββββββ
β
β
β
β
β
ββ
β
ββ
βββββ
β
βββ
β
βββββ
β
β
ββββββ
β
β
βββββ
β
β
ββββββββββββββ
β
βββ
ββ
β
β
ββ
β
β
ββββ
ββββ
β
β
β
βββββ
ββββββ
ββββββββ
βββ
β
ββ
ββββββ
β
ββββ
ββ
βββββββ
β
βββ
β
βββββ
β
β
β
βββββββ
βββ
ββ
β
β
β
ββββββ
ββ
β
β
β
ββββββ
ββ
β
ββ
ββ
βββ
β
ββββ
β
βββ
ββββββββ
β
ββββ
β
ββββββ
ββββ
β
β
β
βββββββ
ββββ
β
β
ββ
βββ
ββ
ββ
βββββ
βββ
βββ
β
βββ
β
ββ
β
β
βββ
ββββββββββ
β
ββ
ββββββ
β
ββββ
β
βββ
β
β
ββββ
ββ
ββββββββββββββββββ
βββ
β
ββ
β
ββ
ββ
ββ
β
βββββββββββββββββ
β
β
ββββ
ββ
ββββ
β
ββββββ
ββ
βββ
β
ββ
β
ββββ
β
βββ
β
β
β
βββ
β
ββββββ
β
ββ
β
β
βββ
β
β
ββββββ
β
ββββ
β
β
βββ
ββββ
βββββ
ββββ
βββββ
ββ
β
ββ
ββ
β
β
β
β
βββββ
β
β
β
β
βββ
β
βββ
ββββββββββββ
ββ
βββ
β
ββ
β
ββ
ββ
β
ββββββββββ
ββββ
β
β
ββ
βββ
β
β
β
ββββββββββ
ββββ
β
β
ββ
β
ββββββββββ
ββ
ββββββββ
β
β
β
β
ββ
ββββ
ββ
β
β
ββββββββ
β
βββββ
β
βββββββββββββ
β
β
β
βββββ
ββββββ
ββββββ
β
ββββββββ
ββββββ
ββ
βββ
ββββ
β
βββββββββββββββ
β
ββββ
βββ
ββ
β
ββββββ
βββ
β
ββββ
βββ
β
βββ
β
ββ
β
ββββββββ
β
ββββ
β
βββββββ
ββ
ββ
β
βββ
ββββββ
β
β
βββββββββ
β
ββββ
βββββ
βββ
ββββββββββββββββ
β
ββββββ
β
β
ββ
ββ
β
ββ
βββββββ
β
ββ
ββ
β
ββββββββ
β
ββ
β
βββββββ
β
β
ββββ
βββ
βββ
β
β
β
β
ββ
ββββββ
βββββ
ββ
βββ
β
βββ
β
ββ
β
ββββββββββββ
β
β
β
ββ
β
β
β
ββββββββββ
βββββ
ββ
β
β
β
ββββββββββ
β
βββββββββ
ββ
β
β
ββββ
βββββ
β
βββββββ
ββ
βββββ
βββββ
β
βββββββββββββ
β
βββ
ββββ
β
ββββββββββββ
ββ
β
ββββββ
ββββ
βββββββββββββ
β
βββββββββ
βββ
βββββ
ββ
βββββββ
β
β
ββββ
β
ββββββββββ
ββββββββ
βββ
β
β
βββ
β
βββββββββββ
β
βββββββ
ββ
β
β
ββ
ββββββββ
β
βββββββββ
ββ
β
βββ
β
ββββ
βββββββββββββ
βββββββ
ββ
β
βββ
ββ
β
β
ββββββ
β
ββββ
ββββββββ
βββ
β
ββββ
β
βββ
β
ββββββββββββ
β
ββ
β
βββββββββ
βββββββ
ββββββββ
β
βββ
β
β
βββββ
β
β
ββ
β
ββββββ
β
ββββββββ
βββββββ
β
β
β
ββββ
β
βββββββββββ
βββββββββββ
ββββββββββββ
ββ
β
βββββ
βββββββββ
ββ
β
β
ββ
ββ
β
βββββ
β
βββ
β
β
βββββ
βββββββββ
β
ββ
β
ββ
βββ
β
β
βββ
β
β
β
ββ
β
β
β
β
β
β
β
βββββββββββ
ββ
β
βββββββββββββ
β
βββ
β
ββββ
β
ββ
β
β
βββββ
β
ββββββ
β
ββ
β
β
β
β
ββ
βββββ
β
βββ
ββ
βββ
βββββ
ββββ
β
ββββββ
β
β
ββββββββββββ
β
β
β
ββ
ββ
βββββββββββ
β
β
β
βββββββββββ
β
β
ββ
β
βββββ
β
βββ
β
β
β
βββ
β
ββ
ββ
β
β
β
β
ββββββββ
ββ
βββββββ
ββββββ
ββ
β
β
ββββ
β
βββ
ββββββββββββ
ββ
β
β
ββββββββ
βββββ
ββ
ββββββββ
ββ
β
β
ββ
β
ββ
β
β
βββ
ββββ
β
ββ
βββββ
ββ
βββ
ββ
β
β
ββββββββββββ
β
βββββ
ββ
ββ
βββ
β
ββββββββββ
β
ββ
βββ
βββββ
β
ββββββ
β
β
β
β
βββββ
β
ββββββ
βββ
βββββ
ββββββ
ββ
ββ
β
β
ββββ
ββ
β
βββββββ
ββββββββ
ββ
βββββ
β
βββββββ
β
ββ
β
β
ββββ
β
β
β
β
βββ
β
β
β
ββ
β
β
β
βββ
βββ
β
ββ
βββ
ββββ
βββ
β
β
βββββββ
β
ββββββ
βββββββββββ
ββββββ
ββ
βββ
β
βββββ
β
βββ
β
ββββββββββββββββββββββ
β
βββββββ
βββββ
β
ββ
β
β
ββ
β
ββββββββ
β
βββββββ
β
ββ
β
β
ββ
ββββ
β
βββββ
ββββββββ
ββββ
βββ
β
ββββ
ββ
ββββββββββββββ
β
β
ββ
ββ
β
βββ
β
βββ
β
βββββ
β
β
β
ββ
ββ
βββββββ
ββ
ββ
ββ
βββ
β
ββ
β
β
ββββββ
β
ββ
ββ
ββββββ
β
ββββ
βββββ
ββββββββ
β
β
βββ
βββ
β
βββ
βββββ
ββ
βββ
ββ
β
β
ββββββ
β
βββ
β
ββββ
βββββββββββββ
β
βββ
β
ββ
β
βββββββββββββββββββββ
β
ββ
β
β
ββ
ββββββββββββ
β
βββββββββββββββ
β
ββββββββββ
β
ββββ
ββ
β
ββ
ββ
ββ
βββββββ
β
βββββββ
β
ββ
β
ββ
β
βββββ
β
β
β
βββ
βββββ
βββββ
ββββ
βββ
β
ββββ
βββ
βββββββββ
βββββ
β
βββ
ββββ
ββ
ββ
β
βββββββββ
ββββ
β
ββ
ββββ
β
βββ
βββ
β
β
β
βββ
β
β
β
βββββββββββ
β
β
ββ
β
β
β
β
β
ββ
β
β
β
ββββ
β
ββ
ββββ
ββ
β
ββββ
ββββββ
β
ββ
β
ββββ
βββββ
β
ββ
β
ββ
β
βββ
ββ
β
βββ
β
ββ
ββ
β
ββββββββββ
β
β
ββ
β
ββββββββββ
βββββββ
β
ββββ
β
ββββββββββ
β
ββββ
β
β
β
βββββ
β
ββββββββββ
β
β
ββββββ
ββββ
ββ
ββ
β
ββ
β
β
βββ
β
βββββ
β
ββββββββββ
ββ
β
ββββ
β
ββββββ
β
β
β
βββββββ
βββββββββ
β
ββ
β
β
β
β
βββββββ
βββββββββββββ
β
ββ
βββ
βββ
ββββββ
β
βββ
β
β
β
β
βββββββββ
ββββββ
β
ββ
β
ββββββββ
β
βββββ
β
ββββββ
ββ
ββββ
β
ββββ
ββ
ββ
β
ββββββ
β
β
β
ββββββββββββ
β
βββ
β
βββ
β
β
ββ
ββββ
β
βββ
β
ββ
β
β
β
βββ
ββ
βββ
ββ
ββ
ββ
β
ββββββββββ
β
β
ββ
βββ
β
β
β
ββ
βββββββ
β
β
ββββββ
β
βββ
βββββββ
β
β
β
βββ
β
ββ
β
β
βββββ
β
ββ
β
ββ
β
ββββββ
ββ
βββββ
β
β
ββ
βββ
ββββ
β
βββ
βββββββ
β
βββ
ββββ
β
ββ
β
ββ
β
ββββ
ββ
β
β
β
β
βββ
β
β
βββ
ββββ
β
ββββ
ββββ
ββββ
βββ
βββββ
β
β
β
β
β
β
ββ
β
ββββββ
ββββββββ
β
ββ
β
βββ
β
ββββ
ββ
ββββ
β
ββββββββ
β
β
ββ
ββ
β
βββββ
β
βββ
β
ββ
β
βββββ
ββ
ββ
β
β
β
β
β
βββ
ββ
βββ
ββ
β
β
β
ββββ
βββ
β
βββ
β
ββββ
β
β
β
β
β
ββββββ
β
β
β
ββ
ββ
ββ
βββ
βββββββ
ββ
ββββ
β
ββ
β
βββ
β
β
ββ
β
β
ββ
βββββ
β
βββββ
β
βββββ
β
β
βββ
β
β
β
β
ββββ
β
ββ
β
β
ββ
β
β
β
β
ββββ
β
βββ
β
ββ
β
ββ
ββ
βββ
ββββββ
β
ββ
β
β
β
β
βββββ
β
β
β
βββββββββ
β
ββ
β
ββ
β
ββββββββββ
β
βββββββββ
β
β
ββ
βββββββββββ
βββ
β
β
ββ
βββββββ
ββββ
ββββ
β
β
β
ββ
β
β
ββ
β
β
β
βββββ
β
βββ
β
ββββ
βββ
β
β
ββββββ
β
ββββ
β
ββββ
β
ββ
ββ
ββ
β
β
βββ
ββ
ββ
β
β
β
ββ
βββ
β
β
β
β
ββββ
ββ
β
β
β
βββββββββ
β
βββ
β
ββββ
β
ββ
ββββ
β
ββ
β
βββ
ββ
β
βββ
β
β
β
βββ
β
β
ββββ
ββ
ββββ
β
β
β
βββββ
β
β
ββ
β
β
β
β
β
ββ
ββ
β
βββ
ββ
ββββββββ
β
β
β
ββββββββββββββββββββ
β
βββ
β
β
βββββββ
ββ
ββββ
β
β
β
ββ
βββ
ββββββββββ
β
ββ
βββ
β
ββ
βββββ
β
βββββ
ββ
β
β
βββββββ
β
β
βββ
β
β
β
β
β
ββββββββ
β
ββ
β
βββββ
ββ
β
β
β
ββ
β
ββ
ββββ
βββ
β
ββ
βββ
ββ
ββ
β
ββββ
ββ
βββ
ββββββ
β
β
β
β
ββ
βββ
β
βββββββ
ββββ
β
βββ
ββββ
β
βββ
β
β
β
βββ
β
β
β
β
β
ββββ
β
βββββ
β
β
β
ββββ
β
ββββββββββ
β
βββ
βββββ
β
β
βββ
β
βββββββ
β
βββ
βββββ
β
β
β
ββ
βββ
β
ββββββββ
β
ββ
β
β
βββ
βββ
β
β
ββ
β
β
β
ββ
ββ
β
β
ββ
ββ
βββββββββ
βββββ
β
β
β
β
ββ
ββββ
ββββ
ββββ
βββ
βββ
ββββ
β
ββ
β
β
β
β
βββββ
β
ββββ
β
β
β
ββββββββββ
β
β
β
βββββββββββββ
βββββ
β
ββββ
ββ
βββ
β
β
β
βββ
β
β
β
ββββ
β
ββββ
β
β
ββ
ββ
β
ββββ
βββ
βββββ
β
ββββββ
β
β
β
βββββββββββββββ
βββ
β
β
β
ββ
β
β
β
ββ
ββ
β
βββ
β
β
ββ
ββββββββ
ββ
β
β
ββ
β
ββ
β
β
βββββ
ββ
ββββββ
ββββββ
ββββββββββββββ
ββ
ββββββββ
ββββ
β
βββββ
β
β
ββ
ββββ
β
βββ
β
β
ββββ
β
β
ββ
β
ββββ
β
ββββ
ββ
β
β
β
ββββ
βββ
β
β
ββββββ
β
ββββββββββ
β
β
β
ββ
β
ββ
β
β
β
β
ββ
ββββββ
βββ
ββ
β
β
ββ
β
βββ
ββββ
βββββ
β
β
ββ
β
β
β
ββ
βββββββββ
β
ββ
ββ
ββ
β
βββββ
β
β
β
β
β
βββ
β
β
ββββ
ββ
β
β
βββ
β
βββ
β
ββββ
ββ
ββ
β
βββ
β
β
βββ
β
ββββ
β
ββββββββββ
β
ββ
β
βββββββ
β
ββ
β
β
ββ
βββ
β
ββ
βββ
β
ββ
βββββ
βββββββββ
ββ
ββ
β
β
ββ
β
ββββββ
ββββββββ
βββββ
β
β
β
βββββββ
ββββββββ
β
β
ββ
β
β
β
ββ
β
ββββββ
βββββββββ
β
ββββ
βββ
β
β
β
β
βββ
β
ββ
β
β
β
β
ββ
β
βββ
β
β
βββ
β
βββ
ββ
β
βββββ
β
β
ββββ
β
ββββ
ββββ
β
β
ββ
ββββββ
β
ββββ
β
βββββ
β
β
β
β
ββββ
β
βββ
β
ββ
ββ
β
ββ
β
ββ
β
β
β
βββ
βββββ
β
βββββββββ
βββ
β
β
ββ
βββ
β
βββββββββββββ
β
βββββββ
β
β
ββ
βββ
ββ
β
β
β
βββββββββββ
β
β
β
β
βββ
β
βββ
β
β
β
ββ
β
ββββββ
β
βββ
β
β
ββββββ
ββ
βββββ
βββ
ββ
β
βββ
ββββ
β
ββββββ
ββββ
ββ
β
βββ
βββββββββ
β
β
β
β
β
ββββββββββββββ
β
ββ
ββββ
ββ
β
ββ
β
β
β
β
β
ββ
β
β
ββ
β
β
β
βββββββββββββββββββ
β
βββ
β
βββ
βββββ
ββ
β
β
ββ
ββββββ
ββ
ββββββ
β
βββ
β
βββββββ
βββββββ
ββ
β
ββββββββ
β
β
β
β
β
β
βββ
βββββ
βββ
β
ββ
β
βββ
β
β
β
ββ
ββ
βββ
ββ
βββ
ββ
β
ββ
ββ
β
ββ
ββ
β
β
β
βββ
β
ββββββ
βββ
βββ
β
ββ
β
β
βββ
ββ
β
β
β
β
βββ
ββ
βββ
ββ
ββββββ
β
β
βββββ
ββ
ββ
β
β
ββ
β
β
β
β
ββ
ββ
β
ββββββββ
βββ
βββ
ββββββ
β
βββββ
ββββ
β
β
ββββββββ
βββ
β
βββ
ββββ
β
β
βββββββ
β
β
β
βββββββββββ
β
βββ
ββββ
β
β
βββββββββββ
β
ββββββ
β
ββ
ββββ
β
βββββ
βββββ
β
ββ
βββββββ
β
βββββ
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
βββ
β
β
β
ββ
β
β
β
β
β
β
β
ββ
βββ
ββββ
ββββββ
β
ββ
ββββ
ββββββββ
ββββ
βββ
β
βββββ
ββ
βββββ
ββββββ
β
βββββ
ββ
ββ
β
βββββββ
ββ
β
β
ββββ
ββ
βββββ
ββββββββ
ββ
ββββ
β
β
ββββ
β
βββββ
β
ββ
β
βββββ
ββ
ββ
βββββ
ββ
β
βββ
β
ββ
β
β
β
βββ
ββ
βββββββ
ββ
ββββ
βββββββ
β
βββ
βββββββ
β
ββββ
β
β
ββ
ββ
βββ
β
βββ
β
β
βββββββββββ
β
βββ
βββ
β
ββ
β
βββββββββββ
ββββββ
β
β
ββ
β
ββ
βββ
β
β
β
β
β
β
βββ
β
β
ββββ
β
β
β
β
ββ
β
ββ
βββββββ
β
β
β
ββββ
β
ββββββββ
ββ
β
ββββββββββββββ
ββββ
β
β
β
β
βββββββββ
β
ββββ
β
β
ββ
βββ
β
ββ
β
ββ
β
ββ
βββ
β
ββββββ
β
β
β
β
ββ
β
ββ
βββ
ββββ
β
βββββ
β
βββ
β
ββ
β
βββ
ββ
β
βββ
β
β
ββ
βββ
ββββ
β
ββ
βββββββ
ββ
β
β
β
ββββββββ
ββββ
ββ
ββ
β
β
β
ββββ
β
β
β
β
β
β
βββ
β
β
β
β
βββ
ββββββ
ββββ
βββββ
βββββββ
β
β
β
βββββ
β
ββββ
βββ
ββ
βββ
β
ββββ
ββ
ββ
β
ββββ
β
ββββββββ
β
β
β
βββββ
ββββββββ
βββββββββββββ
ββ
ββ
β
β
βββ
βββββββββββ
β
β
β
ββ
β
ββ
ββ
β
ββββββββ
β
βββ
β
βββ
β
βββ
β
β
β
β
βββ
ββ
ββ
β
β
β
βββ
β
βββ
β
ββ
βββ
βββββββ
ββ
βββββ
ββ
β
β
βββββ
β
βββββ
β
ββ
β
β
β
βββββ
β
ββββ
ββββ
β
βββββ
β
ββββββββ
ββ
ββ
β
β
βββ
βββ
βββββββββββ
β
β
βββββ
ββ
βββββββ
β
ββ
βββ
β
β
ββββββββ
ββββ
ββ
βββ
β
β
β
ββ
β
ββ
β
β
β
β
β
ββββ
ββββββββ
ββββ
ββ
βββββ
β
β
ββ
βββ
β
βββ
βββββββ
ββββββ
βββββββ
βββ
ββ
ββ
βββββ
β
β
βββ
β
ββββββ
βββ
β
β
ββββ
ββ
β
ββ
βββ
β
ββ
βββββ
ββββ
ββ
ββ
β
β
βββ
β
ββ
ββ
β
ββββ
β
β
β
β
βββ
ββββ
β
β
ββββ
ββββ
β
ββ
β
β
βββ
β
β
β
β
ββ
β
βββ
β
βββββ
β
β
βββββ
β
ββββ
β
β
β
β
βββββ
βββββββββ
β
ββββββββββ
β
β
β
β
β
β
ββββ
βββββββ
β
βββ
β
βββββββ
β
βββ
β
β
β
β
β
β
ββββ
β
β
ββ
β
βββββββββ
β
ββββ
β
βββββ
β
ββββββ
ββ
ββββββββββ
β
ββββ
β
βββ
β
βββ
ββββββ
β
ββ
β
β
ββ
ββββ
β
ββββββββ
β
β
β
β
β
ββ
ββ
ββ
βββ
βββββββββββββ
ββ
βββ
β
βββββ
ββ
ββββββ
β
ββββββ
β
βββββββββββββββββββ
β
β
βββββββββ
β
ββββββββββββ
βββ
ββββ
β
ββββ
ββ
ββββ
β
ββ
β
βββ
β
ββ
β
ββββ
β
ββ
ββ
β
βββββββββββββββ
β
ββ
β
β
ββ
β
β
βββ
β
βββ
ββ
β
β
β
ββ
βββ
β
β
β
ββ
ββββ
ββ
ββ
β
β
β
β
ββ
βββββββββββββ
ββ
βββ
βββ
ββ
β
β
ββ
βββββ
β
βββ
β
β
β
ββ
β
βββ
β
βββ
β
βββ
βββ
β
ββ
βββββ
βββββββ
ββ
βββββ
ββ
β
β
ββββ
β
βββββββββ
β
β
β
βββββββ
ββ
β
ββββ
ββ
βββ
ββββββ
ββββ
ββ
β
βββββββββββββ
ββ
ββ
βββ
ββ
ββ
ββ
ββ
βββββββ
β
βββββ
β
ββ
β
βββββ
βββ
β
β
β
ββββββββ
ββ
β
β
βββββ
β
βββββ
β
β
βββ
β
ββ
ββ
β
ββββ
β
ββββββββ
βββββ
β
βββββ
β
βββββ
β
β
ββ
β
β
β
β
β
β
β
β
β
βββ
β
ββ
β
β
ββ
ββββββββ
β
β
ββββββββ
β
β
ββββ
βββ
β
β
β
ββ
β
βββ
βββββ
ββ
βββββββ
βββββββ
β
βββββ
ββ
ββββββββ
β
β
β
βββ
βββββββββββββ
β
βββββββββββββ
β
βββββββββ
ββ
βββββ
βββ
ββββ
ββ
β
βββ
ββββββ
β
ββ
βββββ
β
β
βββ
βββ
β
ββ
β
β
ββ
ββ
β
ββ
ββ
ββ
βββ
β
βββββββββ
β
ββββ
ββ
βββββ
β
β
ββββββββ
β
β
βββ
ββ
β
ββ
β
β
β
βββββ
βββββββββ
β
β
ββ
βββ
βββββ
β
ββββββββ
β
β
β
β
β
β
β
βββ
ββββ
β
ββ
β
ββ
ββ
β
ββ
β
β
ββββ
β
ββββ
βββ
β
β
β
ββββ
ββ
β
ββ
β
ββββ
β
β
β
ββββ
ββββ
ββ
ββ
β
β
βββ
β
ββββββ
β
β
β
β
β
β
β
β
β
βββββ
β
β
β
βββ
β
βββ
β
β
ββ
β
β
βββββ
β
βββ
β
ββ
β
ββ
ββββ
β
β
β
β
β
β
βββ
β
ββββ
ββ
ββββ
ββ
ββ
β
ββ
βββββ
β
ββββββββββββ
β
βββββββββ
β
ββ
β
ββ
β
βββ
βββββ
β
β
β
ββ
β
ββ
βββββ
ββ
ββββ
β
ββββββ
β
βββ
βββ
β
βββ
ββ
β
ββ
ββββ
β
βββββ
β
βββββ
ββββββββ
βββ
β
ββββ
ββ
βββ
βββββ
βββ
βββββββββ
β
β
ββββββββββββββ
ββ
β
β
ββ
β
β
β
β
β
β
βββ
β
βββββ
ββ
β
β
β
β
β
β
β
βββββββββββββββ
β
ββββ
β
β
βββ
ββββββββ
β
β
ββββ
ββββ
β
βββββ
β
βββ
β
βββββββ
βββ
β
ββ
ββ
β
β
β
β
βββ
ββ
βββ
β
ββ
β
βββββββ
βββ
βββββ
β
ββ
ββββββ
β
β
β
ββββββ
β
βββββββββββ
β
β
βββ
β
βββββ
βββ
β
β
ββββββββ
ββ
ββββββββ
β
β
βββββββββ
β
β
β
ββ
ββ
βββ
β
ββ
β
β
β
β
βββ
βββββββββ
β
βββ
β
βββ
β
ββββ
ββ
ββ
ββ
βββββββ
β
β
ββ
β
β
β
β
β
βββββββββββ
βββ
β
βββββ
β
βββββββ
β
βββ
ββ
β
βββ
ββββ
β
βββ
ββββ
β
ββββ
β
β
ββ
β
ββ
β
ββββ
β
ββββ
β
βββββββββββββββββ
β
β
β
β
ββββ
ββ
β
β
β
β
β
ββββ
β
β
β
β
β
ββ
β
ββ
βββββ
ββ
β
β
ββ
ββ
β
β
βββ
β
ββββ
β
β
β
ββββββ
ββββ
β
β
β
ββββ
β
ββ
β
β
β
ββββ
βββββ
ββ
ββββββ
ββ
ββββββ
βββββββ
β
βββββββββββββ
β
ββ
β
β
ββββ
β
ββ
ββββ
β
β
β
ββββ
β
ββββ
β
βββ
β
β
ββββ
β
ββββββ
β
βββββ
β
βββββ
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββ
β
β
β
β
β
β
β
βββ
β
ββ
β
β
ββ
β
β
β
ββ
β
β
βββ
βββββ
βββββ
βββββ
β
β
ββ
β
β
β
βββββ
β
β
ββ
β
ββ
β
βββββββββββ
ββ
ββ
ββ
ββββ
βββ
ββ
β
βββββ
β
βββ
β
β
ββ
β
ββ
βββ
ββ
ββ
ββββββ
β
βββ
β
β
βββββ
β
β
β
β
β
ββββ
β
ββ
β
β
βββ
β
βββββββ
β
βββββββ
βββββββ
ββ
β
β
β
ββββββββ
β
βββββββ
β
βββ
βββ
β
βββββββββββ
β
ββ
ββββββ
β
βββ
β
β
ββ
ββββββββββ
βββ
β
βββ
ββ
β
β
β
ββββ
ββ
β
βββββ
β
β
β
βββ
β
ββ
β
β
βββββββ
β
βββββ
β
βββ
β
β
β
βββ
ββ
ββββββββ
β
ββ
β
β
β
β
β
βββββββ
β
ββ
ββββββ
β
βββββ
β
βββββββββ
β
β
ββ
ββ
ββββ
β
ββ
ββ
ββ
β
β
ββββββββ
β
β
βββββ
β
β
ββ
β
βββ
β
βββββββ
β
β
ββ
ββββββββββββββ
β
ββ
ββββββββββββ
β
βββ
β
β
β
β
β
ββ
β
βββ
βββ
ββ
β
βββββ
β
ββ
ββββ
β
β
β
β
β
ββ
β
ββββ
βββββ
β
βββ
β
β
β
β
β
ββββ
ββ
β
β
β
β
β
βββ
β
ββ
β
βββ
ββββββββ
β
ββ
β
β
βββββββ
β
β
β
βββ
βββ
β
ββ
ββ
βββ
β
β
β
ββ
β
β
β
βββ
β
βββ
βββ
ββ
β
β
β
ββ
β
ββ
β
βββββ
β
βββ
ββ
β
βββ
ββ
β
ββ
β
ββββββ
β
β
β
βββ
βββββ
ββ
ββββββ
ββββββ
β
ββ
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
β
βββ
ββββββ
β
ββββ
β
β
β
β
ββ
ββ
βββββ
βββββββ
β
βββββββ
β
βββ
β
β
ββββ
β
ββ
β
βββββ
β
ββ
ββ
βββ
ββββββββ
β
ββ
β
β
β
β
βββββββββ
β
β
β
β
β
βββ
β
β
ββ
β
β
βββ
β
ββββββ
βββ
ββββββ
β
β
βββββ
β
ββ
β
βββββββββββ
βββββ
β
ββ
β
β
β
ββ
ββ
β
ββ
βββ
β
ββ
ββ
ββββββ
βββ
β
βββ
β
ββββ
β
ββ
βββ
β
βββββ
β
ββββββ
β
βββ
βββ
β
βββββββββββββββ
β
β
ββ
β
βββββ
ββ
β
β
ββββ
β
ββββ
βββ
β
βββββ
ββββββββ
ββββ
ββ
β
β
β
βββββ
β
β
ββββββ
ββ
βββ
ββ
β
β
β
ββ
βββ
βββ
ββββ
β
βββ
β
β
β
β
β
β
ββ
βββ
ββββββ
β
ββββββ
β
βββββ
ββ
ββ
β
ββ
βββ
β
βββββββββββ
βββ
β
β
βββ
ββ
β
β
β
ββ
β
βββ
β
βββ
β
β
ββ
β
β
ββ
ββββ
β
βββββ
ββ
ββ
β
ββ
ββ
β
βββββββββ
βββ
β
βββββββββ
βββββ
βββ
β
β
ββ
β
β
βββ
βββ
β
βββ
β
ββ
ββββ
ββββββ
βββ
β
ββ
βββββ
β
β
β
βββββββ
ββ
β
β
ββ
βββ
ββ
ββ
ββ
β
β
β
βββ
β
β
βββ
βββββ
ββ
β
ββ
β
βββ
β
β
ββββββ
β
ββ
β
ββββ
ββ
ββ
β
β
β
βββββ
β
β
ββ
β
β
ββββββββββ
β
βββ
β
βββββββ
ββββ
β
β
ββ
β
β
βββββ
β
βββββ
ββββ
β
ββ
βββ
ββ
ββ
β
β
β
β
ββ
β
ββ
ββ
ββββ
β
β
β
β
ββ
β
ββββββ
β
ββ
β
β
β
βββββββββββ
ββ
ββ
β
ββ
ββ
βββ
ββ
β
β
ββ
β
βββββ
β
βββ
ββββ
β
ββββ
β
β
β
ββ
ββ
β
ββ
βββββ
βββ
ββββ
ββ
β
ββ
β
β
βββββββ
ββ
β
ββ
ββββ
β
β
β
β
β
ββ
ββββ
ββββββββ
ββββββ
ββ
βββββ
ββ
βββββ
β
ββ
β
βββ
β
βββββ
β
ββ
β
ββββββ
β
βββ
β
βββ
β
β
βββ
β
β
β
βββ
β
β
β
β
ββββ
ββ
βββββ
β
β
ββ
ββββ
βββ
β
ββ
ββ
ββ
βββββ
β
ββ
β
β
β
β
βββββββ
β
ββ
ββ
ββββββ
β
ββ
β
ββ
β
βββ
β
β
β
β
βββ
βββ
β
βββ
βββ
β
ββ
β
βββ
β
ββββ
ββ
ββββββββ
ββ
ββββ
βββ
ββ
β
ββ
βββββββ
β
ββ
β
β
ββ
ββ
β
β
ββ
ββ
βββββ
β
β
β
β
βββ
β
βββββ
β
ββ
β
ββ
ββ
βββ
β
β
ββββ
β
β
ββββββ
ββ
β
ββββ
β
ββββ
ββ
β
βββββββββββ
β
β
ββ
ββββ
βββ
β
ββ
β
ββββββββ
β
β
β
β
β
β
βββββ
ββ
βββ
β
ββββ
β
β
ββ
ββββ
β
ββββ
β
βββ
β
β
ββ
β
β
ββ
β
β
β
ββ
ββββ
β
βββ
β
βββ
ββ
βββ
β
β
β
β
βββ
ββ
βββ
β
βββ
β
ββ
ββ
ββ
βββ
ββ
ββββ
ββ
β
ββββ
βββββββ
β
βββ
β
β
β
ββββ
β
β
ββββββ
ββββββ
β
βββββββββββ
β
β
ββ
βββββ
β
βββββββββ
βββββββ
ββ
βββββ
β
ββββ
β
βββ
βββ
ββ
βββββββββββββ
β
β
β
ββββ
β
ββ
β
ββββ
β
β
βββββββ
β
β
ββββ
β
ββββββ
β
ββ
ββ
ββββββ
β
ββ
β
ββ
ββ
β
β
β
β
βββ
ββ
ββ
ββ
β
β
ββ
β
ββββ
ββββββ
β
ββ
ββ
β
ββ
βββ
βββ
ββ
β
β
β
ββ
βββ
β
ββββββ
ββ
βββββββββ
β
βββββββ
β
ββββ
β
ββ
β
βββββ
ββββββ
βββ
βββββ
β
ββ
β
β
ββββββββ
βββββ
β
β
β
β
β
βββββββββββββ
β
βββββ
ββ
ββ
β
β
β
ββββββββββββββ
ββββββββββββ
ββ
β
βββ
β
β
ββββ
ββ
β
βββββββ
βββ
β
β
β
β
β
β
βββββ
ββ
ββββ
ββββ
βββββββ
β
ββ
β
β
β
β
β
βββ
β
ββββ
β
βββ
ββββ
ββ
β
β
β
β
ββββββ
β
β
ββ
β
ββββββββ
βββββ
β
βββββ
β
βββββ
β
β
β
ββ
βββ
ββββ
ββ
ββ
β
β
ββ
β
β
βββ
ββ
βββββββ
β
ββ
β
β
β
ββ
β
β
β
β
β
β
β
βββ
β
β
β
ββββββ
β
β
β
β
ββββββ
β
βββ
ββ
ββββββ
βββββ
βββββ
β
β
ββ
β
β
ββββββββββ
β
β
βββββββ
βββ
ββββββ
β
βββββββββ
ββ
β
ββββββ
β
βββββββ
ββ
βββββββ
βββ
β
βββ
β
βββ
ββββββββββ
β
βββ
β
ββ
β
ββββββ
β
β
ββ
ββ
β
β
β
β
ββ
β
βββ
βββ
ββββ
β
ββ
β
β
β
β
βββ
β
ββββββββ
βββββ
β
ββ
β
β
β
βββ
β
β
β
β
β
β
β
β
β
βββββ
ββ
β
β
ββββ
ββββ
ββ
βββββββ
β
βββ
βββββββββ
β
β
β
β
ββ
β
ββ
β
β
β
βββββββ
ββββββ
ββββββββββ
ββ
ββ
β
βββββ
βββ
β
β
β
βββββ
ββ
β
βββ
β
β
β
ββββββ
β
ββββ
β
ββ
β
β
β
β
ββ
βββββ
β
β
β
ββββββββββββββ
ββ
β
β
ββ
ββ
β
ββ
β
β
β
β
β
βββββ
β
βββββ
ββ
ββββ
ββββ
β
β
β
ββββ
β
β
β
ββ
β
ββββ
β
βββ
β
ββ
ββββββ
βββ
β
βββ
β
ββββββ
βββ
βββββ
β
β
β
β
β
β
βββ
β
β
βββ
β
β
β
ββββββ
β
β
βββ
ββ
β
β
ββββ
β
β
β
β
β
β
β
β
β
β
β
β
ββ
βββ
βββββββ
β
ββ
β
β
ββββββββ
ββ
β
β
β
ββββ
β
ββββββββββ
β
ββ
βββ
β
ββ
βββ
β
β
β
ββββββ
β
β
ββ
β
βββββββββββ
β
β
β
ββ
ββ
β
β
β
ββ
β
β
β
ββββ
βββ
β
β
β
β
β
ββββ
β
β
β
β
β
βββ
β
βββ
β
ββ
β
ββ
β
β
β
βββ
β
β
ββββ
β
ββ
β
ββ
βββββ
ββ
β
β
ββββββ
ββ
ββ
βββ
β
β
βββββ
β
β
β
β
β
β
β
β
β
β
ββ
β
βββ
β
β
β
β
β
β
ββββββ
β
βββ
ββ
βββββ
ββ
ββ
β
β
ββββ
β
βββ
β
ββββββ
β
β
βββ
β
ββββ
ββ
ββ
β
ββ
β
β
β
β
β
β
ββ
β
ββ
β
ββ
ββ
ββββ
ββ
ββ
β
ββ
β
β
ββββ
β
βββββ
β
ββ
β
ββ
β
βββ
β
β
ββ
βββββ
β
β
ββββ
ββ
ββββ
β
β
β
βββββββ
β
β
βββ
ββββββββ
β
ββ
ββ
βββ
β
β
β
βββββ
ββ
βββ
β
ββ
βββ
β
ββ
βββββββ
β
β
ββββββββββββββββββ
ββ
β
βββββββββ
ββ
ββ
βββββ
β
βββ
ββ
ββ
ββββββ
β
ββββ
β
ββ
β
βββββββββ
ββββ
ββ
ββ
β
ββββ
β
ββββ
β
ββ
β
β
βββ
β
ββ
β
β
βββββ
βββ
β
βββββββββββ
β
βββ
β
ββββ
β
ββ
β
ββ
ββ
ββ
β
ββ
βββ
βββββββββ
β
ββββ
β
β
βββββ
β
βββββ
β
ββββ
β
β
βββ
β
β
ββ
ββ
β
βββββββ
ββ
β
β
β
β
βββ
ββββ
ββ
βββββββββ
β
β
βββ
β
βββ
ββ
β
ββββ
ββ
ββββ
β
β
ββββ
β
ββ
β
ββ
ββ
ββ
β
β
β
βββββ
β
β
β
βββ
ββββ
β
β
βββ
β
β
ββββ
β
βββ
β
βββ
β
β
β
ββ
β
ββ
βββββββββ
β
β
βββ
β
ββ
ββ
β
β
β
β
βββββββββ
βββ
β
βββββββββ
ββ
β
ββ
ββ
ββ
ββ
ββ
ββββ
β
β
β
β
ββ
β
βββ
β
β
β
ββ
β
βββββββ
βββ
βββ
β
β
β
ββ
βββ
βββ
βββ
β
ββ
β
βββββββ
βββββ
ββ
β
β
ββ
βββ
βββ
β
βββ
β
ββ
β
ββ
β
βββ
βββ
β
ββ
β
β
β
ββ
β
β
β
β
ββββ
ββ
ββ
ββββ
ββ
β
β
ββ
β
β
ββββββββββ
β
β
β
ββββββ
β
β
β
β
β
βββββ
β
ββ
β
ββ
ββ
βββ
β
βββββββ
ββ
β
β
ββ
βββ
ββββββββ
ββ
ββ
β
ββββ
ββββββββββββ
β
β
ββ
β
β
ββ
β
β
βββ
β
βββ
β
β
ββ
ββββ
β
β
βββ
β
β
βββββββββ
ββ
β
β
ββββ
β
β
ββββ
β
β
ββ
β
β
β
βββ
β
βββββ
β
β
β
β
βββββ
β
ββββββ
β
β
βββ
β
ββ
β
ββββββ
β
β
β
βββ
βββ
βββββββ
β
β
β
ββ
βββ
ββ
β
βββ
β
ββ
ββββ
βββ
β
β
βββ
β
ββ
β
β
βββββββββ
β
ββ
ββ
ββββ
ββββ
β
βββββ
ββββ
ββ
βββ
βββββββββββ
ββ
β
β
β
βββ
ββ
β
β
ββββ
βββ
β
β
βββ
β
β
β
ββ
βββ
ββ
β
ββ
β
β
βββ
β
β
β
β
β
β
β
β
ββββββββββ
β
βββββββ
β
βββ
βββββ
β
β
β
ββ
β
β
ββ
β
ββ
ββ
β
β
β
β
ββ
β
β
β
β
β
ββ
β
ββ
β
β
β
βββ
β
β
β
ββ
βββ
β
β
β
βββββ
βββ
β
β
β
β
β
β
ββ
β
β
β
β
β
ββ
β
β
β
βββββββββ
β
βββ
β
β
ββ
β
β
β
β
ββββββββ
βββ
β
β
ββ
ββ
β
β
βββ
βββββ
β
βββββ
β
ββββββ
ββ
βββ
βββββ
β
ββββββ
β
β
β
β
βββββββ
βββββββ
β
β
βββ
βββββββ
β
β
ββ
β
ββ
ββββββ
β
βββ
β
βββββ
β
βββββ
β
βββ
β
β
β
β
βββ
β
ββ
ββββ
βββββββ
βββ
βββ
β
ββ
β
β
ββ
ββ
ββ
ββ
βββ
ββ
β
ββββ
β
ββ
β
ββ
ββ
β
β
βββββββββ
βββ
β
β
β
ββββ
β
β
β
ββββ
β
βββββ
ββ
β
βββ
βββ
β
β
β
ββ
ββ
β
βββββββ
β
ββββββ
ββ
ββββ
β
ββ
β
β
β
βββββ
β
βββββ
βββ
β
βββββ
βββββ
β
β
βββββ
β
βββββ
ββ
ββ
β
β
ββ
ββββββ
β
β
ββ
β
β
ββ
β
β
ββ
ββ
β
βββ
βββββ
β
β
β
βββ
βββ
β
βββ
β
β
βββββββββββββββββ
ββ
β
β
βββ
β
β
β
β
ββ
β
β
βββ
ββ
β
β
βββ
β
β
β
β
ββββ
βββ
β
β
β
β
ββ
βββ
ββ
βββββ
βββββ
β
ββ
β
ββββ
β
β
β
β
ββ
β
ββ
βββ
βββ
β
β
βββ
β
β
βββ
ββ
β
ββββββββββ
ββ
β
ββ
β
ββ
ββ
β
β
βββ
β
ββ
β
ββββ
β
β
ββ
β
βββ
β
β
β
βββββ
βββ
βββ
β
β
ββ
β
β
βββ
βββ
β
β
βββ
βββββββ
ββββ
β
ββ
ββ
β
β
ββ
β
β
ββ
β
βββ
β
ββ
β
β
β
ββ
β
β
β
β
ββ
ββ
ββββ
βββ
β
ββ
β
β
ββ
β
β
βββ
βββββ
β
β
β
ββββββββ
βββ
β
ββ
ββββ
β
ββ
ββββββ
ββββ
β
β
ββββ
β
β
ββ
β
β
β
βββββββ
β
β
β
β
βββ
β
ββ
βββ
β
βββ
β
βββ
β
ββ
ββββββ
β
β
β
β
β
βββββ
β
β
β
ββ
β
β
β
β
β
β
ββββ
β
βββ
ββ
β
ββ
β
β
ββββ
β
β
ββ
βββ
β
βββ
β
β
β
βββββββ
β
ββ
ββ
β
βββ
β
β
β
β
ββ
ββ
βββ
ββββ
β
βββ
β
ββββ
βββββ
β
ββ
βββ
βββ
ββββ
β
β
β
β
β
β
β
β
βββββ
ββββββββ
β
β
β
βββββββ
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββ
ββββ
β
βββ
ββββββ
ββ
β
β
ββ
ββ
β
β
ββ
β
ββββββ
βββββ
β
β
β
βββββ
βββ
β
ββββ
β
β
βββ
β
β
ββ
β
ββ
β
β
β
β
β
βββ
β
β
β
ββββββ
β
βββββββ
β
ββββ
βββ
βββ
βββ
ββ
β
ββββ
β
ββ
β
β
βββ
ββββββ
β
β
βββ
β
β
β
ββββ
β
βββ
βββββ
βββ
β
β
β
β
βββββ
β
β
β
βββββββ
β
βββ
β
β
ββ
βββ
β
ββ
βββββ
β
β
β
β
β
ββ
βββββ
β
β
ββ
ββββ
β
β
β
β
ββ
β
βββββ
ββββββββ
β
ββ
β
βββ
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
ββββββββ
β
β
ββββ
β
β
β
ββ
ββββββ
ββββ
β
β
ββ
ββββ
β
ββββ
ββ
β
β
ββ
βββ
β
β
β
βββ
β
β
β
β
ββββ
βββ
β
β
β
β
β
β
β
ββ
ββ
β
β
β
βββββ
βββ
βββ
ββββ
β
β
ββ
ββββ
ββ
β
β
β
β
β
βββ
β
ββ
βββ
β
β
β
β
β
β
β
ββ
β
β
ββ
β
β
ββ
βββ
β
βββββ
β
β
βββββββββββββββ
βββ
ββ
β
βββ
β
β
ββ
ββ
ββββ
βββ
β
β
ββββ
β
β
β
β
β
ββ
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
ββ
β
β
βββββ
β
ββββββ
ββ
β
βββ
ββ
β
βββ
β
β
β
βββ
β
βββββββ
β
β
ββ
ββ
β
βββ
β
β
ββ
ββββ
ββ
βββ
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
βββββββ
β
β
β
β
βββββ
β
βββ
β
β
β
β
β
β
β
β
ββ
β
β
ββββ
β
β
β
β
β
β
β
ββββ
β
ββ
β
β
ββ
ββ
β
ββ
β
βββ
βββββββββ
β
ββ
β
ββ
βββ
β
β
β
βββ
β
βββββ
β
β
ββββ
ββββ
β
β
β
ββ
β
ββ
β
βββ
β
β
β
ββ
ββββββ
β
β
ββ
β
ββββββ
β
β
β
ββββββββ
β
β
β
β
ββ
βββ
β
βββ
ββ
β
β
β
βββββββ
β
β
β
β
ββ
β
ββββ
β
β
ββ
β
β
β
β
β
β
β
ββββββ
β
β
β
ββ
βββ
β
β
βββ
ββββββ
βββ
β
ββ
β
ββ
β
β
βββββ
β
β
βββ
β
ββββ
β
ββ
β
β
β
β
β
ββ
β
ββ
β
β
β
β
ββ
β
ββ
ββββββ
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
ββββ
β
β
β
β
ββ
ββ
β
βββ
β
ββ
β
ββ
β
ββ
ββ
βββ
ββ
β
ββββ
ββ
β
ββββββ
β
ββ
βββ
ββ
βββ
β
β
ββββ
β
ββββ
β
β
βββ
β
β
ββββ
β
β
βββ
β
β
ββ
βββββ
β
β
ββββ
β
β
β
ββ
β
βββ
β
ββββββ
ββ
β
ββββ
β
β
β
ββ
β
ββ
βββ
βββββ
β
βββ
ββ
β
β
β
β
β
ββ
ββ
β
β
β
ββ
β
β
β
β
ββ
β
β
ββ
β
β
ββ
β
ββ
β
ββ
β
β
β
β
ββ
βββ
β
β
β
ββββ
β
β
ββ
ββ
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
β
β
β
β
β
βββ
β
βββββ
β
ββ
ββ
β
β
ββ
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
ββ
ββ
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
ββ
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
ββ
β
βββ
β
β
ββ
β
β
β
β
β
β
ββ
β
ββ
β
ββ
β
ββ
ββ
ββ
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββ
β
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
ββ
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
ββ
β
β
β
βββ
β
β
β
β
β
β
βββ
β
β
β
β
β
β
β
β
β
β
ββ
β
ββ
β
β
β
β
β
β
βββ
ββ
β
β
β
βββ
ββ
ββ
β
β
β
β
β
β
ββ
ββ
βββ
β
β
β
ββ
β
β
βββ
β
βββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
β
β
ββ
β
β
β
β
β
β
ββ
β
β
β
βββ
β
β
ββ
β
β
ββ
β
ββ
β
β
ββ
β
β
β
β
β
ββ
β
β
β
β
100
300
1,000
3,000
10,00
030
,000
100,0
00
Cascade size
Stru
ctur
al v
iralit
y
... but fails to account for the variance in structure given size
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 56 / 62
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 57 / 62
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 57 / 62
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 57 / 62
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 57 / 62
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 57 / 62
Information diβ΅usionSummary
β’ Most cascades fail, resulting in fewer than two adoptions, onaverage
β’ Of the hits that do succeed, we observe a wide range ofdiverse diβ΅usion structures
β’ Itβs diοΏ½cult to say how something spread given only itspopularity
β’ βThe structural virality of online diβ΅usionβ, Anderson, Goel,Hofman & Watts (Under review.)
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 58 / 62
Outline
Search predictions"Right Round"
Week
Ran
k
40
30
20
10
cccccccccccccccccccccccccccccccccccccccccc
Marβ09 Aprβ09 Mayβ09 Junβ09 Julβ09 Augβ09
BillboardSearch
Web diversity
Dai
ly P
erβC
apita
Pag
evie
ws
0
10
20
30
40
50
60
70
β
β
ββ
β
Over $25k
Under $25k
Black&
Hispanic
White
No College
Some College
Over 65
Under 65
Female
Male
Income Race Education Age Sex
Information diβ΅usion
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 59 / 62
Conclusion
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 60 / 62
Lessons learned
Data jeopardy
Regardless of scale, itβs diοΏ½cult to find the right questions to askof the data
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 61 / 62
Lessons learned
Hacking
Cleaning and normalizing data is a substantial amount of the work
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 61 / 62
Lessons learned
Modeling
Understanding human activity is often useful for detectingmalicious activity
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 61 / 62
Lessons learned
Modeling
Simple methods (e.g., linear models) work surprisingly well,especially with lots of (diverse) data
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 61 / 62
Thanks. Questions?
Also, weβre hiring:bit.ly/msrnyc_appsci
bit.ly/msrnyc_eng
@jakehofman (Microsoft Research) Computational Social Science November 6, 2014 62 / 62