Create your own Social Network Mobile Application | Rapidsoft Technologies
Social Network Technologies
-
Upload
luigi-de-russis -
Category
Technology
-
view
394 -
download
0
description
Transcript of Social Network Technologies
![Page 1: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/1.jpg)
Social Network
Technologies
An overview Luigi De Russis
![Page 2: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/2.jpg)
Prerequisite
07/03/2014 Social Networks Technologies
2
Did you know…
the data center definition?
the difference between a logical server and
a physical server?
the difference between cache and
database access?
![Page 3: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/3.jpg)
Let’s think!
07/03/2014
3
Social Networks Technologies
![Page 4: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/4.jpg)
Social network
A “typical” web site?
07/03/2014
4
Social Networks Technologies
Traditional web
site
![Page 5: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/5.jpg)
Social network
A “typical” web site?
07/03/2014
5
Social Networks Technologies
Traditional web
site
In the beginning, probably…
![Page 6: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/6.jpg)
Social network
A “typical” web site?
07/03/2014
6
Social Networks Technologies
Traditional web
site
… but soon or later…
![Page 7: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/7.jpg)
Why?
![Page 8: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/8.jpg)
07/03/2014 Social Networks Technologies
8
What tools and technologies are we
using for building a website?
Web site: tools and technologies
![Page 9: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/9.jpg)
Web site: tools and technologies
JVM + JDK + J2EE
Tomcat (or similar)
MySQL (or similar)
PHP
Apache
MySQL (or similar)
07/03/2014
9
Social Networks Technologies
Java/JSP PHP
For example…
![Page 10: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/10.jpg)
Web site: tools and technologies
JVM + JDK + J2EE
MySQL (or similar)
Tomcat (or similar)
PHP
MySQL (or similar)
Apache server
07/03/2014
10
Social Networks Technologies
Java/JSP PHP
Typically, a vertical stack
(with one programming language)
![Page 11: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/11.jpg)
Now let’s try with these sites…
07/03/2014 Social Networks Technologies
11
![Page 12: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/12.jpg)
Now let’s try with these sites…
07/03/2014 Social Networks Technologies
12
PHP
MySQL
Apache
Rails
MySQL
Unicorn
Django
MySQL
Gunicorn
Django
PostgreSQL
Gunicorn
![Page 13: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/13.jpg)
Now let’s try with these sites…
07/03/2014 Social Networks Technologies
13
PHP
MySQL
Apache
Rails
MySQL
Unicorn
Django
MySQL
Gunicorn
Django
PostgreSQL
Gunicorn
Are you really, really sure?
![Page 14: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/14.jpg)
Not so sure?!
07/03/2014
14
Social Networks Technologies
https://github.com/twitter https://github.com/facebook
Let’s have a look at some pages and projects…
https://enginnering.twitter.com/opensource/projects
What did you notice?
https://code.facebook.com/projects/
![Page 15: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/15.jpg)
What did you notice?
tracing system, package manager, various servers, NoSQL database, caching system, etc.
code-related tools, code transformer, various servers, distributed file system, caching system, NoSQL database, etc.
07/03/2014
15
Social Networks Technologies
A lot of different components
![Page 16: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/16.jpg)
What did you notice?
Java, Scala, Ruby, C++, C, Objective-C, Shell scripting, Python, JavaScript
PHP, OCAml, C++, JavaScript, Python, Java, Objective-C, Processing, C, Ruby, Shell scripting, Haskell, Emacs Lisp, ActionScript
07/03/2014
16
Social Networks Technologies
A lot of different languages
![Page 17: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/17.jpg)
Now, have a look again at this…
07/03/2014
17
Social Networks Technologies
Impressive?!
![Page 18: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/18.jpg)
This is ONLY the tip of the iceberg…
![Page 19: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/19.jpg)
Social Network Characteristics
07/03/2014 Social Networks Technologies
19
Wildly popular over last few years
Facebook has more than 1 billion (monthly active) users
Twitter has more than 600M users
Distributed across the planet
Changed how content is created and consumed
Explosion of smartphones
photos and video are now easy to shoot and share
e.g., Facebook has more than 350M photos uploaded
each day
![Page 20: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/20.jpg)
Facebook as an example…
07/03/2014
20
Social Networks Technologies
![Page 21: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/21.jpg)
On December 2013…
07/03/2014 Social Networks Technologies
21
More than 3.9 trillion feed actions processed per day
More than 1 billion monthly active users
100 million search queries per day
More than 200 billion monthly page views
2,5 billion content items shared per day
2.7 billion ‘Likes’ per day
350 million photos uploaded per day
Over 500 TB new data ingested per day
757 million active users per day
![Page 22: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/22.jpg)
Growth Rate
07/03/2014 Social Networks Technologies
22
0
200
400
600
800
1000
1200
1400
Dece
mber
200
4
Dece
mber
200
5
Dece
mber
200
6
Ap
ril 20
07
Oct
ober
200
7
Augus
t 200
8
Janu
ary
2009
Feb
ruary
200
9
Ap
ril 20
09
July
20
09
Septe
mber
200
9
Dece
mber
200
9
Feb
ruary
201
0
July
20
10
Septe
mber
201
0
Janu
ary
2011
Feb
ruary
201
1
June
201
1
Septe
mber
201
1
Feb
ruary
201
2
Ap
ril 20
12
July
20
12
Oct
ober
201
2
Janu
ary
2013
Ap
ril 20
13
July
20
13
Oct
ober
201
3
Janu
ary
2014
Millions of Users
![Page 23: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/23.jpg)
07/03/2014
23
Social Networks Technologies
Each active user wants to write and upload something NOW
No perceptible delay is allowed, in any case
No matter how “big” the content is
Reality Check
Data and services have to be available 24/7, everywhere
![Page 24: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/24.jpg)
07/03/2014
24
Social Networks Technologies
Moreover, new features and applications are added
continuously to Facebook
Reality Check
No perceptible delay is allowed, in any case
Each active user wants to see and share something NOW
(she sees the recent status of her friends and pages, anyway)
No matter where the information is (geographically speaking)
![Page 25: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/25.jpg)
Reality Check
07/03/2014 Social Networks Technologies
25
“Rendering a single page of Facebook
involves hundreds of machines examining
tens of thousands of pieces of data from
dozen of services - all in real time.”
- from the Infrastructure page of the Facebook Newsroom
![Page 26: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/26.jpg)
Reality Check
07/03/2014 Social Networks Technologies
26
![Page 27: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/27.jpg)
The main problem
07/03/2014 Social Networks Technologies
27
Scalability
The ability of a system, network or process to handle a growing
amount of work in a capable manner or its ability to be enlarged
to accommodate that growth.
![Page 28: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/28.jpg)
The main problem
07/03/2014 Social Networks Technologies
28
Scalability
The ability of a system, network or process to handle a growing
amount of work in a capable manner or its ability to be enlarged
to accommodate that growth.
Affected by
several factors
![Page 29: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/29.jpg)
How to handle such situations?
07/03/2014
29
Social Networks Technologies
![Page 30: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/30.jpg)
Solution
07/03/2014 Social Networks Technologies
30
No standard solutions…
… each social network made different choices
strongly dependent from the original core
We are in the Cloud Computing realm
We are going to analyze briefly the Facebook
and the Instagram cases
![Page 31: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/31.jpg)
Facebook architecture at 100 feet
How to handle such situations?
07/03/2014
31
Social Networks Technologies
![Page 32: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/32.jpg)
Servers and Data Centers
07/03/2014 Social Networks Technologies
32
Facebook has 4 data centers
Prineville, Oregon
Forest City, North Carolina
Luleå, Sweden
Altoona, Iowa (in construction)
They build their servers and data centers from the ground up (efficiently)
Servers and data center design is open source
see The Open Compute Project (http://opencompute.org)
![Page 33: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/33.jpg)
Complex Infrastructure
07/03/2014 Social Networks Technologies
33
Large number of software components
Multiple storage systems
Multiple caching systems
Hundreds of specialized services
Failure is routine!
Keep things as simple as possible!
![Page 34: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/34.jpg)
Software Architecture
07/03/2014 Social Networks Technologies
34
![Page 35: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/35.jpg)
Web Tier
07/03/2014 Social Networks Technologies
35
Gather Data from the other Tiers
Runs PHP code
Widely used for web development
One single source tree for all the entire code
Same “binary” on every web tier box
![Page 36: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/36.jpg)
Web Tier
07/03/2014 Social Networks Technologies
36
At the beginning: Zend Interpreter for PHP
reasonably fast (for an interpreter)
rapid development
no recompiling
but, at scale, performance matters!
Then: HipHop compiler for PHP
400% faster
but slow down development
they add an HipHop interpreter
but compiler and interpreter sometime disagree
![Page 37: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/37.jpg)
Web Tier
07/03/2014 Social Networks Technologies
37
Now: HipHop Virtual Machine
The best of both worlds
9x increase in web request throughput
5x reduction in memory consumption
All this is open source
e.g., you can find HipHop Virtual Machine at
http://hhvm.com
![Page 38: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/38.jpg)
Storage Tier
07/03/2014 Social Networks Technologies
38
Multiple storage systems
MySQL
Hbase (NoSQL) - Messaging and Insight
Haystack (BLOBS)
BLOB: Binary Large Objects (Photos, Videos, Email
attachments, etc.)
large files, no updates/appends, sequential reads
![Page 39: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/39.jpg)
Cache Tier
07/03/2014 Social Networks Technologies
39
Memcache
speak only with the Web Tier
do one thing very well
improved performance by 10x
key-value store
![Page 40: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/40.jpg)
Cache Tier
07/03/2014 Social Networks Technologies
40
Tao
abstract the Storage Tier
in production for more than a 3-4 years
higher CPU load than memcache
used for the social graph
![Page 41: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/41.jpg)
Service Tier
07/03/2014 Social Networks Technologies
41
![Page 42: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/42.jpg)
Service Tier
07/03/2014 Social Networks Technologies
42
Example: News Feed
one of the hundreds of services at Facebook
Characteristics
real-time distribution
writers can potentially broadcast to very large audience
readers wants (and have) different and dynamic ways to filter data
The service should maintain an index and rank the data (in multiple ways)
![Page 43: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/43.jpg)
News Feed Service
07/03/2014 Social Networks Technologies
43
- write one location
![Page 44: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/44.jpg)
News Feed Service
07/03/2014 Social Networks Technologies
44
1000s of machines
leafs are in multiple sets, and each set has the entire index
Dealing with (daily) failures
hardware/software, server/network, intermittent/permanent, etc.
if a leaf is inaccessible, failover request to a different set
if an aggregator is inaccessible, “just” pick another
More leafs than aggregators
Reads are more expensive than writes
High network load between aggregator and leafs
fundamental to keep a full leaf set within a single rack on machines
![Page 45: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/45.jpg)
Software architecture
07/03/2014 Social Networks Technologies
45
![Page 46: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/46.jpg)
Instagram architecture at 100 feet
How to handle such situations?
07/03/2014
46
Social Networks Technologies
![Page 47: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/47.jpg)
Infrastructure
07/03/2014 Social Networks Technologies
47
Instagram follows some core principle when
choosing a system:
keep it very simple
don’t re-invent the wheel
go with proven and solid technologies when you can
It runs Ubuntu on Amazon EC2
![Page 48: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/48.jpg)
Software Architecture
07/03/2014 Social Networks Technologies
48
Web Tier
Django (Python) on more than 25 Amazon High-CPU Extra-Large machines
Gunicorn is the chosen Web Server
Storage Tier
PostgreSQL
Redis (key-value storage)
Amazon S3 for photos
Cache Tier
memcache
![Page 49: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/49.jpg)
Conclusions
07/03/2014
49
Social Networks Technologies
![Page 50: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/50.jpg)
Conclusions
07/03/2014 Social Networks Technologies
50
Most Social Networks starts like “traditional”
website
They change and/or evolve their hardware and
software infrastructure when:
users growth
some functionalities are added/revised
Scalability is a relevant problem…
They are complex entities and, sometimes, required
complex or innovative solutions
![Page 51: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/51.jpg)
References
07/03/2014 Social Networks Technologies
51
Instagram Engineering Blog, http://instagram-
engineering.tumblr.com
Facebook Newsroom, http://newsroom.fb.com
Facebook Investor Relations, http://investor.fb.com
HPCA 2012 Facebook Keynote,
http://www.ece.lsu.edu/hpca-
18/files/HPCA2012_Facebook_Keynote.pdf
![Page 52: Social Network Technologies](https://reader035.fdocuments.us/reader035/viewer/2022062307/555b856ed8b42ae81d8b4921/html5/thumbnails/52.jpg)
License
07/03/2014 Tips for Working Successfully in a Group
52
This work is licensed under the Creative Commons “Attribution-NonCommercial-ShareAlike Unported (CC BY-NC-SA 3,0)” License.
You are free:
to Share - to copy, distribute and transmit the work
to Remix - to adapt the work
Under the following conditions:
Attribution - You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
Noncommercial - You may not use this work for commercial purposes.
Share Alike - If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/