Post on 13-May-2015
5 PILLARS OF CLOUDINESS
wayne wallscloud evangelist
alex brandtsr developer
Tuesday, July 23, 13
FIVE PILLARS OF CLOUDINESS
1 2 3 4 5
Parallel Modular Horizontal Agile Secure
2
Tuesday, July 23, 13
Pillar #1:Parallel
Tuesday, July 23, 13
1MRecords
Answer
Node
4
Tuesday, July 23, 13
1MRecords
Answer
Node1-250K
Node251K-500K
Node501K-750K
Node751K-1M
5
Tuesday, July 23, 13
6
1MRecords
Answer
Node1-250K
Node251K-500K
Node501K-750K
Node751K-1M
Tuesday, July 23, 13
6
1MRecords
Answer
Node1-250K
Node251K-500K
Node501K-750K
Node751K-1M
Tuesday, July 23, 13
7
Tuesday, July 23, 13
DATA
CALCULATIONMAP
CALCULATIONREDUCE
DATA
CALC
D
C
R
D
D
C
R
D
D
C
R
D
SHUFFLE
DATA
7
Tuesday, July 23, 13
8
Tuesday, July 23, 13
Thread 1
Deadlock
9
Thread 2
Tuesday, July 23, 13
Thread 1
Deadlock
9
Thread 2
Tuesday, July 23, 13
File A
Thread 1
Deadlock
9
Thread 2
Tuesday, July 23, 13
File A
Thread 1
Deadlock
9
Thread 2
Tuesday, July 23, 13
File A File B
Thread 1
Deadlock
9
Thread 2
Tuesday, July 23, 13
File A File B
Thread 1
Deadlock
9
Thread 2
Tuesday, July 23, 13
File A File B
Thread 1
Deadlock
9
File B
Thread 2
Tuesday, July 23, 13
File A File B
Thread 1
Deadlock
9
File B
Thread 2
Tuesday, July 23, 13
File A File B
File A
Thread 1
Deadlock
9
File B
Thread 2
Tuesday, July 23, 13
File A File B
File A
Thread 1
Deadlock
9
File B
Thread 2
Tuesday, July 23, 13
File A File B
File A
Thread 1
Deadlock
9
X
File B
Thread 2
Tuesday, July 23, 13
File A File B
File A
Thread 1
Deadlock
9
X
File B
Thread 2
Tuesday, July 23, 13
File A File B
File A
Thread 1
Deadlock
9
X X
File B
Thread 2
Tuesday, July 23, 13
Solution? Scheduling
Y
XA
B
A
B
Y
X
1
2
3
4
10
Tuesday, July 23, 13
1M records
Answer
Node
1M records
Answer
Node1-250K
Node251K-500K
Node501K-750K
Node750K-1M
11
Tuesday, July 23, 13
1M records
Answer
Node
1M records
Answer
Node1-250K
Node251K-500K
Node501K-750K
Node750K-1M
11
Tuesday, July 23, 13
12
Tuesday, July 23, 13
13
Tuesday, July 23, 13
pillar #2: modular
Tuesday, July 23, 13
ONE PERSON BUILDS A CAR
15
Tuesday, July 23, 13
CHASSIS ENGINE BODY INTERIOR
16
Tuesday, July 23, 13
CHASSIS ENGINE BODY INTERIOR
17
Tuesday, July 23, 13
CHASSIS ENGINE BODY INTERIOR
18
Tuesday, July 23, 13
CHASSIS ENGINE BODY INTERIOR
SEATS FLOOR
DASHBOARD
GASAND
BRAKE PEDALS
18
Tuesday, July 23, 13
CHASSIS ENGINE BODY INTERIOR
19
Tuesday, July 23, 13
CHASSIS ENGINE BODY INTERIOR
20
Tuesday, July 23, 13
defect
CHASSIS ENGINE BODY INTERIOR
20
Tuesday, July 23, 13
defect defect
CHASSIS ENGINE BODY INTERIOR
20
Tuesday, July 23, 13
defect defect defect
CHASSIS ENGINE BODY INTERIOR
20
Tuesday, July 23, 13
defect defect defectdefectCHASSIS ENGINE BODY INTERIOR
20
Tuesday, July 23, 13
defect defect defectdefect LEMONCHASSIS ENGINE BODY INTERIOR
20
Tuesday, July 23, 13
Pillar #3:horizontal
Tuesday, July 23, 13
2 GB2 CORES
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
VERTICAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
VERTICAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
VERTICAL
HORIZONTAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
VERTICAL
HORIZONTAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
SHARED NOTHING THROUGH STATELESSNESS
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
SHARED NOTHING THROUGH STATELESSNESS
22
8 GB4 CORES
Tuesday, July 23, 13
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
2 GB2 CORES
VERTICAL
HORIZONTAL
SHARED NOTHING THROUGH STATELESSNESS
2 GB2 CORES
22
8 GB4 CORES
Tuesday, July 23, 13
23
Tuesday, July 23, 13
24
Tuesday, July 23, 13
ONE SERVER NEAR
CAPACITY
24
Tuesday, July 23, 13
ONE SERVER NEAR
CAPACITY
ADD ANOTHER SERVER
24
Tuesday, July 23, 13
ONE SERVER NEAR
CAPACITY
ADD ANOTHER SERVER
25
Tuesday, July 23, 13
1234 MESSAGE QUEUE
26
Tuesday, July 23, 13
1234MESSAGE QUEUE
27
Tuesday, July 23, 13
1
2
3
4
28
MESSAGE QUEUE
Tuesday, July 23, 13
1
2
3
4
28
56789101112 MESSAGE QUEUE
Tuesday, July 23, 13
29
5678 9101112MESSAGE QUEUE
1
2
3
4
Tuesday, July 23, 13
29
5678 9101112
ADD MORE SERVERSBASED ON
MESSAGE QUEUE
MESSAGE QUEUE
1
2
3
4
Tuesday, July 23, 13
MESSAGE QUEUE
30
5
6
7
8
9
10
11
12
13141516
ADD MORE SERVERSBASED ON
MESSAGE QUEUE
1
2
3
4
Tuesday, July 23, 13
31
5678 9101112
13141516
ADD MORE SERVERSBASED ON
MESSAGE QUEUE
1234
MESSAGE QUEUE
Tuesday, July 23, 13
32
5678 9101112
13141516
ADD MORE SERVERSBASED ON
MESSAGE QUEUE
1234
MESSAGE QUEUE
Tuesday, July 23, 13
33
5678 9101112
13141516
ADD MORE SERVERSBASED ON
MESSAGE QUEUE1234
MESSAGE QUEUE
Tuesday, July 23, 13
33
5678 9101112
13141516
ADD MORE SERVERSBASED ON
MESSAGE QUEUE1234
MESSAGE QUEUE
DECREASE SERVERSBASED ON
MESSAGE QUEUE
Tuesday, July 23, 13
MESSAGE QUEUE
34
5678 9101112
13
14
15
16
ADD MORE SERVERSBASED ON
MESSAGE QUEUE1234
DECREASE SERVERSBASED ON
MESSAGE QUEUE
Tuesday, July 23, 13
Pillar #4:agile
Tuesday, July 23, 13
36
Tuesday, July 23, 13
Features In Progress Blocked Tabled Testing Complete
Feature A
Feature B
Feature C
37
Agile - why Monday
Tuesday, July 23, 13
Agile - why
Features In Progress Blocked Tabled Testing Complete
Feature A
Feature B
Feature C
38
Tuesday
Tuesday, July 23, 13
Agile - why
Features In Progress Blocked Tabled Testing Complete
Feature A
Feature B
Feature C
Feature E
Feature D
38
Tuesday
Tuesday, July 23, 13
Features In Progress Blocked Tabled Testing Complete
Feature AFeature BFeature C
Feature E
Feature G
Agile - why
39
Wednesday
Tuesday, July 23, 13
Features In Progress Blocked Tabled Testing Complete
Feature AFeature BFeature C
Feature E
Feature D
Feature F
Feature G
Agile - why
39
Wednesday
Tuesday, July 23, 13
Features In Progress Blocked Tabled Testing Complete
Feature A
Feature B
Feature C
Feature E
Feature D
Feature F
Feature G
Agile - why
40
Friday
Tuesday, July 23, 13
Features In Progress Blocked Tabled Testing Complete
Feature A
Feature B
Feature C
Feature E
Feature D
Feature F
Feature G
Feature H
Feature I
Agile - why
40
Friday
Tuesday, July 23, 13
Features In Progress Blocked Tabled Testing Complete
Feature A
Feature B
Feature C
Feature EFeature D
Feature F
Feature G
Feature H
Feature I
Agile - why
41
Friday - WK 2
Tuesday, July 23, 13
Agile - why
57%
14%
29%
Failed Successful Challenged
49%
42%
9%
Waterfall Agile
Source: The CHAOS Manifesto, The Standish Group, 2012
42
Tuesday, July 23, 13
43
Continuous Integration/Continuous Delivery
Tuesday, July 23, 13
44
Quality
Tuesday, July 23, 13
45
Release Cycle
Tuesday, July 23, 13
IMAGE: blogs.bu.edu
46
Tuesday, July 23, 13
47
Lint Testing
Test Driven Development
Functional Testing
Black Box
Integration Testing
Unit Testing
Behavior Driven Development
White Box
Smoke Testing
Acceptance Testing
Tuesday, July 23, 13
ENSURES QUALITY
47
Lint Testing
Test Driven Development
Functional Testing
Black Box
Integration Testing
Unit Testing
Behavior Driven Development
White Box
Smoke Testing
Acceptance Testing
Tuesday, July 23, 13
48
Lint Testing
Test Driven Development
Functional Testing
Black Box
Integration Testing
Unit Testing
Behavior Driven Development
White Box
Smoke Testing
Acceptance Testing
Tuesday, July 23, 13
ENSURES CONFIDENCE
48
Lint Testing
Test Driven Development
Functional Testing
Black Box
Integration Testing
Unit Testing
Behavior Driven Development
White Box
Smoke Testing
Acceptance Testing
Tuesday, July 23, 13
Pillar #5:secure
Tuesday, July 23, 13
50
Tuesday, July 23, 13
51
PRESENTATION
APPLICATION
STORAGE
Tuesday, July 23, 13
51
PRESENTATION
APPLICATION
STORAGE
INTERFACESECURITYACROSS ALL LAYERS
Tuesday, July 23, 13
52
Tuesday, July 23, 13
53
Host-based Authentication
Tuesday, July 23, 13
54
User Authentication
Tuesday, July 23, 13
55
Hybrid Cloud
Tuesday, July 23, 13
56
Trustworthiness as an Organization
Tuesday, July 23, 13
APPLICATION dissection
alex brandtsr developer
wayne wallscloud evangelist
Tuesday, July 23, 13
Data Schema
•articles• url
• text
• tags-index
• notations
• location
• note
• votes
• created_at-index
• etag
• parsed_at
•users• username-unique index
• name
• hash (md5)
58
•subscribers• uuid4-user
• uuid4-bookmark
• subscribed_at
Tuesday, July 23, 13
LB
FRONT END API
MQ
CONSUMER
LB
KEY/VALUESTORE
DOCUMENTSTORE
INTERNET
Routing
•Topic
•users
•articles
•Fanout
•article.sanitize
•article.references
Consumers
•create user
•password user
•article reference
•email user
•create article
•article sanitize
OBJECTSTORE
Tuesday, July 23, 13
LB
FRONT END API
MQ
CONSUMER
LB
KEY/VALUESTORE
DOCUMENTSTORE
INTERNET
Routing
•Topic
•users
•articles
•Fanout
•article.sanitize
•article.references
Consumers
•create user
•password user
•article reference
•email user
•create article
•article sanitize
OBJECTSTORE Parallel
Modular
Horizontally scalable
Agile
Secure
Tuesday, July 23, 13
LB
FRONT END API
MQ
CONSUMER
LB
KEY/VALUESTORE
DOCUMENTSTORE
INTERNET
Routing
•Topic
•users
•articles
•Fanout
•article.sanitize
•article.references
Consumers
•create user
•password user
•article reference
•email user
•create article
•article sanitize
OBJECTSTORE Parallel
Modular
Horizontally scalable
Agile
Secure
Tuesday, July 23, 13
LB
FRONT END API
MQ
CONSUMER
LB
KEY/VALUESTORE
DOCUMENTSTORE
INTERNET
Routing
•Topic
•users
•articles
•Fanout
•article.sanitize
•article.references
Consumers
•create user
•password user
•article reference
•email user
•create article
•article sanitize
OBJECTSTORE Parallel
Modular
Horizontally scalable
Agile
Secure
Tuesday, July 23, 13
LB
FRONT END API
MQ
CONSUMER
LB
KEY/VALUESTORE
DOCUMENTSTORE
INTERNET
Routing
•Topic
•users
•articles
•Fanout
•article.sanitize
•article.references
Consumers
•create user
•password user
•article reference
•email user
•create article
•article sanitize
OBJECTSTORE Parallel
Modular
Horizontally scalable
Agile
Secure
Tuesday, July 23, 13
KEY/VALUESTORE
OBJECTSTORE
LB
FRONT END API
MQ
CONSUMER
LB
DOCUMENTSTORE
INTERNET
Routing
•Topic
•users
•articles
•Fanout
•article.sanitize
•article.references
Consumers
•create user
•password user
•article reference
•email user
•create article
•article sanitize
Parallel
Modular
Horizontally scalable
Agile
Secure
Tuesday, July 23, 13
65
https://github.com/raxsavvy/margarine
Thank You!
@rackspace@waynewalls@hhoover@alunduil
Tuesday, July 23, 13