Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

Distribute the workload

Helgi Þormar ÞorbjörnssonDutch PHP Conference, Amsterdam, 19th May 2011

Thursday, 19 May 2011

Who am I?

VP of Engineering at Orchestra.io

Developer at PEAR

From Iceland

Developer at PEAR

From Iceland

@h on Twitter

Why Distribute?

Budget

Why Distribute?

Budget

Efficiency

Why Distribute?

Budget

Efficiency

Perception

Budget

Spend wisely

Budget

Spend wisely

Commodity servers

Budget

Spend wisely

Commodity servers

Low overhead, high yield

Budget

Spend wisely

Commodity servers

Low overhead, high yield

Cloud Computing (EC2)

Efficiency

10 small servers > 1 big

Venue Security

1000 people can exit quicker through 10 small doors than 1 big

1000 people can exit quicker

through 10 small doors than 1 big

1000 people can exit quicker

through 10 small doors than 1 big

1000 people can exit quicker through 10 small doors than 1 big

Perception

Defer intensive processes

Perception

Give instant feedback

Perception

Users keep on browsing

Perception

Users keep on browsing

“It all depends on how we look at things, and not how

they are in themselves.”

- Carl G. Jung

Chapter from Nature

Ant Colonies

Algorithms

Scheduling

Algorithms

Scheduling

Vehicle Routing

Algorithms

Scheduling

Vehicle Routing

Assignment

Algorithms

Scheduling

Vehicle Routing

Assignment

Algorithms

Scheduling

Vehicle Routing

Assignment

Algorithms

Scheduling

Vehicle Routing

Assignment

How do ants fit?

Strength in numbers

How do ants fit?

Strength in numbers

Work together

How do ants fit?

Strength in numbers

Work together

Size benefits them

Teamwork

When faced with a problem they will solve the problem as one.

What if they were bigger?

Types of Ants

Military

Types of Ants

Military

Types of Ants

Military

Tunnel diggers

Types of Ants

Military

Tunnel diggers

Food gatherers

How does this map to my application?

Colony = Application

Colony = Application Ants = Components

Ants do many different types of work to keep their colony running

Architect for Distribution

Characteristics

Decoupling

Characteristics

Decoupling

Elasticity

Characteristics

Decoupling

Elasticity

High Availability

Characteristics

Decoupling

Elasticity

High Availability

Concurrency

Decoupling

Application

DB API

Cache FE

Application

DB API

Cache FE

ApplicationDB API

Cache FE

ApplicationDB API

Cache FE

ApplicationDB API

Cache FE

ApplicationDB API

Cache FE

Elasticity

Cloud Computing

Load Balancing

HA Proxy

My Favourite

Monitoring

When do I need more servers?

Needs to be around from the start!

Keep records

Spot trends

Different types

Hardware Performance

Different types

Software Performance

Different types

Availability

Different types

Availability

Resourcing

Different types

Availability

Resourcing

Applications

ApplicationsNew Relic

CloudKick

ScoutApp

CloudKick

ScoutApp

Nagios

CloudKick

ScoutApp

Nagios

CloudKick

ScoutApp

Nagios

Circonus

Automation

Want to sleep easy at night?

Go out partying without worrying about getting a phone call?

Plug into your monitoring

Bringing together Monitoring and Elastic behaviour into one

beautiful whole!

Add some intelligence to add / remove servers as needed based

on current information.

This is why good monitoring is essential or this wouldn’t be

possible

Just make sure it doesn’t turn into...

Skynet!!Thursday, 19 May 2011

High Availability

Get a highly available and resilient setup by following a few

of those recommendations

Remember, even Google has outages

Benefits

Easy management

Benefits

Easy management

Ability to stop / start servers quickly

Benefits

Easy management

Responsibilities are separate

Benefits

Easy management

Quickly move to a new cluster

Benefits

Easy management

Quickly move to a new cluster

Reduced risk

What to avoid

Local Sessions

Store sessions in DB / Memcache

Solution

Local Memory

Networked Memcache

Solution

Local Files

Local Uploads

Writing to /tmp

Store on S3 or a networked FS

Solution

Serve up static files from CDNs

Solution

Servers can vanish at any given time

Internal APIs

Application

S3GFS FS

Internal Storage API

Application

MySQLMongo Cache

Internal DB API

Service Oriented Architecture

Sort of :-)

Eventually Consistent

CAP Therom

Consistency

Availability

Partition Tolerance

Consistency

All nodes see the same data at the same time

Availability

Node failures do not prevent survivors from continuing to

operate

Partition Tolerance

The system continues to operate despite arbitrary message loss

Consistency

Availability

Partition Tolerance

Queue Systems

Good for

Good forImage Processing

Distributed Logs

Data Mining

Distributed Logs

Data Mining

Mass Emails

Distributed Logs

Data Mining

Mass Emails

Intensive transformation

Distributed Logs

Data Mining

Mass Emails

Intensive transformation

Search

Common Tools

Gearman

Common Tools

Gearman

Hadoop

Common Tools

Gearman

Hadoop

Zero MQ (0MQ)

Common Tools

Gearman

Hadoop

Zero MQ (0MQ)

RabbitMQ

Common Tools

Gearman

Hadoop

Zero MQ (0MQ)

RabbitMQ

And many others!

Gearman

Your Client Code

Gearman Client API(C, PHP, Perl, MySQL UDF, ...)

Gearman Job Servergearmand

Gearman Worker API(C, PHP, Perl, Python, ...)

Your Worker Code

Your App Gearman

A Story!

Financial Software

3000+ Clients

Each one has 5 external data sources

Each data source is a web service

Ran every 6 hours every day

Gearman

Web Services Processing

But! That wasn’t enough

Job kicked off on login

Another Story!

CloudSplit

Near Real Time Cloud Analytics

Clients install logging agent locally

syslogd

Public API

Multiple Persistent Gearman Servers

Internal DB API

Agent syslogd

Load Balanced

Agent syslogd

Gearman

Load Balanced

Agent syslogd

Gearman

Load Balanced

PersistentThursday, 19 May 2011

Agent syslogd

Gearman

Worker

Load Balanced

Agent syslogd

Gearman

Worker

Internal API

Load Balanced

Agent syslogd

Gearman

Worker

Internal API

Load Balanced

Agent syslogd

Gearman

CouchDB

Worker

Internal API

Load Balanced

CouchDB Setup

Write vs Read

Writes

Multi Master setup

Writes

Multi Master setup

Replicated

Writes

Multi Master setup

Replicated

Deals with writes only

Writes

Multi Master setup

Replicated

Deals with writes only

Multi Master setup

Replicated from write cluster

Multi Master setup

Slaves handle website requests

Multi Master setup

Slaves handle website requests

Heavy Map/Reduce usage for data

Supervisord

phpadvent.org/2009/daemonize-your-php-by-sean-coates

Map/Reduce

Master gets a problem to solve

Breaks into multiple sub-problems

Distributed to multiple workers

A worker can take the same steps

Answer passed back to Master

Reduce

Takes in answers from the map workers

Reduce

Combines together to get an answer

Reduce

Combines together to get an answer

There can be multiple reducers

process petabytes of data in few hours on commodity server farm

CouchDB

Highly Concurrent

CouchDB

Highly Concurrent

Schema free, document based

CouchDB

Highly Concurrent

RESTful API

CouchDB

Highly Concurrent

RESTful API

Map/Reduce Views

CouchDB

Highly Concurrent

RESTful API

Map/Reduce Views

Easy Replication

Hadoop

Hadoop is a framework for running applications on large clusters of commodity hardware.

The Hadoop framework transparently provides applications both reliability and data motion

Uses Map/Reduce concept to farm out work

Distributed FS to handled node failure automagically

Join 2 datasets together of a significant size

500 GB worth of log files with a large location dataset

Async Message System

Thin and lightweight

High Performance

Simple

High Performance

Simple

Scalable

One socket can load balance to multiple end points

Multiple end points can be funnelled into a single socket

Handle deployments to multiple servers

Scale is an example of that

Mongrel2 is a web server that uses it in a similar way as fastcgi

Move around text (JSON) and Binary data for real time communication

Could have replaced syslogd and the external API in my previous example

Code time? :-)

Questions?

helgi@orchestra.ioTwitter: @h

Joind.in: http://joind.in/3212

Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

Technology

Transcript of Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

Dpc 4 H Summer2008v2

DPC-12000/15000 SERIES INSTALLATION AND OPERATION …powerboxinc.com/support/dpc15000_instructions.pdfThe DPC-12000 is identical in form and function to the DPC-15000, except the DPC-12000

DPC REQUEST INFO - Liberty HealthShare€¦ · REIMBURSEMENT REQUEST FAX: (216) 456-8115 EMAIL: bills@libertyhealthshare.org DIRECT PRIMARY CARE (DPC) DPC REQUEST INFO *Request considered

DPC Onboard

DPC Jan01 Social_Media_2012

Normam 03 - DPC

DPC on Mechanics

DPC-2016 UCSM.ppt

Culture conditions and its impact on cell lineages and ......dpc 0.5 dpc 0.5 dpc 3.5 dpc 4.5 dpc 2.5 dpc 1.5 dpc 3 Development Important changes occur during these steps (modified

DPC-X937 X637 X537 - KENWOODmanual.kenwood.com/files/B60-5378-00.pdf · DPC-X937/DPC-X637/DPC-X537 B60-5378-00 00 MA (E,Y) AP 0304 PORTABLE CD PLAYER INSTRUCTION MANUAL This instruction

Distribute the workload, PHP Barcelona 2011

Dpc #6 Google Tools

DPC-2 Pattern Controller

DPC Textbook Guide

DPC Engineering

DPC 2203C EMTA

WcsPlus - Powered by DPC

DPC treatment

DPC Procedure com

DPC SA Onboard