Denser, cooler, faster, stronger: PHP on ARM microservers

Jez Halford

Jez Halford

Freelance web development consultant

jezhalford.com

@jezhalford

● World’s largest designer of semiconductors

● Over 60 billion ARM chips shipped to date

● Began moving into server CPUs a few years ago

Shoot for the moonHewlett Packard EnterpriseMoonshot

● Very high server density

● More efficient cooling

● Built in fault-tolerance

Moonshot

● 45 cartridges, supporting up to4 servers per cartridge

● 4 redundant power supplies

● 2 ethernet switches

● 5 cooling fans

● iLo Management interface

Shoot for the moon

CPU Eight 64 bit cores @ 2.4GHz

Memory 64GB

Storage 120GB / 240GB / 480GB SSD

Network 2x 10 Gigabit NICs

ProLiant m400

“Drinking our own Champagne”

Goals

● Robust, highly available website

● Scalable to handle very high load

● Use standard, generally available technologies where possible

Milestones

● Modify the PHP app from Windows-specific to platform agnostic

● Design and prove the server architecture

● Go live

Windows to anything

● Remove hard-coded paths

● Improve autoloading

Windows to anything

● Config injection over environment detection

if (gethostname() == ‘qa-box’) {

// set environment for QA

}

if (getenv(‘APP_ENV’) == ‘qa’) {

// set environment for QA

}

Server Architecture

PHP on ARM

sudo apt-get install php7

High Availability

● Two NICs per cartridge

● Bonded to work as one

● Allows one chassis switch OR one NIC to fail and operation to continue

http://www.unixmen.com/linux-basics-create-network-bonding-on-ubuntu-14-10/



Runningon multiple servers● Load balancing● Filesystem● Application cache● Session storage● Log aggregation● Data persistence

Load Balancing

● Paid-for version of nginx● Support & extra features

http { upstream myapp1 { server srv1.example.com; server srv2.example.com; server srv3.example.com; }

server { listen 80;

location / { proxy_pass http://myapp1; } }}

Load Balancing

ARM Nginx Node

ARM Web Node

ARM Web Node

ARM Web Node

Web

Bonded IP address

Load Balancing

F5

ARM Nginx

Node A

ARM Nginx

Node B

ARM Web Node A1

ARM Web Node A2

ARM Web Node A3

Web

ARM Web Node B1

ARM Web Node B2

ARM Web Node B3

Moonshot Chassis A

Moonshot Chassis B

Caching statics at the load balancer● Nginx respects Cache-Control headers● In our setup these come from Apachehttp {

upstream myapp1 { server srv1.example.com; server srv2.example.com; server srv3.example.com; }

proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=STATIC:10m inactive=24h max_size=1g;

server {listen 80;

location / { proxy_pass http://myapp1; proxy_set_header Host $host; proxy_cache STATIC; proxy_cache_valid 200 1d; proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; }

}

Runningon multiple servers● Load balancing ✔● Filesystem● Application cache● Session storage● Log aggregation● Data persistence

Filesystem

● Both are free, open source distributedfilesystem solutions

● Multiple nodes work together to form a single mountable filesystem

● Generally slow at writing, but great forstoring PHP source and static assets

Filesystem

● Copy files to each node at regularintervals

● Nominated “master” node is the source

● Has a replication lag

Filesystem

We used existing, external and mountable storage ARM already had

Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache● Session storage● Log aggregation● Data persistence

Application Cache

● 64GB on each node!

● Potentially 2.8 Terabytes per 45 node moonshot chassis

Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache ✔● Session storage● Log aggregation● Data persistence

Session Storage

Options!

Session Storage

● Zero configuration

● Slow access

● Persistent

● Needs either a shared filesystem or sticky sessions

Session Storage

● Simple

● Very easy configuration

● Non-persistent

● It’s for caching, not sessions

Session Storage

● Fast

● Persistent

● Generally fault-tolerant with Redis Cluster

● Cannot always guarantee consistency

Session Storage

Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache ✔● Session storage ✔● Log aggregation● Data persistence

Log Aggregation

● Lots of apps on lots of servers

● Lots of logs generated

● Debugging is hardif we have to look inlots of places

Log Aggregation

rsyslog● Aggregate logs from MySQL, Apache, Nginx across

several servers...

● ...into one file per app on a “management” server

Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache ✔● Session storage ✔● Log aggregation ✔● Data persistence

Data Persistence

Data Persistence

● Active / Active, not Master / Slave

● True horizontal scalability

● High availability

A PXC Cluster

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

A PXC Cluster

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

ARM Web Node

Split Brain

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

ARM DB Node 6

Consistency over availability

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

Networking failure

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

More than half the nodes fail

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

Single node failures

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

Rejoining

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

Mixed failures

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

Multiple node failures

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

PXC Startup

ARM DB Node 1

sudo service mysql bootstrap-pxc

Cluster size: 1

PXC Startup

ARM DB Node 1

ARM DB Node 2

sudo service mysql start

Cluster size: 2

PXC Startup

ARM DB Node 1

ARM DB Node 2

ARM DB Node 3


Cluster size: 3

PXC Startup

ARM DB Node 1

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4


Cluster size: 4

PXC Startup

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4


Cluster size: 5

PXC

● Very good horizontal scaling

● An absolute nightmare in cascading failures

● Those failures are very rare

Load balancing the database

● How does the application know which DB node to talk to?

● HAProxy! A TCP Load Balancer


ARM Web Node A1

PHP

HAProxy

127.0.0.1:3306Individualdatabasenodes...

An individual web node:

ARM Web Node A1


ARM Web Node A2

ARM Web Node A3

HAProxy

HAProxy

HAProxy

ARM DB Node 3

ARM DB Node 5

ARM DB Node 1

ARM DB Node 2

ARM DB Node 4


● Each HAProxy instance carries out regular health checks on each node

● When a node fails a check it is removed from the pool until it passes again

● The health checks need to verify PXC cluster state, as well as responsiveness to incoming connections

https://github.com/olafz/percona-clustercheck



Runningon multiple servers● Load balancing ✔● Filesystem ✔● Application cache ✔● Session storage ✔● Log aggregation ✔● Data persistence ✔

#

F5

Bonded IP address

ARM Nginx

Node A

ARM Nginx

Node B

ARM Web Node A1

ARM Web Node A2

ARM Web Node A3

Web

ARM Web Node B1

ARM Web Node B2

ARM Web Node B3

ARM DB Node 1

ARM DB Node 5

ARM DB Node 2

ARM DB Node 3

ARM DB Node 4

PXC ClusterMoonshot Chassis A

Moonshot Chassis BMoonshot Chassis C

F5 ARM Web

Node A1

Web

#

ARM Web

Node A2ARM Web

Node A3

ARM Web

Node B1

ARM Web

Node B2

ARM Web

Node B3

ARM DB

Node 1

ARM DB

Node 5

ARM DB

Node 2

ARM DB

Node 3

ARM DB

Node 4

PXC Cluster

Bonded IP address

ARM Nginx

Node A

ARM Nginx

Node B

ARM Web

Node A1

#

ARM Web

Node A2ARM Web

Node A3

ARM Web

Node B1

ARM Web

Node B2

ARM Web

Node B3

ARM DB

Node 1

ARM DB

Node 5

ARM DB

Node 2

ARM DB

Node 3

ARM DB

Node 4

PXC Cluster

Bonded IP address

ARM Nginx

Node A

ARM Nginx

Node B

Moonshot Chassis A

Moonshot Chassis B

Moonshot Chassis C

Moonshot Chassis D

● Lower total cost of ownership

● Highly fault tolerant

● Uses standard technologies

What next?

● Several providers of hosted ARM servers

AaaS{ ARM-as-a-Service }

scaleway.com

armasaservice.com

mininodes.com

unixy.net

nx-box.net

http://www.armasaservice.com/default.aspx

http://www.armasaservice.com/default.aspx

Hewlett Packard Enterprise Moonshot

https://www.hpe.com/uk/en/servers/moonshot.html

Thanks for listening

joind.in/talk/01006

@jezhalford jezhalford.com

Denser, cooler, faster, stronger: PHP on ARM microservers

Technology

Transcript of Denser, cooler, faster, stronger: PHP on ARM microservers