1401 Avenue du Mondial 98 Immeuble Oxygène

34000 Montpellier04 99 13 68 60

1 KALIOP.COM

How to deploy & optimize eZ Publish high-traffic / bigdata context

Le 20/09/2013

High traffic ? Huge volume ?

2

High-traffic ?

Exemples > 5 M pages / vues / mois

Peak loads ?

Big Data ?

Transactions ?(import – export - REST)

User profiling ?(auth – settings – peruser UI)

500K + user/day500K + user/day

+200%

1M+ datas

100k transact./day

50%+ authenticated(portal features)

+

++

+

++

+++

Critical issues : Critical mass : Complexity to solve :

Exemples > 5 M pages / vues / mois

Replicated ?(PRA – Worldwide cloud)

2+ ++++

Le 20/09/2013

Keys of success

3

Exemples > 5 M pages / vues / mois

Project management approach

Technical expertise Tools

Do not focus solely on technical expertise !

Web Architect / SCRUM Master

Web Architect / SCRUM Master

Le 20/09/2013

Keys of success / PM approach

4

Exemples > 5 M pages / vues / mois

Project management approach

Role of the Web Architect : Never stop to (re)design. Arbitrate & find workarounds for all the contradictory issues : UX, front-end, performances, dev, hosting, delay, cost, stability, SEO, marketing operations...

Role of the project manager (or SCRUM Master) : Protect the team. Understand, explain & make the POC possible , code refactoring, continuous integration & devOps

Role of the project owner : Build the team & be clear on the KPIs

Exemple : possible to challenge the UX ? (VS performances)Exemple : ready to work without a Gantt ? (VS kanban)

Le 20/09/2013

Keys of success / PM approach

5

Tools

Monitoring : complexity to monitor all the stack(in most cases, only on low level services, not on PHP / slow queries)

Metrics : complexity to have metrics on all the the stack(in most cases, only on low level services, or end-users trafic)

Deployment tools : the easier it is, the more consuming it is(in most cases, copy the files & purge the cache...)

Testing : complexity to automate unit testing / user testing(in most cases... nothing)

Capistrano

Le 20/09/2013

Keys of success / Technical expertise

6

Technical expertise(not only eZ)

FPM

FastCGI

APC

And more...

Le 20/09/2013

Keys of success / Technical expertise

7

eZ FS2 StaleCache

INITemplatesMédiasViewcacheTemplate-block

Médias (copy)

HTTP CachesESI

++++

Easy to hostSave ressourcesEasy to setup/switchStaleCacheSimplify the deployment process

----

High NFS activityRequire top-level ($$$) NFSRequire an high eZ quality devRequire frequent purges

APC APC APC

Stale Cache( no flock )

Le 20/09/2013

Keys of success / Technical expertise

8

eZ DFS Cluster

INITemplatesCache (copy)

INITemplatesCache (copy)

INITemplatesCache (copy)

Médias (copy)

HTTP CachesESI

APC APC APC

EZ DFS

MédiasViewcacheTemplate-block...

META DATA

++++

Compatible with most of platformsLow NFS activityEasy to setup/switchStaleCache

----

Waste of ressourcesClusterDB bottleneckComplex deployment processRequire an high eZ quality devRequire frequent purges

Le 20/09/2013

Keys of success / Technical expertise

9

DevOps mode

Do not expect the hosting architecture (sizing) to make up for the application issues...

Do not expect the development (hack, trick) to make up for the hosting architecture issues...

Le 20/09/2013

Keys of success / Technical expertise

10

DevOps mode(expertise in the middle)

APCapc.filters = "-cache.*\.php"(exlude compiled templates)

MySqltmpdir → tmpfs (avoid « on disk » storage)Add missing indexes (remote_id)

SolrSplit read/write operations

VarnishSynchronize with eZ (x-node pruge)

APC

Le 20/09/2013

Keys of success / Technical expertise

11

eZ PublishOptimize your code(what you can deduce)

Reduce theCache-block activity

& volume

Optimize the viewcache

activity

RemoveUseless SQL

Use small range keysSet an expiry valueUse ezpagedata() insteadReplace with ESINo more than 1K files !

Avoid too many view parametersOptimize smartviewcacheSynchronize with Varnish

Replace with SolrUse PHP cacheAvoid datamap loadOptimize

Solrsplit read / write activities

Delay « commit » (not only index)

Schedule your background tasks

Le 20/09/2013

Keys of success / Technical expertise

12

eZ PublishOptimize your code(read tutos)

Le 20/09/2013

Keys of success / Technical expertise

13

eZ PublishOptimize your code(Learn how it works)

Basic optimization :DelayedIndexing = true (→ will not delay hard commits for all actions ! )OptimizeOnCommit = disabledCRON // ezfoptimizeindex (3 minutes)CRON // ezfindexcontent (1 day)

Medium optimization :OptimizeOnCommit = disabledDisableDeleteCommits = true (+ CRON to commit)DisableDirectCommits=true (+ CRON to commit)CRON // bin/php/ezsolrcommit.php (3 minutes)CRON // ezfoptimizeindex (1 day)

Other great optimizations : - Replicate Solr, and split the Read (search..) / Write (commit, optimize), to avoid concurrencies - Avoid hard commits (soft commits to avoid fsync ops, commitWithin to delay pending docs within Solr)

addDoc Hard Commit (rebuild indexes + fsync)

Soft Commit (rebuild indexes)

CommitWithin (commit all pending docs within X secs)

Le 20/09/2013

Keys of success / Technical expertise

14

Optimize your code(Get metrics)

(what you can not guess)

APC

EZ DFS

META DATA

GET METRICS

Le 20/09/2013

Keys of success / Technical expertise

15

Optimize your code(Get metrics)

(what you can not guess)

Why session_start lock ???

eZ Bug on the checkValidity=false setting...

will always create a session_start on the index.php

Le 20/09/2013

Keys of success / Technical expertise

16

Optimize your code(Get metrics)

(what you can not guess)

Why a unique table insert needs 9,6 sec ???

Server side tuning requested (memory, table optimize, host the clusterDB on another server...)

Le 20/09/2013

Keys of success / Technical expertise

17

Optimize your code(Get metrics)

(what you can not guess)

Why my code provides 1988 cURL calls ???

Mmhh...How to say...

NEXT

Le 20/09/2013

Enough to resolve the following context ?

18

High-traffic ?

Exemples > 5 M pages / vues / mois

Peak loads ?

Big Data ?

Transactions ?(import – export - REST)

User profiling ?(auth – settings – peruser UI)

500K + user/day500K + user/day

+200%

1M+ datas

100k transact./day

50%+ authentifié(portal features)

+

++

+

++

+++

Critical issues : Critical mass : Complexity to solve :

Exemples > 5 M pages / vues / mois

Replicated ?(PRA – Worldwide cloud)

2+ ++++

Le 20/09/2013

Enough to resolve the following context ?

19

Externalize the user database (if too big)

Build a full ESI front-end (pure HTTP cache logical)

Replace the DB cluster with Redis or memcached

Externalize medias on CDN

Use a custom PHP Cache (exemple)

...

NO ! Be creative !

Le 20/09/2013

Enough to resolve the following context ?

20

Be creative !

Object.datamap → PHP arrayPHP cache file (eZPHPCreator)

On publish

Store all the attributesPre-generate related values

On read

Read object datamap

Fetch, filter, facet, sort..

HTML

Le 20/09/2013

Enough to resolve the following context ?

21

Ready for eZP5...

Native HTTP CacheNative Storage abstractionNative pure MVCNative ESI support…Full Symfony stack