Web Clustering, Integration with Terracotta, BigMemory, Quartz and Grails

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Clustering With Terracotta (And BigMemory!)

Web Sessions, Ehcache, and QuartzClustering

Ryan Vanderwerf

Chief Architect /

Directory of Products

ReachForce

@RyanVanderwerf

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Clustering With TerracottaMy BackgroundChief Architect and Director of Products @ ReachForce

Former Architect of developerprogramcom, specializing in the

mobile industry

Built SaaS Startups in since 1997

Co-Chair of Austin Groovy and Grails User Group

Help maintain various plugins – Quartz, GVPS (Video), Struts1

Co-Author of 'Effective Gradle Implementation' Video Series

on Packt Publishing (Aug 2014)


Clustering With TerracottaWhat Will We Cover?

Clustering basicsReasons for clusteringTypes of clusteringHTTP session clustering

Ehcache & 2nd level cacheColorcache demo

Quartz overviewGrails-Quartz plugin

Quartz2 pluginGrails 2 Quartz job demoGrails 2 Ehcache/Quartz Demo

BigMemory Max – Overview & DemoTownsend camera demo


Clustering With Terracotta

This beer is

tasty, but you

really don't

want this kind

of situation!


Clustering With TerracottaClustering Basics

Answer the question: Do I really need to

maintain state across the cluster? Can that

state be moved out of the back end?

Don't Do It!



Clustering Basics

Usually a way to maintain state across

nodesSoftware and hardware working together

Different levels of clustering for different purposes

When you are stuck with a large stateful system....



Many acting as one

We mean many servers, applications, and data grouped togetherCovers caching of data for fast access across all nodes

Load balancers are the front line of most clustered setupsMaintain state across many nodes in performant manner (avoid this at all costs, but sometimes we're stuck with it)


Clustering With TerracottaTerracotta

Owners of Quartz and EhcacheOpen source with most features freeCommercial if you need multiple active mirror groupsAcquired by Software AG a couple years ago Continuing OS commitment to Quartz & Ehcache


Clustering With TerracottaTerracotta

Final version of OS Terracotta 3.7.7

BigMemory is their main product going forward

Offered 8GB Heap Version of BigMemory for

short period – may bring this back

Continuing OS commitment to Quartz &

Source is open, community needs to

make fork to make it compatible with

Quartz 2.2.x+



What Will We Cover?




Quartz2 pluginGrails 2 Quartz job demo

Grails 2 Ehcache/Quartz Demo




Terracotta

TCP based server – simplified

networking

Central Primary Controller

with hot spare

TIM Modules OSGi based for

integration into many kinds of

apps


Clustering With TerracottaTerracotta Primary Uses

Distributed HTTP Sessions

EhCache and 2nd Level Hibernate Cache

Distributed Quartz scheduling

+ BigMemory Primary Uses

Distributed Large Heap Management For HA

Searching Across Heap (Arc)



Terracotta Commercial Only Features

Big Memory (Max and Go)

Terracotta 'Enteprise' bundle (Now

deprecated for BigMemory)

Extra security features with console, roles, etc

Quartz Location API (Quartz Where)



What Will We Cover?








Clustering With TerracottaProblem: Evenly distribute load across many

servers

Many techniques to handle this

HTTP Session Ring Type on

most web containers


Clustering With TerracottaTypical Web Session Cluster

Session data sent in a ring, in case one node

fails session is not lost


Clustering With TerracottaOther Techniques

Use load balancer with sticky source ip

Doesn't give balanced load – due to large

IP blocks sticking to one server (and large

ISPs that use proxys)


Clustering With TerracottaOther Techniques

Save sessions in database

Too slow – serialization in and out of DB has

dreadful performance unless session ultra light



What Will We Cover?









Terracotta Web Sessions

Easy setup

Session visibility

ScaleIn Memory SpeedEfficient

Scale Up!


Clustering With TerracottaTypical Terracotta Cluster

Session data sent to central serverIncremental changes only sent – much faster



Terracotta Web Sessions Setup

Tomcat 6 / 7 – add valve to context.xml

Websphere, Jboss, others

Add filter to web.xml




Tomcat 6 / 7 – add valve to context.xml

<Valve className="org.terracotta.session.TerracottaTomcat60xSessionValve" tcConfigUrl="localhost:9510">




Copy runtime jar to container lib directory

common/terracotta-toolkit-1.1-runtime-2.1.0.jar

See Matrix for proper toolkit version for your stack – I.e.


Clustering With TerracottaTerracotta Web Sessions Setup

JBoss – add valve to context.xml

<Valve className="org.terracotta.session.TerracottaJboss51xSessionValve" tcConfigUrl="localhost:9510">

<filter> <filter-name>terracotta</filter-name>  <filter-class>org.terracotta.session.<container-specific-class></filter-class> <init-param> <param-name>tcConfigUrl</param-name> <param-value>localhost:9510</param-value> </init-param></filter>


Jetty, Weblogic, Websphere

Jetty, Weblogic, Websphere<filter-mapping>  <filter-name>terracotta</filter-name> <url-pattern>/*</url-pattern>  <dispatcher>ERROR</dispatcher> <dispatcher>INCLUDE</dispatcher> <dispatcher>FORWARD</dispatcher> <dispatcher>REQUEST</dispatcher></filter-mapping>



What Will We Cover?








Clustering With TerracottaTerracotta Townsend App Demo



Terracotta Townsend App Demo


Clustering With TerracottaTerracotta Townsend App Demo



Terracotta Townsend App Demo



What Will We Cover?









Application / Database Caching

Why do I need this?

Take load off database

Increase performanceEase scalability




Solutions

EhcacheJboss CacheJgroups + roll your own caches

Terracotta / BigMemoryHazelcast




EhcacheFine on it's own but....Hard to debugUses Jgroups under the cover but can cause

odd network issues with multicast – no cloud

No real console to see what is going on




JBoss Cache

Fine on it's own but....JbossCache Grails plugin only tested with Grails 1.1Uses Jgroups under the cover but can cause odd

network issues with multicastNo real console to see what is

going on




Jgroups + Roll Your Own

JGroups hard to manageResponsibility to manage cache – why not let a

framework help you?

ComplexNo real console to see what is going on




Hazelcast

Created by a former founder of Terracotta

Open source and free with paid support available

Grails plugin available but likely out of date (2012)

Supports most things TC and others except Quartz

but has similar type of job execution technology for

data grid



Application / Database CachingTerracotta – Ehcache

Integrates into console

Centralized server makes setup and management

less complex

Works in Grails 2.x

Fast – only transmits deltas




Terracotta – Ehcache

Open source free edition sufficient for most installations

Great developer console tells you shows full insight of

cache

Express install – easy to setup if you follow these

slides


Tomcat

Ver

Quartz Plugin

TC Ver

Quartz TC

Cluster?

Quartz DB

Cluster?

Ehcache Ver TK Ver

7 2.1 3.7.7 Y Y 2.6.8 1.6-5.7.0

6 2.0/2.1 3.7.5 Y Y 2.6.6 1.6-5.5.0

7 2.0/2.1 3.7.5 Y Y 2.6.6 1.5-4.2.0

7 2.0 3.6.x Y Y 2.5.2 1.5-4.2.06 2.0 3.6.x Y Y 2.5.2 1.5-4.2.0

Clustering With TerracottaValid Setup Versions – Updated!Terracotta – Ehcache



Grails 2.x Setup Checklist


BuildConfig.groovyMake sure versions of Ehcache-core match TC

InstallMake sure TC Toolkit jar match TC installMake sure ehcache-terracotta jar math TC

install

dependencies { runtime 'net.sf.ehcache:ehcache-core:2.6.8' runtime 'net.sf.ehcache:ehcache-terracotta:2.6.8' runtime 'org.terracotta:terracotta-toolkit-1.6-runtime:5.7.0' runtime 'org.terracotta.quartz:quartz-terracotta:2.3.7' }

Clustering With TerracottaTerracotta 3.7.7 Grails 2.x Ehcache Setup –

BuildConfig.groovy



What Will We Cover?









ColorCache DemoTerracotta – Ehcache



ColorCache Demo




What Will We Cover?








Clustering Quartz

What Is Quartz?

Open source Java API Used to schedule, persist,and distribute jobsGreat Java and Grails support

Great community support and large usage (thousands)Most common solution for scheduling execution in Java

applications

Managed and funded by OS Support from

Terracotta AG


Clustering Quartz

Grails-Quartz Plugin

Integrates Grails with Quartz 2.2 (yay!)

Officially supported by Pivotal

For Quartz 2.2+ use latest Plugin, use RC-7 for

Quartz 2.1 support with Terracotta

For BigMemory, Latest Quartz 2.2 is supported


Clustering QuartzQuartz2 Plugin

Integrates Grails with Quartz 2.xWorks best with Clustered JDBCStore due to bug in Terracotta (https://jira.terracotta.org/jira/browse/QTZ-310) fixed in next major Quartz release!Supports Groovy based JobDetailNot Officially supported by SpringSourceSupports (nosql) engines like Mongo or Redis


Clustering QuartzWhy Cluster Quartz?

Distribute Load

Scale easily

Handle many batch jobs at once

Persist scheduled work queue in case of

crash

Fail-over

Visibility of workload



What Will We Cover?








// Each method (except the one for custom trigger) takes optional 'params' argument. // You can use it to pass some data to your job and then access it from the job:class MyJob { def execute(context) { println context.mergedJobDataMap.get('foo') }}// now in your controller (or service, or something else):MyJob.triggerNow([foo:"It Works!"])

Clustering QuartzInstalling Quartz and Creating a Job

Dynamic Jobs


class MyJob { static triggers = { simple name: 'mySimpleTrigger', startDelay: 60000, repeatInterval: 1000 } def group = "MyGroup" def execute(){ print "Job run!" }}

Clustering Quartz

Installing Quartz and Creating a Job

Add plugin to

BuildConfig.groovy:

grails create-job <jobName>


Clustering Quartz

Installing Quartz and Creating a JobCron trigger example

class MyJob { static triggers = { cron name: 'myTrigger', cronExpression: "0 0 6 * * ?" } def group = "MyGroup" def execute(){ print "Job run!" }}


// creates cron trigger;MyJob.schedule(String cronExpression, Map params?)// creates simple trigger: repeats job repeatCount+1 times with delay of repeatInterval milliseconds;MyJob.schedule(Long repeatInterval, Integer repeatCount?, Map params?) )

Clustering QuartzInstalling Quartz and Creating a Job

Dynamic Jobs


// schedules one job execution to the specific date;

MyJob.schedule(Date scheduleDate, Map params?)

//schedules job's execution with a custom trigger;

MyJob.schedule(Trigger trigger)

// force immediate execution of the job.

MyJob.triggerNow(Map params?)

Clustering QuartzInstalling Quartz and Creating a JobDynamic Jobs


// Each method (except the one for custom trigger) takes optional 'params' argument. // You can use it to pass some data to your job and then access it from the job:class MyJob { def execute(context) { println context.mergedJobDataMap.get('foo') }}// now in your controller (or service, or something else):

MyJob.triggerNow([foo:"It Works!"])

Clustering QuartzInstalling Quartz and Creating a JobDynamic Jobs


Clustering Quartz

Prepping Your Environment for Distributed Quartz

How do you want to run the jobs?

Replicate your grails app X times and

distribute across that?

Have separate replicated application the

picks up the jobs

WAR file under application server


Clustering Quartz

Prepping Your Environment for Distributed Quartz

Standalone application

Just Java classes running scheduler

command line?

Standalone plugin?

Custom standalone i.e.

https://gist.github.com/1804182 ?


Clustering Quartz

Terracotta vs. JDBCJobStore Clustering

Terracotta has remote GUI console

Terracotta shows quick status of jobs

Terracotta doesn't require a database to persist jobs

Terracotta Open Source Free

Good support via Terracotta.org forumsBoth solutions should have a time server synchronizing clocks on all machines


Clustering Quartz

JDBCJobStore Clustering

Most common and least work to configure

Set up JobStore and Delegate

Configure database table prefix

Setup database schema

Works with Grails 'quartz' (Quartz 2.x) or

'quartz2' (Quartz 2) plugin


Clustering QuartzTerracotta Clustering

No Database setup required

Now works with Quartz 2.x and 'quartz' plugin

Doesn't work with Quartz2 plugin because it implements a different class for the JobDetails interface that is not JobDetailsImpl. Terracotta will throw errors because it assume JosDetailsImpl class is used. JIRA logged at https://jira.terracotta.org/jira/browse/QTZ-310 - update fixed for next major Quartz release (api change)


Clustering Quartz

Terracotta Clustering

Slick GUI Interface

Excellent failover and hot spare ability, as well as integrating with Hibernate 2nd level cache, and HTTP Session caching

Open source edition free, works great for most small to medium installs

Commercial version allows more than one active cluster server and role based access to admin console


Clustering Quartz

Setting Up Open Source Terracotta Clustering

Download Terracotta 3.7.7 OS from Terracotta.org

Install 'quartz' grails plugin (RC-7)

Create quartz.properties in grails-app/conf or

src/java


Sample quartz.properties file:org.quartz.scheduler.instanceName = MyClusteredSchedulerorg.quartz.scheduler.instanceId = AUTO# Configure ThreadPoolorg.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPoolorg.quartz.threadPool.threadCount = 25org.quartz.threadPool.threadPriority = 5org.quartz.jobStore.class=org.terracotta.quartz.TerracottaJobStore# the path below should point to your terracotta config file. This can can be a URL as well like http://org.quartz.jobStore.tcConfigUrl = /opt/terracotta-3.7.5/tc-config.xml

Clustering QuartzSetting Up Open Source Terracotta Clustering


Clustering Quartz

Starting TerracottaDownload Terracotta 3.7.7 from Terracotta.org

http://www.terracotta.org/downloads/open-source/

Copy $TERRACOTTA_HOME/config-samples/tc-config-express-

reference.xml to $TERRACOTTA_HOME/tc-config.xml

Run ./start-tc-server.sh -f /path/to/tc-config.xml

Run ./dev-console.sh


Clustering Quartz

JDBCJobStore ClusteringSimple to configure

Schema to create tables included in distribution

Uses central database to persist jobs, and

unique nodeIDs (AUTO will work)

Time server must sync instances regularly, and

be within 1 second of each other

Works with Quartz 2.x or Quartz 2.x ('quartz' or

'quartz2' plugin


Clustering Quartz

JDBCJobStore Clustering● Sample quartz config highlights (see code

samples for complete file)● org.quartz.jobStore.isClustered = true

● org.quartz.jobStore.clusterCheckinInterval = 20000

● org.quartz.jobStore.misfireThreshold = 60000

● org.quartz.jobStore.class =

org.quartz.impl.jdbcjobstore.JobStoreTX

● org.quartz.jobStore.driverDelegateClass =

org.quartz.impl.jdbcjobstore.oracle.OracleDelegate

https://jira.terracotta.org/jira/browse/QTZ-310


Sample quartz config highlights (see code samples for complete file)

org.quartz.jobStore.isClustered = true

org.quartz.jobStore.clusterCheckinInterval = 20000

org.quartz.jobStore.misfireThreshold = 60000

org.quartz.jobStore.class =

org.quartz.impl.jdbcjobstore.JobStoreTX

org.quartz.jobStore.driverDelegateClass =

org.quartz.impl.jdbcjobstore.oracle.OracleDelegate

Clustering Quartz

JDBCJobStore Clustering


Clustering QuartzGo Advanced!

Use Job and Trigger Listeners (there is no groovy version

of this, you will have to make regular class files) to split

large jobs apart into smaller ones. (Cluster Bomb Pattern)

Example applications using distributed jobs:

Email Campaign Tool

Data processing

Email Verification Tool



What Will We Cover?








Clustering Quartz

Setup and Run Quartz 2 Plugin

Use Job and Trigger Listeners (there is no groovy

version of this, you will have to make regular class

files) to split large jobs apart into smaller ones.

See Docs at https://github.com/9ci/grails-quartz2



What Will We Cover?



Quartz overviewQuartz1 plugin

Quartz2 plugin




Clustering Quartz

BigMemory Max

Now has main feature parity with older Terracotta

Distributed Large Heap Management for HA

GO: Get around regular single JVM heap limits

Max: In memory data store with search (SQL)


Clustering Quartz

Typical Memory Layout with or without BigMemory


Clustering Quartz

Typical TC/BigMemory Scale Out

https://gist.github.com/1804182


Clustering Quartz

BigMemory Hybrid (Go + Max)


Tomcat Version

Quartz Plugin

TC Ver Quartz TC Cluster?

Quartz DB Cluster?

Ehcache Ver

TK Ver

7 2.2 4.1.3 Y Y 2.8.3-ee

4.1.3-ee

6 2.2 4.1.3 Y Y 2.8.3-ee

4.1.3-ee


Valid Setup Versions – Updated!

BigMemory Max



What Will We Cover?




Quartz2 plugin



https://jira.terracotta.org/jira/browse/QTZ-310


Clustering Quartz

Setup and Run Quartz 2 Plugin

Example applications using distributed jobs:

Email Campaign Tool

Data processing

Email Verification Tool



What Will We Cover?




Quartz2 plugin


Townsend camera demo



What Will We Cover?




Quartz2 plugin




Clustering Terracotta

More Information

Quartz2 Plugin: https://github.com/9ci/grails-quartz2Open Source DL: http://terracotta.org/downloads/open-source/cataloghttps://jira.terracotta.org/jira/browse/QTZ-310Official Quartz Plugin: http://grails.org/plugin/quartzHomepage of Quartz plugin http://grails-plugins.github.com/grails-quartz/Quartz Home: http://quartz-scheduler.org/Toolkit API:http://terracotta.org/documentation/4.1/bigmemorymax/terracotta-toolkit/toolkit-usageExample Grails App for BM 4.1.3https://github.com/rvanderwerf/terracotta-grails-demo



Contact Me

Via twitter: @RyanVanderwerfGoogle+/email: [email protected]: http://rvanderwerf.blogspot.comReachForce: www.reachforce.com



Copyright Notices

Page 1 Image: Creative Commons by Striving to a goalCluster Bomb Picture: Sascha Grant (Creative Commons)Screenshots for ColorCache and Townsend property of Terracotta, Inc.BigMemory Pictures: Property of Terracotta, Inc.Hops Pictures: Flickr and Google Creative CommonsThe rest (c) Ryan Vanderwerf 2013-2014

Web Clustering, Integration with Terracotta, BigMemory, Quartz and Grails

Software

Transcript of Web Clustering, Integration with Terracotta, BigMemory, Quartz and Grails