Devoxx 2013 - Hazelcast

@PeterVeentjer#DV13-{session hashtag}

Distributed Systems Using Hazelcast

Peter Veentjer

• Working for Hazelcast

• Senior developer

• Solution architect

• Author of ‘Pragmatic Guide to Hazelcast 3'

• 13 years Java experience

• Big love

• Concurrency control

• Distributed computing

Whoami

What is Hazelcast?

• Leading Open Source Java In Memory Data/Compute Grid

• Main goal is to simplify development of:

• Scalable systems

• Highly available systems

Why Hazelcast?

• 2.5 MByte JAR

• no other dependencies!

• no need to install software!

• Its a library, not an application framework

• Scale at the application level

• Apache 2 License

• Free to use

• no limitations

Distributed Data-structures

• IAtomicLong

• IdGenerator

• Lock/Condition

• CountDownLatch

• Semaphore

• Queue

• Map

• MultiMap

• Set

• List

• Topic

• Executor

• TransactionalMap/Q/S…

• Write your own!

So what can I do with it?

Scale up

Scale Out

High Availability

Raspberry Pi Cluster

When?• Messaging Solution

• Event processing

• Clustered Scheduling

• Job Processing

• Cluster management

• HTTP session clustering

• Caching

Which companies

• Financials

• JP Morgan, Morgan Stanley, HSBC, Deutsche Bank

• Telco’s

• AT&T, Ericsson

• Gaming

• Ubisoft/Blue Byte

• E-Commerce

• Apple, eBay

Which Open Source Projects

• WSO2 Carbon

• Mule ESB

• Vert.x

• Apache Camel

• Apache Shiro

• OrientDB

• Alfresco

• Karaf

NO SQL FAMILY

Hazelcast

Coherence

Infinispan

Gigaspaces

Datagrids Key Value Stores Document DB’s Column DB’s

MarkLogic

CassandraRedis

CouchDB

HBaseRiak

Voldimort

Graph DB’s

How Often

• In October 2013

• 3M startups

• 48K unique

Where is the code!

HazelcastInstance hz = Hazelcast.newHazelcastInstance();

Creating Hazelcast Cluster

XML Configuration

Config config = new Config(); … make config modifications HazelcastInstance hz = Hazelcast.newHazelcastInstance(config);

Programmatic Configuration

Map<String,String> products = new HashMap(); map.put("1","IPhone");

Map<String,String> products = new ConcurrentHashMap(); map.put("1","IPhone");

HazelcastInstance hz = Hazelcast.newHazelcastInstance(); Map<String,String> products = hz.getMap("products"); cities.put("1","IPhone");

!<map name="products"> <time-‐to-‐live-‐seconds>4</time-‐to-‐live-‐seconds> <indexes> <index>name</index> </indexes> <..> </map>

Map Configuration

Map Scalability and High Availability

271 Partitions

partitionid = hash(key) % partitioncount

Member 1

Member 2

Member 3

Member 1

Member 2

Member 3

Member 1

Member 2

Member 3

Member 1

Member 2

Map Persistence

• Data Connection

• MapLoader

• MapStore

• Write through

• Write behind

Multimap

• Collection as value

• Serialization Overhead

• Lost update

BlockingQueue queue = new LinkedBlockingQueue(); queue.offer("1"); Object item = queue.take();

!HazelcastInstance hz = Hazelcast.newHazelcastInstance(); BlockingQueue queue = hz.getQueue("queue"); queue.offer("1"); Object item = queue.take();

ITopic topic = hz.getTopic("topic"); //publishing topic.publish(msg); !//subscribing topic.addMessageListener(new TopicSubscriber()); !public class TopicSubscriber implements MessageListener<String> { public void onMessage(Message<String> m) { System.out.println(m.getMessageObject()); } }

Client

• Simplified

• Encryption

• Load Balance Policy

• C++ and C# version

HazelcastInstance client = HazelcastClient.newHazelcastClient();

Client

Network Communication

• Cluster Discovery

• Multicast

• TCP/IP Cluster

• AWS Cluster

• Normal Network Communication

• TCP/IP

Advanced Hazelcast

IMap map1 = hz.getMap(“map1”); IMap map2 = hz.getMap(“map2”);<map name=“map1”> <time-‐to-‐live-‐seconds>10</time-‐to-‐live-‐seconds> </map> !<map name=“map2”> <time-‐to-‐live-‐seconds>10</time-‐to-‐live-‐seconds> </map>

Wildcard configuration

IMap map1 = hz.getMap(“map1”); IMap map2 = hz.getMap(“map2”); !<map name=“map*”> <time-‐to-‐live-‐seconds>10</time-‐to-‐live-‐seconds> </map>

Wildcard configuration

XML Variables

Map: In Memory Format

• 2 Options

• BINARY

• OBJECT

• Predicates

• EntryProcessor

!ILock lock = hz.getLock(“someLock”); IAtomicLong counter = getAtomicLong(“someCounter”); IMap map = hz.getMap(“someMap”); map.put(“somePerson”,new Person()); map.get(“somePerson”);

Data Locality: Problem

Member 1

Member 2

Member 3

SomeLock

SomeCounter

SomePerson

name@partitionkey

ILock lock = hz.getLock(“someLock@foo”);IAtomicLong counter = hz.getAtomicLong(“someCounter@foo”); IMap map = hz.getMap(“someMap”); map.put(“somePerson@foo”, new Person()); Person p = map.get(“somePerson@foo”);

Data Locality: Fixed

Member 1

Member 2

Member 3

SomeLock

SomeCounter

SomePerson

Foo Partition

IAtomicLong c1 = … IAtomicLong c2 = hz.getAtomicLong(“c2@“+c1.getPartitionKey());

Adding object in same partition

Executor

• Execute task anywhere

• Execute task on key owner

• Execute task

• one/all/subset members

• Synchronisation

• ExecutionCallback

• Futures

Concurrency Control

• Locks

• TransactionalMap.getForUpdate

• Map.Lock

public void increment(int accountId,int amount){ Account account = accounts.get(accountId); account.balance+=amount; accounts.put(accountId, account); }

Race problem

public void increment(int accountId,int amount){ accounts.lock(accountId); try{ Account account = accounts.get(accountId); account.balance+=amount; accounts.put(accountId, account); }finally{ accounts.unlock(accountId); } }

Pessimistic Increment

public void increment(int accountId,int amount){ accounts.lock(accountId); try{ Account account = accounts.get(accountId); account.balance+=amount; accounts.put(accountId, account); }finally{ accounts.unlock(accountId); } }

Pessimistic Increment

public void increment(int accountId,int amount){ for(;;){ Account oldAccount = accounts.get(accountId); Account newAccount = new Account(oldAccount); newAccount.balance+=amount; if(accounts.replace(accountId, oldAccount,newAccount)){ return; } } }

Optimistic Increment

public void increment(int accountId,int amount){ for(;;){ Account oldAccount = accounts.get(accountId); Account newAccount = new Account(oldAccount); newAccount.balance+=amount; if(accounts.replace(accountId, oldAccount,newAccount)){ return; } } }

Optimistic Increment

Bad: Send Data to Function

Good: Send Function to Data

Read Data

Write Data

Write Function

private class BalanceTask implements Runnable,Serializable{ private int accountId, amount; ! public void run() { for(;;){ Account oldAccount = accounts.get(accountId); Account newAccount = new Account(oldAccount); newAccount.balance+=amount; if(accounts.replace(accountId, oldAccount,newAccount)){ return; } } } }

Increment with runnable

public void increment(int accountId, int amount){ BalanceTask task = new BalanceTask(accountId,amount); executorService.executeOnKeyOwner(task,accountId); }

Increment with runnable

class BalanceProcessor extends AbstractEntryProcessor<Integer,Account>{ int amount; BalanceProcessor(int amount) { this.amount = amount; } ! @Override public Object process(Map.Entry<Integer, Account> entry) { entry.getValue().balance+=amount; return null; } }

Increment with EntryProcessor

public void increment(int accountId, int amount){ BalanceProcessor processor = new BalanceProcessor(amount); accounts.executeOnKey(accountId, processor); }

Using entry processor

SPI• You can write your own distributed data-structures

• Examples

• Distributed Actors implementation

• Remote interfaces

Serialization API

• Serializable/Externalizable

• Portable

• ByteArraySerializable

• StreamSerializer

• Kryo

• Jackson Smile

• Protobuf

!boolean compress = …; !SerializerConfig serialiserCfg = new SerializerConfig() .setTypeClass(Product.class) .setImplementation(new ProductSerializer(compress)); !Config config = new Config(); config.getSerializationConfig().addSerializerConfig(serializerCfg); HazelcastInstance hz = Hazelcast.newHazelcastInstance(config);

Pluggable Serialization

Kryo SerializationSi

10,000

15,000

20,000

Serialization Kryo Kryp + Compr

IExecutorService executor = hz.getExecutor(“executor”); executor.execute(()-‐>System.out.println”hello”)); !//configuration SerializerConfig sc = new SerializerConfig(). setImplementation(new KryoSerializer(false)). setTypeClass(Runnable.class); Config config = new Config(); config.getSerializationConfig().addSerializerConfig(sc);

Kryo: Lambda expression

IMap<Integer,Account> accounts = hz.getMap(“accounts”); Map<Integer,Integer> result = accounts .map(new AccountSumMapper()) .reduce(new AccountSumReducer()) .submit()

Hazelcast 3.2? Map Reduce

Enterprise vs Community edition

• Support contracts

• Elastic Memory

• Security

• Management Center

The Future

• Topologies

• Hosted Management Center

• Dynamic Creation

• Map of Maps

• Tiered storage

We are hiring!

Questions?

You can find us at the Hazelcast booth!

Good Question? Get a Hazelcast

Devoxx 2013 - Hazelcast

Software

Transcript of Devoxx 2013 - Hazelcast

Hazelcast Essentials

DEPLOYMENT GUIDE Hazelcast IMDG Deployment and Operations ... · DEPLOYMENT GUIDE Hazelcast IMDG Deployment and Operations Guide For Hazelcast IMDG 3.11

SAP HANA Cloud Platform Community BOF @ Devoxx 2013

Distributed Systems Using Hazelcast - Jfokus · PDF file• Working for Hazelcast! ... • Vert.x! • Apache Camel! • ... Creating Hazelcast Cluster @PeterVeentjer

Disrupt 2 Grow - Devoxx 2013

Devoxx 2013 Summary/Keynotes

Spring Integration Hazelcast Extension

Vert.x devoxx london 2013

Devoxx 2013 io t

Devoxx 2013 JavaScript Software Factory

Du JavaScript propre ? Challenge accepted ! @Devoxx France 2013

JavaScript Devoxx France 2013

Vaadin roadmap-devoxx-2013

Devoxx 2013 Summary (non-technical)

Clustering Your Application with Hazelcast - Talip Ozturk (Hazelcast)

UI Engineer - the missing profession, devoxx 2013

Devoxx UK 2013: Sandboxing with the Vagrant-Binding API

Hazelcast Dima Ionut Daniel. Contents What is Hazelcast? Hazelcast Features Hazelcast Core Distributed Events Concurrent Features Distributed Computing.

54202488 Hazelcast Presentation

Hazelcast Documentation 3.2