Real Time Marketing - Volker Janz - Buzzwords 2016

2

INNOGAMES

VolkerJanzTeamLeadSoftwareDeveloper

@prenomenonlinkedin.com/in/vjanzxing.com/profile/Volker_Janz3

3

InnoGames isoneoftheworldwideleadingdevelopersandpublishers ofonlinegames.

Thecompanyhasover160millionregisteredplayersfromallovertheworld.

Morethan400people fromover30countriesarecurrentlyworking intheofficesinHamburgandDusseldorf.

INNOGAMES

4

How to create asuccessful company withfree-to-playonlinegames?

5

MOTIVATION

Develop anawesomegame

Get users Sellvirtual goods

Profit!

7

MOTIVATION

Profit!

Develop anawesomegame

Get users Sellvirtual goods

8

MOTIVATION

Marketing CRM

Email,Banner,TV,Social Media

Email,Interstitials,In-Gamemessages

9

MOTIVATION

Marketing CRM

Email,Banner,TV,Social Media

Email,Interstitials,In-Gamemessages

10

MOTIVATION

11

MOTIVATION

Thisis Bob.Bobplays Forge of Empires.

12

MOTIVATION

Bobgets aninterstitial butit is notusefulfor him.

Get 10%free diamonds now!

BUY

13

MOTIVATION

Bobcontinues to play the game.Now heis alreadylosing abattle for the third timeinarow.

14

MOTIVATION

Bobis frustrated butafew seconds laterhegets auseful offer.

Get 4BattleTanksfor adiscount!

BUY

15

MOTIVATION

Bobfinally wins the battle and continues to playthe game – achurn wasprevented.

16

Individualmarketing based onuser behavior inreal-time

17

SYSTEMOVERVIEW

quest

build

fight invite

18

SYSTEMOVERVIEW

500.000.000

19

SYSTEMOVERVIEW

500.000.000EVENTSPERDAY

seconds

20

SYSTEMOVERVIEW

Games

OtherSystems

Gateway

Near /Real-TimeProcessing

Hadoop

BatchProcessing

MessageBus

BusinessIntelligence

Analytics

Other(e.g.PowerBI)

DATAPIPELINE DATAPLATFORM

milliseconds hours

ProductionStorage

21

SYSTEMOVERVIEW

Games

OtherSystems

Gateway


Hadoop

BatchProcessing

MessageBus


Analytics

Other(e.g.PowerBI)


ProductionStorage

22

SYSTEMOVERVIEW

Games

OtherSystems

Gateway


MessageBus

DATAPIPELINE

ProductionStorage

Real-TimeCRMisuse case of theDataPipeline

23

SYSTEMOVERVIEW

GamesGa

teway


MessageBus

ProductionStorage

Sendevents Pushevents tothe Message Bus

Process eventsas adata stream

UpdatePlayerProfile

Trigger campaignbased onPlayerProfileand event

24

SYSTEMOVERVIEW

Keytechnologies:

Sendevents Pushevents tothe Message Bus

Process eventsas adata stream

UpdatePlayerProfile

Trigger campaignbased onPlayerProfileand event

http://kafka.apache.org/

26

KAFKA

Kafka isadistributed,partitioned,replicatedcommitlogservice.Itprovidesthefunctionalityofamessagingsystem,butwithauniquedesign.

- kafka.apache.orgProducer Producer Producer

KafkaCluster(Broker)

Consumer Consumer Consumer

27

KAFKA

Producer

KafkaCluster

Consumer

Broker1

Broker2

Partition0Replica 1

Partition1Replica 2

Partition0Replica 2

Partition1Replica 1

topic:“logs“ topic:“logs“

28

KAFKA

@

kafka01 kafka02

kafka03 kafka04

kafka05

8cores64Gmemory1.8Tdisk space

server.properties (version: 0.8.2.2):…num.replica.fetchers=4controller.message.queue.size=10auto.leader.rebalance.enable=truecontrolled.shutdown.enable=true

num.partitions=8log.retention.hours=168log.flush.interval.ms=10000log.flush.interval.messages=20000log.flush.scheduler.interval.ms=2000log.roll.hours=48log.retention.check.interval.ms=300000log.segment.bytes=1073741824

num.network.threads=8socket.request.max.bytes=104857600socket.receive.buffer.bytes=1048576socket.send.buffer.bytes=1048576queued.max.requests=16fetch.purgatory.purge.interval.requests=100producer.purgatory.purge.interval.requests=100…

29

KAFKA

@

kafka-topics.sh --list --zookeeper x.x.x.x:2181

crm_triggermaintenanceraw

/opt/kafka/app/bin/kafka-topics.sh --zookeeper x.x.x.x:2181 --describe --topic raw

Topic:raw PartitionCount:16 ReplicationFactor:2 Configs:retention.ms=345600000Topic: raw Partition: 0 Leader: 3 Replicas: 3,4 Isr: 3,4Topic: raw Partition: 1 Leader: 4 Replicas: 4,5 Isr: 4,5Topic: raw Partition: 2 Leader: 5 Replicas: 5,1 Isr: 5,1Topic: raw Partition: 3 Leader: 1 Replicas: 1,2 Isr: 1,2Topic: raw Partition: 4 Leader: 2 Replicas: 2,3 Isr: 2,3Topic: raw Partition: 5 Leader: 3 Replicas: 3,5 Isr: 3,5Topic: raw Partition: 6 Leader: 4 Replicas: 4,1 Isr: 1,4Topic: raw Partition: 7 Leader: 5 Replicas: 5,2 Isr: 2,5Topic: raw Partition: 8 Leader: 1 Replicas: 1,3 Isr: 1,3Topic: raw Partition: 9 Leader: 2 Replicas: 2,4 Isr: 2,4Topic: raw Partition: 10 Leader: 3 Replicas: 3,1 Isr: 1,3Topic: raw Partition: 11 Leader: 4 Replicas: 4,2 Isr: 2,4Topic: raw Partition: 12 Leader: 5 Replicas: 5,3 Isr: 3,5Topic: raw Partition: 13 Leader: 1 Replicas: 1,4 Isr: 1,4Topic: raw Partition: 14 Leader: 2 Replicas: 2,5 Isr: 2,5Topic: raw Partition: 15 Leader: 3 Replicas: 3,2 Isr: 2,3

http://storm.apache.org/

31

STORM

ApacheStormisafreeandopensourcedistributedrealtime computationsystem.

- storm.apache.org

32

STORM

SPOUTDatastream sources

33

STORM

BOLTDatastream processing

34

STORM

TOPOLOGYComposition of Spouts and Bolts

35

STORM

EXAMPLETemperature alerting

TemperatureBoltReadsthe current temperature

from asensor every fewmilliseconds

FilterBoltChecksif the temperature value is higherthan aspecific threshold – only inthis case

itsendsoutatuple

WarnBoltIf it receives atuple it raises

analarm

36

STORM

public class FilterBolt extends BaseRichBolt {private OutputCollectorBase collector;

@Overridepublic void prepare(Map conf, TopologyContext ctx, OutputCollectorBase collector) {

this.collector = collector;}

@Overridepublic void execute(Tuple input) {

int temperature = input.getInteger(0);

if(temperature >= 50) collector.emit(new Values(temperature));collector.ack(input);

}

@Overridepublic void declareOutputFields(OutputFieldsDeclarer declarer) {

declarer.declare(new Fields("temperature"));}

}

FilterBoltChecksif the temperature value is higherthan aspecific threshold – only inthis case

itsendsoutatuple

37

STORM

PlayerPartitioner

Bolt

CRMBolt

KafkaBoltKafkaSpout

ValidationBolt

CRMTopology

38

STORM

CRMBolt

GameEvent

PlayerProfile

UpdateHandler

CampaignHandler

PlayerProfile

CampaignEvent

Executeupdatehandlers that aredefined for the incoming event.

Executecampaign handlers thatare defined for the incomingevent.

Handlersare JavaScriptsnippetsthat are processed by theNashornengine.

1

2

39

STORM

public class JsEngine {

private final ScriptEngine engine;

public JsEngine(final EventHandlerRegistry registry, final Storage storage) {final NashornScriptEngineFactory nashornScriptEngineFactory = new NashornScriptEngineFactory();engine = nashornScriptEngineFactory.getScriptEngine(

new String[] {"-doe", "-strict"},Thread.currentThread().getContextClassLoader(),new JsClassFilter()

);

put("registry", registry);put("storage", storage);put("objectMapper", new ObjectMapper());

}

public void put(final String key, final Object value) {engine.put(key, value);

}

public Object eval(final String string) throws ScriptException {return engine.eval(string);

}

public Object get(final String key) {return engine.get(key);

}

}

40

STORM

var identifier = "count-login-events"

var filter = new EventFilter("login")

var action = new EventAction(function(event) {framework.actions.countEvent(event)

})

framework.registerUpdater(identifier, new EventHandler(filter, action))

var content = {"id": 3, "game": "foe", "markets": ["de"]}

var trigger = {"eventType": "fight", "fn": function(event) {return (event.getData().result == "loss")

}}

var segment = function(player) {return (

player.isPayer() == false &&player.getIngameLevel() < 100 &&player.fightsLoss() > 2

)}

framework.registerCampaign(content, trigger, segment)

Updater:

Campaign:

41

STORM

0.10.0Nov em be r 2 0 1 5

F lux

Security

Roll ing Upgrades

Improved Logging

42

STORM

1.0.0Ap r i l 2 0 1 6

Improved Performance

Distr ibutedCache

Pacemaker

StateManagement

43

STORM

1.0.0Ap r i l 2 0 1 6

Automat ic Backpressure

Nat iveWindowing API

Resource AwareScheduler

Debugging /Prof i l ing

44

STORM

Alternatives?

KafkaStreams SparkStreaming

Flink

45

STORM

Checkout:

TheStreamProcessor as aDatabase:BuildingOnlineApplicationsdirectly onStreamswith ApacheFlinkand ApacheKafkaStephanEwenBerlinBuzzwords 2016

Introducing KafkaStreams,the new stream processing library ofApacheKafkaMichaelNollBerlinBuzzwords 2016

TheFutureof ApacheStormP.TaylorGoetzHadoop Summit Dublin2016

http://cassandra.apache.org/

47

CASSANDRA

ApacheCassandra isamassivelyscalableopensourcenon-relationaldatabase thatofferscontinuousavailability,linearscaleperformance,operationalsimplicity andeasydatadistributionacrossmultipledatacentersandcloudavailabilityzones.

- Datastax

DynamoDB BigTable

48

CASSANDRA

SortedMap<RowKey,SortedMap<ColumnKey,ColumnValue>>

Hashvalue

49

CASSANDRA

Badmodel Good model

ddmmyyhhTimestamp 1 Timestamp 2

Payload1 Payload 2ddmmyyhhplayer_id

TimeUUID1 TimeUUID2

Payload1 Payload 2

• Spread data evenly across the cluster.• Minimize number of partitions read.• Ensure that row and column keys are unique.• Modelcolumn families around query patterns.• Denormalize and duplicate data for read performance.

Non-goals Goals

• Minimize number of writes.• Minimize data duplication.

50

CASSANDRA

CREATE TABLE IF NOT EXISTS player_profile.event (unique_player_id bigint,day int, # yyyymmddtimeid timeuuid,type text,event text,PRIMARY KEY ((unique_player_id, day), timeid)

);

unique_player_id=1day=20160501

timeid=b9e29dc7-…

type=fight event={…}

timeid=b6e29dc7-…

type=pay event={…}

http://spark.apache.org/

52

SPARK

ApacheSparkisafast andgeneral-purpose clustercomputingsystem.

- spark.apache.org

53

SPARK

C*

C*

C*

C*

C*Spark

C*Spark

C*Spark

C*Spark

https://github.com/datastax/spark-cassandra-connector+

DataLocality=

Analyticalqueries onCassandradata

54

SPARK

import com.innogames.spark.jobserver.extras.JavaSparkCassandraJob;import com.typesafe.config.Config;import org.apache.spark.sql.cassandra.CassandraSQLContext;

public class App extends JavaSparkCassandraJob {

public static final String CASSANDRA_TABLE = "event";

@Overridepublic Object runJob(Object sc, Config jobConfig) {

final CassandraSQLContext cassandraSqlContext = (CassandraSQLContext) sc;cassandraSqlContext.setKeyspace("player_profile");

if(!cassandraSqlContext.isCached(CASSANDRA_TABLE)) {cassandraSqlContext.cacheTable(CASSANDRA_TABLE);

}

return cassandraSqlContext.table(CASSANDRA_TABLE).javaRDD().filter(row -> row.getInt(1) % 2 == 0

).count();}

}https://github.com/spark-jobserver/spark-jobserver

Cached PlayerProfileusing ashared Sparkcontext:

CONCLUSION

seconds

56

Games

OtherSystems

Gateway


Hadoop

BatchProcessing

MessageBus


Analytics

Other(e.g.PowerBI)


milliseconds hours

ProductionStorage

CONCLUSION

Kafkais awesome.

Stormmakes it easyto implement real-timeapplications but...

...we had performance issues and defining topologies wasnotdeveloper friendly.

Thesedisadvantages and issues were resolved with Storm0.10.0and 1.0.0.

Reacting to user behavior inreal-timeis notonly achallenge for data engineers butalsofor game designers.

Combining Stormwith Nashornwasagreat idea that enables us to define processing logicwith JavaScriptthat can be changed atruntime.

Real Time Marketing - Volker Janz - Buzzwords 2016

Documents

Transcript of Real Time Marketing - Volker Janz - Buzzwords 2016