2
INNOGAMES
VolkerJanzTeamLeadSoftwareDeveloper
@prenomenonlinkedin.com/in/vjanzxing.com/profile/Volker_Janz3
3
InnoGames isoneoftheworldwideleadingdevelopersandpublishers ofonlinegames.
Thecompanyhasover160millionregisteredplayersfromallovertheworld.
Morethan400people fromover30countriesarecurrentlyworking intheofficesinHamburgandDusseldorf.
INNOGAMES
13
MOTIVATION
Bobcontinues to play the game.Now heis alreadylosing abattle for the third timeinarow.
14
MOTIVATION
Bobis frustrated butafew seconds laterhegets auseful offer.
Get 4BattleTanksfor adiscount!
BUY
seconds
20
SYSTEMOVERVIEW
Games
OtherSystems
Gateway
Near /Real-TimeProcessing
Hadoop
BatchProcessing
MessageBus
BusinessIntelligence
Analytics
Other(e.g.PowerBI)
DATAPIPELINE DATAPLATFORM
milliseconds hours
ProductionStorage
21
SYSTEMOVERVIEW
Games
OtherSystems
Gateway
Near /Real-TimeProcessing
Hadoop
BatchProcessing
MessageBus
BusinessIntelligence
Analytics
Other(e.g.PowerBI)
DATAPIPELINE DATAPLATFORM
ProductionStorage
22
SYSTEMOVERVIEW
Games
OtherSystems
Gateway
Near /Real-TimeProcessing
MessageBus
DATAPIPELINE
ProductionStorage
Real-TimeCRMisuse case of theDataPipeline
23
SYSTEMOVERVIEW
GamesGa
teway
Near /Real-TimeProcessing
MessageBus
ProductionStorage
Sendevents Pushevents tothe Message Bus
Process eventsas adata stream
UpdatePlayerProfile
Trigger campaignbased onPlayerProfileand event
24
SYSTEMOVERVIEW
Keytechnologies:
Sendevents Pushevents tothe Message Bus
Process eventsas adata stream
UpdatePlayerProfile
Trigger campaignbased onPlayerProfileand event
26
KAFKA
Kafka isadistributed,partitioned,replicatedcommitlogservice.Itprovidesthefunctionalityofamessagingsystem,butwithauniquedesign.
- kafka.apache.orgProducer Producer Producer
KafkaCluster(Broker)
Consumer Consumer Consumer
27
KAFKA
Producer
KafkaCluster
Consumer
Broker1
Broker2
Partition0Replica 1
Partition1Replica 2
Partition0Replica 2
Partition1Replica 1
topic:“logs“ topic:“logs“
28
KAFKA
@
kafka01 kafka02
kafka03 kafka04
kafka05
8cores64Gmemory1.8Tdisk space
server.properties (version: 0.8.2.2):…num.replica.fetchers=4controller.message.queue.size=10auto.leader.rebalance.enable=truecontrolled.shutdown.enable=true
num.partitions=8log.retention.hours=168log.flush.interval.ms=10000log.flush.interval.messages=20000log.flush.scheduler.interval.ms=2000log.roll.hours=48log.retention.check.interval.ms=300000log.segment.bytes=1073741824
num.network.threads=8socket.request.max.bytes=104857600socket.receive.buffer.bytes=1048576socket.send.buffer.bytes=1048576queued.max.requests=16fetch.purgatory.purge.interval.requests=100producer.purgatory.purge.interval.requests=100…
29
KAFKA
@
kafka-topics.sh --list --zookeeper x.x.x.x:2181
crm_triggermaintenanceraw
/opt/kafka/app/bin/kafka-topics.sh --zookeeper x.x.x.x:2181 --describe --topic raw
Topic:raw PartitionCount:16 ReplicationFactor:2 Configs:retention.ms=345600000Topic: raw Partition: 0 Leader: 3 Replicas: 3,4 Isr: 3,4Topic: raw Partition: 1 Leader: 4 Replicas: 4,5 Isr: 4,5Topic: raw Partition: 2 Leader: 5 Replicas: 5,1 Isr: 5,1Topic: raw Partition: 3 Leader: 1 Replicas: 1,2 Isr: 1,2Topic: raw Partition: 4 Leader: 2 Replicas: 2,3 Isr: 2,3Topic: raw Partition: 5 Leader: 3 Replicas: 3,5 Isr: 3,5Topic: raw Partition: 6 Leader: 4 Replicas: 4,1 Isr: 1,4Topic: raw Partition: 7 Leader: 5 Replicas: 5,2 Isr: 2,5Topic: raw Partition: 8 Leader: 1 Replicas: 1,3 Isr: 1,3Topic: raw Partition: 9 Leader: 2 Replicas: 2,4 Isr: 2,4Topic: raw Partition: 10 Leader: 3 Replicas: 3,1 Isr: 1,3Topic: raw Partition: 11 Leader: 4 Replicas: 4,2 Isr: 2,4Topic: raw Partition: 12 Leader: 5 Replicas: 5,3 Isr: 3,5Topic: raw Partition: 13 Leader: 1 Replicas: 1,4 Isr: 1,4Topic: raw Partition: 14 Leader: 2 Replicas: 2,5 Isr: 2,5Topic: raw Partition: 15 Leader: 3 Replicas: 3,2 Isr: 2,3
35
STORM
EXAMPLETemperature alerting
TemperatureBoltReadsthe current temperature
from asensor every fewmilliseconds
FilterBoltChecksif the temperature value is higherthan aspecific threshold – only inthis case
itsendsoutatuple
WarnBoltIf it receives atuple it raises
analarm
36
STORM
public class FilterBolt extends BaseRichBolt {private OutputCollectorBase collector;
@Overridepublic void prepare(Map conf, TopologyContext ctx, OutputCollectorBase collector) {
this.collector = collector;}
@Overridepublic void execute(Tuple input) {
int temperature = input.getInteger(0);
if(temperature >= 50) collector.emit(new Values(temperature));collector.ack(input);
}
@Overridepublic void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("temperature"));}
}
FilterBoltChecksif the temperature value is higherthan aspecific threshold – only inthis case
itsendsoutatuple
38
STORM
CRMBolt
GameEvent
PlayerProfile
UpdateHandler
CampaignHandler
PlayerProfile
CampaignEvent
Executeupdatehandlers that aredefined for the incoming event.
Executecampaign handlers thatare defined for the incomingevent.
Handlersare JavaScriptsnippetsthat are processed by theNashornengine.
1
2
39
STORM
public class JsEngine {
private final ScriptEngine engine;
public JsEngine(final EventHandlerRegistry registry, final Storage storage) {final NashornScriptEngineFactory nashornScriptEngineFactory = new NashornScriptEngineFactory();engine = nashornScriptEngineFactory.getScriptEngine(
new String[] {"-doe", "-strict"},Thread.currentThread().getContextClassLoader(),new JsClassFilter()
);
put("registry", registry);put("storage", storage);put("objectMapper", new ObjectMapper());
}
public void put(final String key, final Object value) {engine.put(key, value);
}
public Object eval(final String string) throws ScriptException {return engine.eval(string);
}
public Object get(final String key) {return engine.get(key);
}
}
40
STORM
var identifier = "count-login-events"
var filter = new EventFilter("login")
var action = new EventAction(function(event) {framework.actions.countEvent(event)
})
framework.registerUpdater(identifier, new EventHandler(filter, action))
var content = {"id": 3, "game": "foe", "markets": ["de"]}
var trigger = {"eventType": "fight", "fn": function(event) {return (event.getData().result == "loss")
}}
var segment = function(player) {return (
player.isPayer() == false &&player.getIngameLevel() < 100 &&player.fightsLoss() > 2
)}
framework.registerCampaign(content, trigger, segment)
Updater:
Campaign:
43
STORM
1.0.0Ap r i l 2 0 1 6
Automat ic Backpressure
Nat iveWindowing API
Resource AwareScheduler
Debugging /Prof i l ing
45
STORM
Checkout:
TheStreamProcessor as aDatabase:BuildingOnlineApplicationsdirectly onStreamswith ApacheFlinkand ApacheKafkaStephanEwenBerlinBuzzwords 2016
Introducing KafkaStreams,the new stream processing library ofApacheKafkaMichaelNollBerlinBuzzwords 2016
TheFutureof ApacheStormP.TaylorGoetzHadoop Summit Dublin2016
47
CASSANDRA
ApacheCassandra isamassivelyscalableopensourcenon-relationaldatabase thatofferscontinuousavailability,linearscaleperformance,operationalsimplicity andeasydatadistributionacrossmultipledatacentersandcloudavailabilityzones.
- Datastax
DynamoDB BigTable
49
CASSANDRA
Badmodel Good model
ddmmyyhhTimestamp 1 Timestamp 2
Payload1 Payload 2ddmmyyhhplayer_id
TimeUUID1 TimeUUID2
Payload1 Payload 2
• Spread data evenly across the cluster.• Minimize number of partitions read.• Ensure that row and column keys are unique.• Modelcolumn families around query patterns.• Denormalize and duplicate data for read performance.
Non-goals Goals
• Minimize number of writes.• Minimize data duplication.
50
CASSANDRA
CREATE TABLE IF NOT EXISTS player_profile.event (unique_player_id bigint,day int, # yyyymmddtimeid timeuuid,type text,event text,PRIMARY KEY ((unique_player_id, day), timeid)
);
unique_player_id=1day=20160501
timeid=b9e29dc7-…
type=fight event={…}
timeid=b6e29dc7-…
type=pay event={…}
53
SPARK
C*
C*
C*
C*
C*Spark
C*Spark
C*Spark
C*Spark
https://github.com/datastax/spark-cassandra-connector+
DataLocality=
Analyticalqueries onCassandradata
54
SPARK
import com.innogames.spark.jobserver.extras.JavaSparkCassandraJob;import com.typesafe.config.Config;import org.apache.spark.sql.cassandra.CassandraSQLContext;
public class App extends JavaSparkCassandraJob {
public static final String CASSANDRA_TABLE = "event";
@Overridepublic Object runJob(Object sc, Config jobConfig) {
final CassandraSQLContext cassandraSqlContext = (CassandraSQLContext) sc;cassandraSqlContext.setKeyspace("player_profile");
if(!cassandraSqlContext.isCached(CASSANDRA_TABLE)) {cassandraSqlContext.cacheTable(CASSANDRA_TABLE);
}
return cassandraSqlContext.table(CASSANDRA_TABLE).javaRDD().filter(row -> row.getInt(1) % 2 == 0
).count();}
}https://github.com/spark-jobserver/spark-jobserver
Cached PlayerProfileusing ashared Sparkcontext:
CONCLUSION
seconds
56
Games
OtherSystems
Gateway
Near /Real-TimeProcessing
Hadoop
BatchProcessing
MessageBus
BusinessIntelligence
Analytics
Other(e.g.PowerBI)
DATAPIPELINE DATAPLATFORM
milliseconds hours
ProductionStorage
CONCLUSION
Kafkais awesome.
Stormmakes it easyto implement real-timeapplications but...
...we had performance issues and defining topologies wasnotdeveloper friendly.
Thesedisadvantages and issues were resolved with Storm0.10.0and 1.0.0.
Reacting to user behavior inreal-timeis notonly achallenge for data engineers butalsofor game designers.
Combining Stormwith Nashornwasagreat idea that enables us to define processing logicwith JavaScriptthat can be changed atruntime.
Top Related