Esper · Esper • Language, compiler and runtimefor DSM & CEP • Basic version is a library that...
Transcript of Esper · Esper • Language, compiler and runtimefor DSM & CEP • Basic version is a library that...
Esper
• Language,compilerandruntime forDSM&CEP• BasicversionisalibrarythatcanbeimportedinJavaor.NET• EventProcessingLanguage(EPL)enablesrichexpressionsovereventsandtime
• ThemandatoryassignmentusesEsperandwillrequireyoutowriteEPLqueriestogetdatafromastream!
Documentation: http://www.espertech.com/esper/esper-documentationTry online: https://esper-epl-tryout.appspot.com/epltryout/mainform.html
Documentation: http://www.espertech.com/esper/esper-documentationTry online: https://esper-epl-tryout.appspot.com/epltryout/mainform.html
EPLSyntax• Create schema
• DataStreamManagement:
• CEP:
Documentation: http://www.espertech.com/esper/esper-documentationTry online: https://esper-epl-tryout.appspot.com/epltryout/mainform.html
select ID as sensorId, sum(countTags) as numTagsPerSensorfrom AutoIdRFIDExample#win:time(60 seconds) where Observation[0].Command = 'READ_PALLET_TAGS_ONLY’ group by ID
select a.custId, sum(a.price + b.price) from pattern [every a=ServiceOrder ->
b=ProductOrder(custId = a.custId) where timer:within(1 min)]#win:time(2 hour)
where a.name in ('Repair', b.name) group by a.custIdhaving sum(a.price + b.price) > 100
create schema SchemaName(field_name1 type, field_name2 type)
Window• Called aview inEsper• Choose correctwindowwhen writingEPLqueries!• Examples of windows/views:
View Syntax Description
Timewindow win:time(time period) Sliding timewindow
Externally-timed window win:ext_timed(timestampexpression, timeperiod)
Sliding timewindow,based on the millisecond timevalue supplied byanexpression
Timebatchwindow win:time_batch(timeperiod[,optional reference point][,flow control])
Tumbling window that batches events andreleasesthem every specified time interval,with flow controloptions
Externally-timed batchwindow
win:ext_timed_batch(timestamp_expression,time_period[,optional_reference_point])
Tumbling window,based on the millisecond timevalue supplied byanexpression
Documentation: http://www.espertech.com/esper/esper-documentationTry online: https://esper-epl-tryout.appspot.com/epltryout/mainform.html
CEPsystemoverview
CEPsystemoverview
Programming languages used
ApacheFlink• Stream processing systemwith many features• Librarieson top of batchandstream processing• CEPuses the DataStream API• CEPispatternmatchingoverevent streams
ApacheApex
• Stream processingplatform• Nativehadoopsupport• Dependsonhadoop
ApacheKafka
• OriginallybyLinkedIn,open-sourced in2011• 1stversion:High-throughputpublish subscribe• 2ndversion:High-availabilityanddurability• 3rdversion:Stream processing• Then kafkastreams-cep came(unofficial),enablingCEP
Siddhi CEP
define stream TempStream (deviceID long,roomNo int,tempdouble);@info(name ='5minAvgQuery’)fromTempStream#window.time(5min)select roomNo,avg(temp)as avgTempgroup by roomNoinsert into OutputStream;
• Stream processinglibrary,similar toEsper.• Also offersaSQL-likeinterface forquery processing.
SQLandObject-oriented interfaces
val revenue = orders .filter('cCountry === "FRANCE") .groupBy('cID, 'cName) .select('cID, 'cName, 'revenue.sum AS 'revSum)
orders = table_env.scan("Orders") evenue = (orders.filter("cCountry === 'FRANCE'").group_by("cID, cName").select("cID, cName, revenue.sum AS revSum"))
Table orders = tableEnv.scan("Orders");Table revenue = orders.filter("cCountry === 'FRANCE'").groupBy("cID, cName").select("cID, cName, revenue.sum AS
revSum");
Table revenue = tableEnv.sqlQuery("SELECT cID, cName, SUM(revenue) AS
revSum " +"FROM Orders " +"WHERE cCountry = 'FRANCE' " +"GROUP BY cID, cName");
Flink's JavaTableAPI Flink's JavaSQL
Flink’s ScalaTableAPI Flink’s PythonTableAPI
Benchmark
• Few exist ....andtheymay notbegeneralenough• Difficulties inbenchmarking:biases, lack of knowledge,unfaircomparisons
Feature comparison of DSMS/CEPlibraries
Feature comparisonof EPplatforms• Additional cloud features
• Scalability• Highavailability (HA)• Fault tolerance
Internet of ThingsUse Case
Assign 10 => Temp, 11 => Humidity, 12 => FireDefine Fire(area:string,temp: float)From Temp(value > 45) andlast Humidity([string]area=Temp.area,perc < 25)within 5000 from TempWhere area := Temp.area, temp := Temp.valueConsuming Temp, Humidity;
DistributedComplex Event Processing
• Splitqueries into subqueries• Placethem on bestnode
• Which isbestnode?• Minimize latency?• Minimize energy consumption?
DistributedComplex Event Processing
Streamprocessingvs.CEP– Stillconfused?
• Recommendedreading:• What'sthedifferencebetweenESPandCEP?ByDavidLuckham,originatorofComplexEventProcessing(CEP),authorof"ThePowerofEvents". https://en.wikipedia.org/wiki/David_Luckham• http://www.complexevents.com/2019/07/15/whats-the-difference-between-esp-and-cep-2/