Stream Reasoning/CEP
Transcript of Stream Reasoning/CEP
Ereignisse / Streams• Sensor Daten• IoT
• Business Events• DDD/CQRS/ES
• Abgeleitete Events• Log Files
Apache Spark • Batch processing framework• Designed as competition to Hadoop• Offline, data oriented processingÞHigh latency, big throughput
• Defines Cluster/distributed computing topology
Spark Streaming• Extension of Spark• Micro-Batch processing • small batch sizes ~ 1 sec, allows pseudo real-time
• Abstraction of streams with different operators • window functions, transformations, caching/persistence,…
Apache Storm• Complex Event Processing engine/framework• Designed as endless running processing framework• One event at a time => low latency, “real-time”
• Simple basic concept• Spouts produce only data• Blots consumes data, performs calculations and produce (new) data• Connected via a topology
• Low level abstraction • One has to handle “where” data is used – very imperative syntax
Apache Storm - Trident• Extension to Storm• Introduces Micro-Batching to increase throughput • Higher abstraction layer than “vanilla” Storm• More declarative than Storm topology• Concepts for persistence of data / state