Post on 15-Feb-2017
Spark Streaming andExpert Systems
James Nowell24 May 2016
The Problem
Expert Systems
Where Spark Fits
Overview
2
THE PROBLEM
3
Basic IoT devices don’t have much state Inferences can be made with regular conditionals
Simple Streams
4
tizen365.com
Multiple sensors Multiple streams
(sensor fusion!) Partial data Bad data Human entered data
Complex Streams
5
http://www.record-eagle.com/news/lifestyles/traverse-city-s-ice-road-trucker/article_8cb844bd-b613-586d-b7a0-974789984e1d.html
EXPERT SYSTEMS
6
AKA Production Systems, Rule Engines, Inference System CLIPS, Drools Turn a domain expert into code
Mostly deterministicTasks that are simple or procedural for an expert
Fact based reasoningIf TheseFacts then NewFacts
Rete Algorithm
What is an Expert System?
7
Determine the time for the current state
In Practice: Is it moving?
8
Calculate some basic movement information
In Practice: Is it moving?
9
Decide the truck didn’t become a space shipThe converse is also asserted elsewhere
In Practice: Is it moving?
10
Yep, the truck seems to be moving
In Practice: Is it moving?
11
Assert that the thing is, in fact, moving
In Practice: Is it moving?
12
WHERE SPARK FITS
13
Load the statePersistence mechanism varies
Generate a new state with the rules Emit events with the rules
Events are often based on state transitionsSettle the new state first
Persist it allUse a journal model
In an RDD
14
Parallelism App structure similar to any other streaming app Batch is extremely similar to streaming
THE reason to use SparkReprocess your entire corpus with the same code you use for new data
What Spark Buys Us
15
Build the rules at compile timekie-maven-plugin
Maximize condition reuse Use flag events to kill loops and aid debugging Write unit tests
LotsAnd lotsAnd lotsOf unit tests
Write fully fledged “Thing” tests
Things to Know
16
QUESTIONS?
17