with Kaa, Apache Cassandra, and Apache Zeppelin … · Real-time IoT data analytics and...
Transcript of with Kaa, Apache Cassandra, and Apache Zeppelin … · Real-time IoT data analytics and...
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
Real-time IoT data analytics and visualization with Kaa, Apache Cassandra,
and Apache Zeppelin
Agenda
➢ Why Kaa?➢ Why Cassandra?➢ Why Zeppelin?➢ Reference architecture overview➢ Hands-on
➢ Sandbox setup➢ Intel Edison application code walkthrough➢ Cassandra appenders configuration➢ Zeppelin configuration➢ Live demo
➢ Q&A
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
➢ Fully-featured IoT middleware platform➢ 100% open-source (Apache license 2.0)➢ Deploy anywhere: on-prem, in the cloud, in hybrid mode➢ Horizontally, elastically scalable, and fault tolerant➢ Guaranteed data delivery and reliable local storage➢ Built-in transport security➢ Efficient data serialization➢ Rapid application development using C / C++ / Java / Objective C SDKs➢ 10 Kb RAM footprint (with C SDK)➢ Integration with popular device platforms
Why Kaa?
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
➢ Fault tolerant➢ Performant➢ Horizontally scalable➢ Easy deployment➢ Integration with popular analytics platforms
Why Cassandra?
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
➢ Multiple language backend➢ Apache Spark integration➢ Data visualization➢ Dynamic forms➢ Collaboration & Publish➢ Open-source project
Why Zeppelin?
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
Problem description
Zone 1 Zone 2
Zone 3 Zone 4
Zone 5 Zone 6
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
Reference architecture
Solar panels
StructuredData
Solar panels
Raw data
Intel Edison
Kaa SDKClient application
Intel Edison
Kaa SDKClient application Kaa cluster/sandbox
Kaa node
Cassandra appender
Cassandra
Kaa node
Cassandra appender
Zeppelin instance/sandbox
Zeppelin dashboard
Zeppelin dashboard
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
Development environment setup
Sample project repository: https://github.com/kaaproject/kaa-zeppelin-sample
Kaa Sandbox: http://www.kaaproject.org/download-kaa
Intel Edison:https://docs.kaaproject.org/display/KAA/Intel+Edison
Zeppelin:http://zeppelin.incubator.apache.org
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
Data modeling
● Single zone per row● Single panel per row with zone partitions
Kaa data collection schema:{ "type":"record", "name":"PowerReport", "namespace":"org.kaaproject.kaa.sample", "fields":[ {"name":"timestamp", "type":"long"}, {"name":"zoneId", "type":{"type":"string", "avro.java.string":"String"}}, {"name":"panelId", "type":{"type":"string", "avro.java.string":"String"}}, {"name":"power", "type":"double"}]}
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
Single zone per row
● Partition Key: Zone ID● Clustering Key: Timestamp, Panel ID
Zone ID
Timestamp, Panel ID
Power
Timestamp 2,Panel ID
Power
Timestamp N,Panel ID
Power...
Zeppelin query:%cassandra select zone_id, panel_id, power as "power, MW", timeFormat(timestamp) as time from logs.zone_per_row where zone_id={{zone_id='Sierra'}} and timestamp > logs.timeDelta(toUnixTimestamp(now()), 60000) ORDER BY timestamp ASC;
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
Single panel per row with zone partitions
● Partition Key: Zone ID, Panel ID● Clustering Key: Timestamp
Zone ID,Panel ID
Timestamp N
Power
Timestamp N-1
Power
Timestamp 1
Power...
kaaproject.
org© 2016 CyberVision, Inc. All rights
reserved.
Zeppelin query:%cassandra select zone_id, panel_id, power as "power, MW", timeFormat(timestamp) as time from logs.panel_per_row where zone_id={{zone_id='Sierra'}} and panel_id={{panel_id='Panel_0'}} and timestamp > logs.tmBefore(toUnixTimestamp(now()), 60000) ORDER BY timestamp ASC;