A Backend for Sensor Data - thomas-zastrow.de · Tom Zastrow, 7th December 2017 InfluxDB • NoSQL...
Transcript of A Backend for Sensor Data - thomas-zastrow.de · Tom Zastrow, 7th December 2017 InfluxDB • NoSQL...
Tom Zastrow, 7th December 2017
A Backend for Sensor Data
Tom Zastrow, 7th December 2017
Overview
• Hardware– Sensors, Arduino & Raspberry Pi integration
• Software– Sending data around
– Storing data
– Access, analysis and visualization
Tom Zastrow, 7th December 2017
Hardware
Dust
MQ2
MQ135
DHT11
Temp.
Humidity
BMP180
Temp.
Pressure
Tom Zastrow, 7th December 2017
GUIs
Data storage
Sensors
Arduino
Raspberry Pi
RESTstyle
webservice
Influx DB
Yii based
web interface
USB Serial
InfluxDB
line
format
RESTstyle calls
Chronograf,
Grafana, ...
Any other
sensor data
Json via
Internet
Ard
uin
o C
Pyth
on
PHP,
JavaScript
& InfluxDB
SQL
dialect
Tom Zastrow, 7th December 2017
Ingest: Data Flow
RESTstyle
webservice
Influx DB
Json via
Internet
InfluxDB
line
format
{"esp8266id": "tomsensors", "software_version": "arduino2.py",
"sensordatavalues":[{"value_type":"dust",
"value":"736.41"},{"value_type":"mq2",
"value":"356.0"},{"value_type":"mq135",
"value":"247.0"},{"value_type":"DHT11temperature",
"value":"19.0"},{"value_type":"DHT11humidity",
"value":"19.0"},{"value_type":"BMP180pressure",
"value":"947.66"},{"value_type":"BMP180temperature",
"value":"20.4"},{"value_type":"BMP180altitude", "value":"560.98"}]}
sidtomsensors
dust=736.41,mq2=356.0,mq135=247.0,DHT11temperature=19.0,DHT11humidity=19.0,B
MP180pressure=947.66,BMP180temperature=20.4,BMP180altitude=560.98
1512053085000000000
Tom Zastrow, 7th December 2017
InfluxDB
• NoSQL database specialized on time series
• Part of bigger software stack (including Chronograf)
• SQL like query syntax:
select max(BMP180pressure), min(BMP180pressure) from sidtomsensors; name: sidtomsensors time max min ---- --- --- 1970-01-01T00:00:00Z 949.35 946.68
curl -G "http://localhost:8086/query" --data-urlencode "q=select max(BMP180pressure),min(BMP180pressure) from sidtomsensors" --data-urlencode "db=feinstaub"
{"results":[{"statement_id":0,"series":[{"name":"sidtomsensors","columns":["time","max","min"],"values":[["1970-01-01T00:00:00Z",949.35,946.68]]}]}]}
Tom Zastrow, 7th December 2017
GUI
• Build with PHP based framework Yii
• Similar to Django
• Bootstrap already on board
• User management
• Forms
• Diagrams & charts
• Current features:– CSV import from Madavi server
– Generic query execution (inc. CSV export)
– Geo visualization done with JLeaflet JavaScript library
– Chart & graph visualizations
Tom Zastrow, 7th December 2017
Demo!
Tom Zastrow, 7th December 2017
Thanks!
Tom Zastrow, 7th December 2017
Some more things ...
Tom Zastrow, 7th December 2017
Tools, libraries, frameworks, ...
BagIt: File Packaging Formathttps://tools.ietf.org/html/draft-kunze-bagit-14
Swagger: The Worlds most popular API Toolinghttps://swagger.io/
Yii: The Fast, Secure and Professional PHP Frameworkhttp://www.yiiframework.com/
Colorbrewer2: Color advice for cartographyhttp://colorbrewer2.org/
PyCharm: Python IDEhttps://www.jetbrains.com/pycharm/
JLeaflet: JavaScript Library for interactive mapshttp://leafletjs.com/
JsonAPI: A specification for building APIs in Jsonhttp://jsonapi.org/
Slack: Online collaborationhttp://www.slack.com
Orange: Data Mining Fruitful and Funhttps://orange.biolab.si
InfluxDB: Time Series DBhttps://www.influxdata.com/
Affinity Photohttps://affinity.serif.com/de/photo/
Sensor stuffArduino & Raspi,...
Tom Zastrow, 7th December 2017
Orange: Data Mining Fruitful and Fun
• Data analytics
• Data visualization
• Machine learning
„Open source machine
learning and data
visualization for novice
and expert. Interactive
data analysis workflows
with a large toolbox.“
https://orange.biolab.si/
Tom Zastrow, 7th December 2017
Features
Interactive Data Visualization Visual Programming
Add-ons Extend Functionality
Teachers and Students Love It