Django for Internet of Things: from hackathon

to production

Anna Schneider DjangoCon US, 18 July 2016

@windupanna

I’m Anna :)

Co-founder and CTO at WattTime

WattTime.org @wattTime

UnconsciousBiasProject.org @UBP_STEM

what??

Django for Internet of Things: from hackathon

to production

everyone’s favorite framework

🎸🐍Django

Internet of Things (IoT)when a thing you don’t normally think of as a computer can transmit data and respond to controls in real time

🏡🚗📡🎮⏰

some people like to write the code on the thing, I like to talk to things through their APIs

Internet of Things (IoT)

☁️ 🚗☁️you them the thing

a really fun way to write really fragile code really quickly

hackathon

💻☕️👏⚠️

when you can ship your code and trust it while you’re having fun at DjangoCon

production

💻🍎🚢😎

design patterns (and anti-patterns) for writing and deploying Django projects

to monitor and control an IoT device using its API so you can get started fast then build well

what you’ll learn

☕️ 🚢

😱

python manage.py startproject awesome_iot_hackathon

💻☕️👏⚠️

models!

the books app

Book title

author year

Book title

author year

Author name

hometown

Book title

author year

Book title

author year

Author name

hometown

Author name

hometown

the books IoT app

Observation value

device timestamp

Device name

location

Device name

location

Device name

location

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

the IoT appDevice name

location vendor ID

Device name

location vendor ID

Device name

location vendor ID

☕️🚢

vendor’s unique ID(s) whatever data you need

to send to their API

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

the IoT app

plan for big(gish) time series data eg use db_index

🚢

Device name

location vendor ID

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Device name

location vendor ID

Device name

location vendor ID

the IoT app

Attribute vs Status numerical vs str/bool

values

Observation value

device timestamp

Device name

location vendor ID

Observation value

device timestamp

Observation value

device timestamp

Attribute value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Attribute value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Attribute value

device timestamp

Device name

location vendor ID

Device name

location vendor ID

Observation value

device timestamp

Observation value

device timestamp

Observation value

device timestamp

Status is_on

device timestamp

☕️🚢

views?models

views? tasks!

☁️ 🚗☁️you them the thing

request

response

☁️ 💻you them

request

response

normal views

IoT tasks

what tasks do we need to do?

pull new attributes pull new on/off statuses

set attributes set on/off statuses

monitor control

tasks.pydef do_something_awesome(device):

# decide to turn on or off is_on = run_decision_criterion()

# set status using device API set_status(device, is_on)

# create status in db status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), )

# return return status

☕️

the awesome part

tasks.pydef do_something_awesome(device):

# decide to turn on or off is_on = run_decision_criterion()

# set status using device API set_status(device, is_on)

# create status in db status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), )

# return return status

🚢

bad for asynchronous

😱

tasks.pydef do_something_awesome(device_id): # get device from pk device = Device.objects.get( pk=device_id )

# decide to turn on or off is_on = run_decision_criterion()

# set status using device API set_status(device, is_on)

# create status in db status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), )

# return return [status.pk]

pass pks pro: don’t rely on database state

con: may be extra queries

🚢

put it together!

models

tasks

the hackathon appmyapp

models.py Device Attribute Status

tasks.py pull_status set_status pull_attributes set_attributes

client.py admin.py views.py tests.py

☕️

the production apps

devices models.py admin.py views.py tests.py

interactions tasks.py views.py tests.py

vendor client.py tests.py

observations models.py admin.py views.py tests.py

views for adding/removing devices

analytics, DRF for dashboards

models for logging, views for clickable tasks

swappable vendors

🚢

deploy!

models

tasks

apps

deploy tasks?

goals: • run any task (control or monitor) • at frequent, deterministic times • outside of request/response cycle

cron in the cloud

1) the hackathon way

two ways to automate

management commands + Heroku Scheduler

☕️

python manage.py do_something_awesome --device_id=1

Heroku Scheduler

pros: • easy to set up

cons: • limited frequencies

(daily, hourly, 10 min) • “best effort” service

(may skip some)

☕️

2) the production waytask functions + celery periodic task scheduler

🚢two ways to automate

Celery

distributed message queuing system for asynchronous stuff

slow event-driven tasks

send the user sign-up email

scheduled periodic tasks

run the daily report

🚢

web servers

Celery architecture (how)

tasks

tasks

worker servers

messages results result storemessage broker transport queue

messages

tasks

scheduler

🚢

Celery architecture (how)

schedulerdo_awesome do_awesome

workerstatus.pk

☁️turn on

on!Status(is_on=True)

🚢

from celery import shared_task

@shared_task def set_status(device_id): ...

@shared_task def do_something_awesome(device_id): ...

just add decorator :)

tasks.py

🚢

(what)

from celery.schedules import crontab

SCHEDULE = { 'run_task': { # use the actual path 'task': ‘do_something_awesome’,

# every 5 minutes 'schedule': crontab(minute='*/5'),

# args and kwargs 'args': [], 'kwargs': {},

}, }

very close to crontab once per minute - once per year

schedule.py

🚢

(when)

from celery.schedules import crontab

SCHEDULE = { 'run_task': { # use the actual path 'task': ‘do_something_awesome’,

# every 5 minutes 'schedule': crontab(minute='*/5'),

# args and kwargs 'args': [], 'kwargs': {'device_id': d.pk},

} for d in Device.objects.all() }

schedule.py

🚢

static arguments only code is only evaluated once at run time, better to have one task spawn daughters

(when)

😱

from celery.schedules import crontab

SCHEDULE = { 'run_task': { # use the actual path 'task': ‘run_all',

# every 5 minutes 'schedule': crontab(minute='*/5'),

# args and kwargs 'args': [], 'kwargs': {},

} }

@shared_task def run_all(): for d in Device.objects.all(): do_something_awesome(d.pk)

schedule.py

🚢

static arguments only code is only evaluated once at run time, better to have one task spawn daughters

(when)

production, Celerified

devices models.py admin.py views.py tests.py

interactions tasks.py schedule.py views.py tests.py

brand client.py tests.py

observations models.py admin.py views.py tests.py

🚢

django_iot __init__.py celery.py settings.py wsgi.py urls.py

django_iot Procfile requirements.txt manage.py

cookiecutter https://github.com/aschn/cookiecutter-django-iot.git

☕️🚢

• think time series data: mind your models • async not request/response: tasks not views • ☕ ️easy but rigid: Heroku Scheduler • 🚢 flexible but complex: celery periodic tasks • hack better+faster: cookiecutter-django-iot • IoT can be for good, not just for fun and profit

what have we learned?

Thanks!!

Anna Schneider @windupanna

@wattTime

cookiecutter https://github.com/aschn/cookiecutter-django-iot.git