circuitcellar.com 17FEA

TU

RES

Matt built a platform for measuring and recording hand gestures. He used

Micropython to write the firmware for an ARM microcontroller to communicate with

an InvenSense MPU6050 Inertial Measurement Unit (IMU). A ZigBee module enables

him to transmit data to and from a laptop wirelessly.

By Matt Oppenheim

Full-Stack Python

PHOTO 1

The hardware being tested at Beaumont. The processor

board and battery are in the pouch on the student's

wrist and the sensor board is attached to the top of

his hand.

Enabling people with restricted motor control

to interact with software through custom

gesture recognition is described as the “holy

grail” of assistive technology by Technologists

at Beaumont College in Lancaster, UK, which,

as a member of the charity Scope, educates

around 100 students with a variety of physical

and cognitive challenges. In this article, I’ll

describe how I built a platform for measuring

and recording hand gestures and then tested

it at Beaumont College (see Photo 1).

PYTHON & MICROPYTHONI used Micropython to write the firmware

for an ARM microcontroller to communicate

with an InvenSense MPU6050 inertial

measurement unit (IMU) (see Photo 2). I used

a Digi International XBee S2 ZigBee module to

transmit data to and from a laptop wirelessly.

The real-time data visualization and analysis

software was written using Python 3,

leveraging the PyQtGraph library. So I have a

system using Python from the firmware to the

user interface.

Micropython was released in May 2014. I

got the idea of using Micropython to develop

the firmware for this project while listening to

episode 17 of the Talk Python to Me podcast

series (https://soundcloud.com/talkpython),

which is a weekly podcast covering all things

“Pythonic.” Micropython enables firmware

to be written in a language using the same

structures and grammar as Python 3.

When I started programming embedded

hardware, the unofficial mantra was: “If you

can’t do it in assembly, do it in C. If you can’t

do it in C, it’s not worth doing.” The idea of

letting something as high-level as an object-

oriented (OO) language anywhere near an

embedded processor was anathema. However,

using an OO language for programming a

microcontroller is not a new idea, as Chris

Rapid Embedded Hardware and GUI Development

CIRCUIT CELLAR • FEBRUARY 2017 #31918FEATURES

Cantrell showed in a 2006 article, “Embedded

Object-Oriented Programming” (Circuit Cellar,

187).

Python is a mature object OO language

that actually predates Java, although

many have the impression that it is a more

recent language due to the recent rise in

its popularity. It is dynamically typed. This

means you don’t specify the types of the

variables, which is a different approach from

procedural, statically typed languages like C,

where every variable has to be declared (e.g.,

as an integer or a string).

Naturally, there are overheads to

shoe horning an OO language into a

microcontroller. The whole shebang soaks up

about 100 KB of RAM before you start coding,

which would have been a difficult sell with

microcontroller technology 10 years ago. The

final code will almost definitely run slower

than handcrafted C. But it is all about Time

to Solution. Assembler can be included using

a Python decorator (refer to the Micropython

documentation for details) so you can have

the best of both worlds. If I can get reliable,

maintainable code to complete my research

projects more quickly with this tool set while

remaining within the hardware budget, then it is

a viable solution. As I already code workstation-

based applications in Python 3, moving to

Micropython minimizes the change in mindset

that moving to C requires, which improves

development time and code reliability.

All of the development for this project was

done under Linux on a Lenovo X230. I moved

from Microsoft Windows to Linux a couple of

years ago and found my coding productivity

increased. Having a working knowledge of

Linux can only help my employability.

HARDWAREI enjoy designing my own hardware, but

for this project I wanted to use off-the-shelf

technology to make it straightforward for

others to replicate my work should it prove

to be of use as assistive technology. Figure 1

is a block diagram of the hardware. Ten years

ago, I would have probably had to design my

own circuit boards for this project and solder

them up. Now, to make this prototype, all I

have to do is make a few connectors to plug

together ready-made boards. This speeds up

the process considerably, but removes some of

the fun. Figure 2 is a schematic of the Pyboard

v1.0 hardware. The schematic is based on

the Pyboard v1.0 schematic available on the

Micropython github repository. I ported the

schematic from Eagle to Kicad. I was brought

up on Eagle but recently moved over to Kicad.

Why would I do that? Apart from having no

license requirements, the functionality of

the software can be extended by scripting in

guess what? Python.

Micropython has been successfully used on

a range of boards. A list is available at https://

github.com/micropython/micropython/wiki/

PHOTO 2

Here is the project’s hardware

(clockwise from the top left): battery,

MPU6050 sensor, Digi International

XBee S2, Pyboard v1.0, and Adafruit

Powerboost 500C.

circuitcellar.com 19FEATURES

Boards-Summary. I chose the Pyboard v1.0

available on the Micropython website as

it proven to work well with Micropython.

This board contains an STMicroelectronics

STM32F411RE microcontroller, which is a 96-

MHz Cortex M4 CPU with hardware floating

point with 512-KB flash ROM and 128-KB RAM.

There is a micro-SD slot and a 3.3-V voltage

regulator which can supply up to 250 mA. A

v1.1 of the board is now available with a faster

processor and more RAM, but it sips more

current, so I stuck with v1.0. Clearly, using

a microcontroller of this power is an overkill

for just sampling, filtering, and transmitting

sensor data. The long-term aim of this project

is to put the gesture recognition onto the

hardware.

WIRELESS INTERFACEFor wireless communication between the

sensor board and my laptop, I settled on XBee

modules, using the ZigBee wireless protocol.

One module—the controller module—is

connected to my laptop. A second module—the

router module—is connected to the Pyboard.

Why not use Bluetooth? I found that each

time I had a new iteration of code, I had to

repair the Bluetooth board with my laptop

before I could test it. This may only take 10 s

or so, but it was frustrating. In comparison,

XBee connects without any noticeable delay

each time that I change the firmware. The

modules can be turned off and on again during

use and the data transmission will resume.

The XBee boards are configured using the

manufacturer’s software, XCTU, so that each

knows the unique serial number of the other

one as well as the transmission data rate. A

number of parameters can be set, such as

transmit power and the input/output status

of the digital pins on the XBee module. I used

the maximum data rate that I could select

with the XCTU software of 115,200 Hz and set

the module’s transmit power to be medium.

Reading through various forums, it is possible

that this data rate can be increased through

Micro python pyboardPYBv1.0

×1

×2

×3

×4

×5

×6

×7

×8

×9

×10

×11

×12

×13 - RST

×14 - GND

×15 - 3V3

×16 - VINMPU6050

×8

×7

×6

×5

×4

×3

×2

×1

VIN - P10D

GND - P9D

VIN - P10C

GND - P9C

VIN - P10B

GND - P9B

VIN - P10A

GND - P9A

VDD

DOUT

DIN

DIO12

RESET

RSSI

DIO11

RES

DTR

GND

1

2

3

4

5

6

7

8

9

10

XBEE1

DIO0

DIO1

DIO2

DIO3

RTS

DIO5

DIO9

CTS

DIO4

Adafruit powerboost 500C

BAT1

VBATPowerboost500 charger

USB

ENABLE

LBD

5 V

GND

×1

7 -

US

R

×1

8

×1

9

×2

0

×2

1

×2

2

×2

3 -

A3

V3

×2

4 -

AG

ND

VIN - Y16

3V3 - Y15

GND - Y14

RST - Y13

Y12

Y11

Y10

Y9

Digi international XBee S2

87654321

BO

OT

0 -

P1

Blu

e L

ED

- P

2

Ye

llow

LE

D -

P3

Gre

en

LE

D -

P4

Re

d L

ED

- P

5

VB

AT

- P

6

VIN

- P

7

GN

D -

P8

FIGURE 1

Here you can see the connections

between the Pyboard v1.0, Digi

International XBee S2 module,

Adafruit Powerboost 500C board, and

MPU6050 sensor break out board.

CIRCUIT CELLAR • FEBRUARY 2017 #31920FEATURES

directly setting a register in the XBee, but I

didn’t test this.

For any XBee network, one module has

to be configured as a controller. This is

connected to my laptop through a USB port.

The module connected with the Pyboard is

set to be a router in transparent mode. The

interface with the Pyboard is a simple RX, TX

serial connection. Data sent to the router from

the Pyboard is transmitted to the controller

without any need for pairing or handshaking.

I found that the robust connection using the

XBee modules made for faster development

than with using the Bluetooth module I

initially tried. The downside is the expense;

but as this is a testing and development

platform, the increase in productivity through

using XBee outweighs the one-off expense of

the modules. There is a lot more that can be

done with these little modules than I have so

far used them for, such as mesh networks

and encryption. Having encryption available

in the hardware removes the overhead from

firmware. If I were to commercialize an

XBee-based system for gesture recognition,

encryption would be enabled, but I haven’t

used this feature so far.

SENSOR BOARDThe Pyboard contains a Freescale

Semiconductor MMA7660 accelerometer. As

this is limited to ±1.5 g and 6 bits of resolution,

I opted for an external accelerometer/

gyroscope board, the InvenSense MPU6050.

This contains a three-axis accelerometer and

a three-axis gyroscope. The accelerometer

has 16 bits of resolution and the full-scale

deflection range can be set. I used the ±4 g

range and this fits well with the range that

my participants generate with their gestures.

Selecting suitable accelerometer limits allows

the full range of the sensor’s built-in analog-

to-digital converter (ADC) to be used, which

maximizes the resolution of the data.

The interface between the microcontroller

and the sensor is via I2C. I bought the sensor

on a breakout board from eBay. This is

connected to the Pyboard using 0.1" headers

and lengths of insulated wire. My first iteration

of connectors used PVC insulated ribbon cable,

FIGURE 2

The Pyboard v1.0 (Source: micropython github)

CIRCUIT CELLAR • FEBRUARY 2017 #31922FEATURES

as this is what I had to hand. I upgraded

this to 28 AWG silicone insulated wire as this

wire is more flexible than PVC insulated wire,

which is better for wearable technology. I

find that PVC insulation melts back in a messy

and unpredictable way when being soldered,

whereas the silicone insulation does not melt.

The board was powered from a 3.3-V

pin on the Pyboard. The MPU-6050 can be

sampled at up to 8 kHz, which was far faster

than needed for my experiments. I tested it

at frequencies up to 400 Hz. The bottleneck

for data collection was the speed of the XBee

wireless connection.

BATTERY MANAGEMENT BOARDTo power the hardware, I use a lithium

cell connected to an Adafruit Powerboost

500C Charger board which houses a Texas

Instrument TPS61090 DC-DC converter. This

board has a small form factor and several

other features which make it suitable for the

project. The board connects to a 3.7-V lithium

cell and has an output of 5.2 V. Having the

extra 0.2 V above 5 V means there is less

chance of a 5-V circuit having a brown-out

from to the rail dropping below 5 V due to

a transient increase in current draw. An

attached battery can be recharged by using 5

V supplied through the USB micro connector.

The board can be used without a battery

being attached, running directly from 5 V

supplied through the USB connector. This is

useful for development.

FIRMWAREStick a micro-USB cable from your

workstation into the Pyboard and you can start

to develop. The board will appear as a drive

on one of the /dev/ttyACM ports. Bring up a

file explorer and you can see the Micropython

files on the board. Each time that the board is

powered or reset, the Micropython file named

main.py is run using the interpreter on the

board.

On Linux, the following command brings

up a read-evaluate-print-loop (REPL) screen.

screen /dev/ttyACM*

* is the port number to which the board

connects.

Opening the REPL screen enables you to

interactively run Micropython commands

directly on your microcontroller. Control-C

terminates the running Micropython program.

Control-D performs a soft reboot, which

sets main.py running using the Micropython

interpreter on the board. You can install

Micropython directly onto your desktop and

try out commands on that if you prefer.

Learn from my mistakes. One thing to LISTING 1

main_accelerometer.py

def __init__(self): print(‘starting main_accelerometer with frequency: {}’.format(SAMPLE_FREQ)) micropython.alloc_emergency_exception_buf(100) try: self.imu = MPU6050() except NameError as e: print(‘check imu is connected’) print(e) except Exception as exc: print(exc) self.sw = pyb.Switch() self.sw.callback(self.switch) self.run_flag = True self.write_flag = False self.led_3 = pyb.LED(3) self.led_3.on() self.start_time = pyb.micros() self.acc_read_flag = False self.counter = 0 self.acc_data = None self.usb = pyb.USB_VCP() # use uart(3) for the XBee on y9,y10 self.uart = UART(3, 115200) self.uart.init(115200, bits=8, parity=None, stop=1) self.loop()

def loop(self): ‘’’ read and write data from the self.imu ‘’’ self.set_acc_sample_rate(SAMPLE_FREQ) timer_write = pyb.Timer(2, freq=READ_FREQ) timer_write.callback(self.write_usb_true) (delta, old_x, old_y, old_z) = self.read_acc() # use struct to pack the data string to send over uart packer = (‘2shhfff2s’)

while(True): if (self.run_flag): if(self.acc_read_flag): self.counter+= 1 (delta, x, y, z) = self.read_acc() x_acc = x x = self.filter(old_x, x) y = self.filter(old_y, y) z = self.filter(old_z, z) values = (START, self.counter, delta, x, y, z, END) packed_data = struct.pack(packer,*values) # send to xbee via uart(3) self.uart.write(packed_data) old_x, old_y, old_z = x, y, z self.acc_read_flag = False if(self.write_flag): if self.usb.any(): self.process_received(self.read_serial()) if self.uart.any(): self.process_received(self.read_uart()) self.write_flag = False

circuitcellar.com 23FEATURES

avoid is to inadvertently install a Micropython

library over of a regular Python library,

especially in Linux. Use the upip command to

install a Micropython library in Linux.

./Micropython -m upip install micropython- collections

I use a combination of the Eclipse

interactive design environment with the Pydev

plugin and the serial terminal communications

software CoolTerm to interact with the board

over a serial port. These are free software

tools. I open the Micropython file that I am

working on using Eclipse. I connect with the

board using CoolTerm on the /dev/ttyACM*

port. After saving the edited file from Eclipse

to the Pyboard, I do a soft reset on the board

using CoolTerm and see the output from the

board on the CoolTerm screen.

You can split your firmware across files.

One of the tenets of OOP is to have separate

classes or functions for different tasks. A

class should be open to extension but closed

to modification. There are many excellent

books on Python software designs. I included

a couple of links in the Resources section at

the end of this article. Two of my favorites

are Idiomatic Python and The Small Book of

Python Anti-Patterns.

I find it convenient to use main.py just to

start whichever project I am working on at

the moment. This way you can save several

projects on the pyboard and decide which

one to run by just changing a single line of

code in main.py. So for this project, main.py

instantiates the class main_accelerometer.py.

In main_accelerometer.py, I set up sampling

the accelerometer sensor and handle two-way

communication with the laptop.

When a class is first instigated, the __init__ method is called. Listing 1 shows

the __init__ and loop methods for the class

main_accelerometer.py, which is started each

time the Pyboard receives either a hardware

or software reset. There are many libraries

available for data processing in Python such

as numpy (numerical processing for Python).

Due to the limitations of the hardware that

Micropython is implemented on, not all of these

libraries have been ported to Micropython. Line

4 has the python statement try. This will try

the next line(s) of code, and if an exception is

raised, this will be dealt with by the subsequent

exception statement without the program

crashing. In Python, “it is better to beg for

PHOTO 3

Real-time display of three-axis

accelerometer data from the MPU6050

using PyQtGraph

CIRCUIT CELLAR • FEBRUARY 2017 #31924FEATURES

circuitcellar.com/ccmaterials

RESOURCES

J. Knupp, Writing Idiomatic

Python, https://jeffknupp.

com/writing-idiomatic-

python-ebook/.

A. Dewes and C. Neumann,

Python Anti-Patterns: The

Little Book of Python Anti-

Patterns and Worst Practice,

QuantifiedCode, 2016.

R. Faludi, Building Wireless

Sensor Networks, O’Reilly,

2011.

PyDispatcher, http://

pydispatcher.sourceforge.net/.

TalkPython, “Episide #17: Python on

Bare Metal with MicroPython," 2015,

http://talkpython.fm/episodes/show/17/

python-on-bare-metal-with-micropython.

SOURCES

Powerboost 500C board

Adafruit Industries | www.adafruit.com/

product/1944

XBee S2 Module

Digi International | www.digi.com

MPU6050 MEMS MotionTracking Device

InvenSense | www.invensense.com

Pyboard v1.0

MicroPython | www.micropython.org

Coolterm serial port tool

Roger Meier | http://freeware.the-meiers.org/

STM32F411RE

STMicroelectronics | www.st.com

Eclipse IDE

The Eclipse Foundation | www.eclipse.org

forgiveness than ask for permission.”

This is a useful feature for firmware, as

we can try running code as if a sensor was

attached. Then, when an exception is raised

because it is not attached, this can be dealt

with without crashing the program. So, you

can set up your code to allow peripherals to

be added or removed while the firmware is

running by handling the exceptions that are

raised by the hardware not being present.

Line 5 creates an object called self.

imu object which represents the MPU6050

sensor. The code for this is in a separate

file, mpu6050.py. On the Micropython forum

(http://forum.micropython.org), I found that

somebody else had already created a class

to enable using the MPU6050 sensor and

put it on a github, so in the best tradition of

hardware development, I lifted it and made a

few alterations to fit it into my project.

Refer to line 46 in Listing 1. I use the

struct construct to pack the accelerometer

data scan before sending it to the laptop. The

format of the packed data is defined by the

packer variable in line 34. This is a standard

way of transmitting data of a known format.

I add a two-character start and end string

for error-checking purposes. Lines 42–44

show that I apply a simple real-time filter

to the data to smooth out anomalies using

a simple 1-alpha filter. More information

on this filter is posted on my website,

mattoppenheim.com. Line 52 checks for data

coming in through the USB port for when the

Pyboard is directly connected to my laptop

with a USB cable. Line 54 looks for UART data

coming from the XBee, which is connected

to this port. I define which of the Pyboard’s

UART ports is used in line 23 and set up the

transmission speed in line 24. To get timing

information, I use the pyb.micros function in

line 17. The time between samples is sent

with the sensor data so that the sampling

frequency can be checked.

SOFTWAREThe accelerometer data is received

through the controller XBee module attached

to my laptop. The data visualization software

is written in Python3. The code is split across

a number of classes to separate the different

parts of the system which makes it easier to

structure, maintain and extend.

The main.py class starts a graphical user

interface (GUI) thread which sets up a real-

time display for the accelerometer data. It

also starts a data collection thread which

collects the accelerometer data and puts it

into an array structure.

If an XBee module is attached to the

laptop, a serial port will be created. If this

serial port exists, the data collection thread

assumes that this is where accelerometer

data will be coming from. I wrote a script

to generate simulated accelerometer data

for testing purposes. This creates a different

serial port from the one created by attaching

the XBee. If this different port exists, then

the thread will look for accelerometer data

to come from this port. I can also replay

accelerometer data that I previously recorded

through this serial port using another script.

Periodically the GUI thread requests the

accelerometer data array from the data

collection thread so that it can be displayed.

Having the ability to record and replay

sensor data allows me to develop gesture

recognition algorithms after collecting data

from a student demonstrating their chosen

movement.

The trick to real-time data display is to

plot only the data as fast as is necessary

to get a smooth display. This might sound

obvious, but I have seen example code which

tries to update the screen display at over

100 Hz. However, your screen can probably

only refresh at 30 Hz, so there is no point

in trying to refresh a graphical display any

faster than this. Motion picture movies are

filmed at a standard rate of 24 frames per

second (FPS). So, I collect the accelerometer

data as fast as it is available, then render it

circuitcellar.com 25FEATURES

at a sensible rate. The accelerometer data

scan is decoded using the same struct and

packer statements as were used to pack the

data on the microcontroller. The “unpack”

command is used to retrieve it.

INTER-THREAD COMMUNICATIONI have two threads running within the

same process. The GUI thread, kicked off

when I run main.py, which sets up the

graphical interface and the data collection

thread that collects the accelerometer data

and puts it into an array. How do I get these

two threads to communicate in a thread-

safe manner? There are several ways that

this can be done. My way is to use the

pydispatcher library, as it is a tool designed

for the job with a good level of abstraction

from the underlying complexity of thread

management. If we look at the line of Python

code below, which is in the __init__ of

main.py:

dispatcher.connect(self.dispatcher_receive_acc, signal=ads.SENSOR_SIGNAL_ACC, sender=ads.PYBOARD_SENDER)

This sets up a dispatcher object. When any

other object has a line of code like:

dispatcher.send(signal= signal=ads.SENSOR_SIGNAL_ACC, sender=ads.PYBOARD_SENDER, message=message)

Following this, the message part will be

sent to the dispatcher_receive_acc method

in main.py. I find that using pydispatcher

enables me to write code that is easy to

understand and extend.

DATA VISUALIZATIONPython’s PyQtGraph package is designed for

high-quality data visualization (see Photo 3).

So I decided to leverage this to display real

time sensor data. In common with most GUI

packages it wants to be run in the main thread

of the process. It is not a good idea to try to

make different GUI libraries play together in

the same package as they all expect to hog

the main thread to themselves.

Photo 3 shows the real-time data display

of the accelerometer data. I created a variety

of ways to show the accelerometer data. The

x, y, and z data can be seen along the top

of the illustration as line graphs. Bar graphs

visualize the amplitude and direction of each

of these values as well. I used some simple

geometry to calculate the roll and pitch data,

which are shown on two dials. Finally, a 3-D

vector visualizes the amplitude and direction

of the resultant acceleration vector, along

with a trail showing the last ten values. The

displays can be moved around and resized

as they are created in what are called docks,

using PyQtGraph’s dockarea widget. Docks

can be moved, resized, stacked, and torn out

of the main window. All of this functionality

comes built in to the class.

DISCUSSIONI’ve demonstrated that Python can be used

from the firmware (as Micropython) to the

user interface (using Python 3) to complete

a hardware project. Using the Pyboard just

to collect, filter, and transmit sensor data is

a waste of the board’s power. The ultimate

aim of the project is to put hand gesture

algorithms on to the Pyboard and use this

to control a communications device. The

laptop is being used to display the real-time

data to help test and develop the algorithms

in Python. So long as I develop the code on

the laptop without using Python libraries that

have not been ported to Micropython, I am in

with a chance of implementing the same code

on the Pyboard using Micropython. Naturally,

there are issues of speed and memory size

to contend with when porting code onto the

microcontroller.

A number of Python libraries are being

ported to Micropython, though the tool set is

not yet as rich as exists for Python, which can

only be expected after about only two years of

development. For instance, Micropython now

has a unittest library. This library allows you

to write code to test your code, known as unit

testing.

An active forum on the Micropython

website answers the issues that any first-

time user of a new technology inevitably

stumbles on. Python is not the answer to all

of my or anybody else’s programming issues.

But such a large community has been built

around the language that most of the things

that I am looking at doing can be done with

the language. For instance, there is a wrapper

for the computer vision library openCV called

pyopenCV for a future project I am thinking of

to try and recognize gestures using computer

vision.

Now my credo is: “If you can’t do it in

Micropython, try C or assembly. If you still

can’t do it, then it’s not worth doing.”

ABOUT THE AUTHOR

Matthew Oppenheim (www.mattoppenheim.com) splits his time between

working as a Chief Field Geophysicist for Polarcus AS and an Honorary Re-

searcher at Infolab21 at Lancaster University, UK. He works with the tech-

nologists at Beaumont College, which is a member of the charity Scope, to

develop assistive technology.