Demo: eSense - Open EarablePlatform for Human Sensing

Fahim KawsarNokia Bell Labs and TU DelftCambridge, UKfahim.kawsar@nokia-bell-labs.com

Alessandro MontanariNokia Bell LabsCambridge, UKalessandro.montanari@nokia-bell-labs.com

Chulhong MinNokia Bell LabsCambridge, UKchulhong.min@nokia-bell-labs.com

Utku Günay AcerNokia Bell LabsAntwerp, Belgiumutku_gunay.acer@nokia-bell-labs.com

Akhil MathurNokia Bell Labs and UCLCambridge, UKakhil.mathur@nokia-bell-labs.com

Marc Van den BroeckNokia Bell LabsAntwerp, Belgiummarc.van_den_broeck@nokia-bell-labs.com

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).UbiComp/ISWC’18 Adjunct, October 8–12, 2018, Singapore, SingaporeACM 978-1-4503-5966-5/18/10.https://doi.org/10.1145/3267305.3267640

AbstractWe present eSense - an open and multi-sensory in-earwearable platform to detect and monitor human activities.eSense is a true wireless stereo (TWS) earbud with dual-mode Bluetooth and Bluetooth Low Energy and augmentedwith a 6-axis inertial measurement unit and a microphone.We showcase the eSense platform, its data APIs to capturereal-time multi-modal sensory data in a data explorationtool, and its manifestation in a 360◦ workplace well-beingapplication.

Author KeywordsEarable; Human Sensing; Behaviour Analytics

ACM Classification KeywordsH.5.m [Information interfaces and presentation (e.g., HCI)]:Miscellaneous

IntroductionWearables are finally here. Established forms, e.g., a time-piece, a ring, and a pendant are getting a digital makeoverand are reshaping our everyday experiences with new, use-ful, exciting and sometimes entertaining services. However,to have a broader impact on our lives, the next generationwearables must expand their sensing capabilities beyondthe narrow set of exercise-related physical activities.

381

To this end, we present eSense - an aesthetically pleas-ing, and ergonomically comfortable in-ear high definitionwireless stereo wearable [1]. As illustrated in Figure 1, eS-ense is equipped with a microphone, a 6-axis inertial mea-surement unit, and a dual-mode Bluetooth and BluetoothLow Energy (BLE). Leveraging the combination of micro-phone, accelerometer, gyroscope, and BLE, eSense offersthree sensing modalities - audio, motion, and proximity. It ispowered by a CSR processor and a 45 mAh battery. Mostimportantly, eSense is an entirely open data platform thatallows developers to gather real-time data streams of thesemulti-sensory modalities as well as offering them with sev-eral configurations and reprogramming capabilities.

Figure 1: eSense open wearableplatform with audio, motion, andproximity sensing.

Figure 2: A personalisedwell-being feedback app witheSense capturing physical, mentaland social well-being at workplace.

We have extensively explored the characteristics of audio,inertial, and BLE signals captured by eSense in a varietyof experimental settings. We compared eSense againsta smartphone and a smartwatch considering several keyfactors that impact activity recognition pipelines, includingsampling variability, signal to noise ratio, placement invari-ance, and sensitivity to motion artefacts. Analysis of ourexperimental results suggests that eSense is robust in mod-elling these signals and in most conditions demonstratessuperior performance concerning signal stability and noisesensitivities. Inspired by these characteristics, we have de-signed a set of human activity primitives. Bespoke classi-fiers are then trained to model these activities with audio,motion, and BLE signals, and their combinations. The ex-perimental results demonstrate that eSense can reach upto 88% detection accuracy for the targeted human activi-ties [1, 3]. Taken together these and the rest of our findingsdemonstrate the exciting potential of eSense as an in-earwearable sensing platform for designing individual scalemulti-sensory applications.

Fahim Kawsar, Chulhong Min, Akhil Mathur, Marc Van den Broeck, Utku Günay Acer and Claudio Forlivesi

AI Model

AI Model

MFCC

Statistical Features

BLE RSSI

Accelerometer Gyroscope

BLE

Microphone

• Heart Rate • Emotion and Stress • Eating and Drinking • Conversation • Ambient Environment

• Stationary • Walking • On-Transport • Head Gesture • Placement

• Social Interaction • Proxemic Interaction

AI Model

• Sampling Rate • Duty Cycle

CONTEXT PRIMITIVES CONFIGURATION

• Sampling Rate • Duty Cycle

• Packet Interval

+

+

APP

๏ On-device signal processing and behavioural inferences with complete privacy preservation

๏ Data informed design of purposeful wellbeing metrics grounded on positive computing.

๏ Intimate and actionable feedback manifested in conversational and visual representation.

๏ Online multimodal embedded sensing pipeline that captures, combines, and process motion, audio, Bluetooth Low Energy (BLE) signals to model various contextual factors at an individual scale.

๏ Novel cross-modal fusion framework for multi-task personalised learning

๏ Well defined APIs for ready-to-use context in new application scenarios.

360° Enterprise Well-being App UX

Embedded Behavioural Inference Engine

๏ Instrumented with a microphone, a 6-axis inertial measurement unit and dual model Bluetooth and Bluetooth Low Energy (BLE).

๏ An open platform that allows developers to gather real-time data streams of these different sensory modalities.

๏ Purpose-driven design with specialised transport and configuration with reprogrammable flexibility aimed at behavioural inference.

eSenseEarable Platform for Human Sensing

eSense is an open earable platform purposefully designed for human behaviour modelling.

Open API for Real-Time Data Modelling

Accelerometer, Gyroscope, Audio, BLE RSSI Samples

• Sensor Configuration [BLE] • Data Acquisition [BLE] • HD Music [Bluetooth A2DP]

Sense

Learn

Act

Sensor

Sensor

AI/ML Models

CSR Processor Flash memory

Multi-colour LED

Bluetooth

Push button

40mAh Li-ion battery Contactless changing

Speaker

Motion sensor

Microphone

•Size: 18x18x20mm •Wight: 20g

Figure 3: eSense data exploration tool showing real-time datafrom accelerometer, gyroscope, microphone and BLE.

DemonstrationWe will showcase the eSense hardware and its real-timedata APIs manifested in a dynamic data exploration tool asillustrated in Figure 3. We will invite the conference atten-dees to wear the device and see the real-time multi-modalsignal patterns (accelerometer and gyroscope data, BLERSSI, and microphone data) while experiencing eSense.Besides, we will demonstrate a 360◦ workplace well-beingapplication as depicted in Figure 2. The application modelseSense sensory streams for a variety of physical, digital,and social well-being metrics [2]. Then, it provides person-alised and actionable feedback in conversational and visualrepresentations.

REFERENCES1. Fahim Kawsar, Chulhong Min, Akhil Mathur, and

Alessandro Montanari. 2018. Earables forPersonal-scale Behaviour Analytics. IEEE PervasiveComputing 17, 3 (2018).

2. Afra Mashhadi, Akhil Mathur, Marc Van den Broeck,Geert Vanderhulst, and Fahim Kawsar. 2016.Understanding the impact of personal feedback on

382

face-to-face interactions in the workplace. InProceedings of the 18th ACM International Conferenceon Multimodal Interaction. ACM, 362–369.

3. Chulhong Min, Akhil Mathur, and Fahim Kawsar. 2018.Exploring Audio and Kinetic Sensing on Earable

Devices. In Proceedings of the 2018 Workshop onWearable Systems and Applications (WearSys ’18).ACM.

383