Research Questions:

1: How can we visually detectdevices of interest in real timefrom an ego-centric perspec-tive?

2: How can we detect fine-grained gestural input fromwearable sensors?

3: How can we reason whichdevice command should becarried out?

Augmented Humans Interacting withan Augmented World

Vincent BeckerDepartment of Computer Science,ETH Zurich, Switzerlandvincent.becker@inf.ethz.ch

Visual object recognitionSmart

glasses

Decisionsystem

Gesture recognition

EMG armband

Smartwatch

1. Recognize the device for interaction

2. Recognize the user’s input

3. Infer the device action to be carried out

Figure 1: Recognizing the user’s interaction intention withwearable computers.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).MobileHCI ’18 Adjunct, September 3–6, 2018, Barcelona, SpainACM 978-1-4503-5941-2/18/09.http://dx.doi.org/10.1145/3236112.3236179

AbstractOur research explores a seamless interaction with smartdevices, which we divide into three stages: (i) device recog-nition, (ii) user input recognition, and (iii) inferring the appro-priate action to be carried out on the device (cf. Figure 1).We leverage wearable computers and combine them intoone interaction system. This makes interactions ubiquitousin two ways: While smart devices are becoming increas-ingly ubiquitous, our wearable system will be ubiquitouslyavailable to the user, making it possible to interact every-where, any time.

Author KeywordsHCI; Wearable Computing; Smart Devices

ACM Classification KeywordsH.5.2 [Information interfaces and presentation (e.g., HCI)]:User Interfaces

IntroductionMiniature computers have become an essential part of ourlives. They can sense, process, communicate, and act, andare embedded into everyday devices and tools, and caneven be worn on the human body. Augmented devices offermore convenient and intelligent services, computers aug-menting humans enable instant information access. Yet,interactions between these two sides, the augmented hu-

man within an augmented world, are still very different fromhuman-human and from device-device interactions. Ourresearch aims at bridging this gap by employing machinelearning on wearable devices. This has the benefit of tak-ing an ego-centric perspective, naturally perceiving whatthe human perceives: cameras can see what the humansees, body sensors can recognize gestures. Moreover, thecontrolled devices do not require built-in sensing and intel-ligence, as the knowledge is acquired by the ego-centricsystem. Hence, the device can become as simple as possi-ble, outsourcing its interface to a wearable system [3].

Figure 2: Visual object recognitionrunning on a smartphone.

Figure 3: Finger identification andforce estimation using a wearableEMG armband.

Research Goals and Expected ResultsWe explore techniques to support the interaction processwith smart devices. We structure our work around the differ-ent interaction stages as shown in Figure 1 and reflected bythe research questions (c.f. sidebar). We plan to apply andadvance visual object recognition algorithms based on deeplearning in order to recognize smart devices from wearablecameras such as those incorporated in smart glasses. Onemain goal is to enable a fast recognition, even on resource-constrained devices. We have created a prototype for asmartphone camera [1], which we will further develop andport to smart glasses. An addition we consider, is to com-bine the object recognition with a wearable eye tracker inorder to more precisely know what the user is interested in.

We further investigate novel wearable gesture recognitionsystems. We are specifically interested in electromyogra-phy (EMG) of the forearm, which enables the recognitionof fine-grained hand gestures. We created a system to rec-ognize the user’s finger in touches and estimate the ap-plied force [2] (cf. Figure 3). We are further developing afull-hand gesture recognition system based on EMG andare also investigating other modalities and devices such assmartwatches.

After having built the visual object recognition and gesturerecognition subsystem, we plan to combine them to en-able the entire interaction pipeline. Our current prototype [1]demonstrates this concept as it combines device recogni-tion and gestural input. A crucial question is whether wecan build a decision system, which interprets the performedgesture and maps it to a command for the currently con-trolled device without hard-coding such a mapping.

Finally, we are examining the potential of visually aug-mented reality (AR) for interaction with smart devices. Itcomplements our efforts of recognizing human input by thepossibility of visualizing device output. AR can show infor-mation about the capabilities of a device or its state on thespot, which otherwise would not be perceivable.

Making the interaction of humans with smart devices moresimilar to natural communication could make the devicesmore usable and easily accessible to more people. It wouldgive humans immediate access to information about andcontrol over the devices, at any time with little mental andphysical effort.

REFERENCES1. V. Becker, M. Bâce, and G. Sörös. Wearable machine

learning for recognizing and controlling smart devices.In Proc. MobileHCI ’17 Workshop.

2. V. Becker, P. Oldrati, L. Barrios, and G. Sörös. Poster:TouchSense: Classifying and Measuring the Force ofFinger Touches with an Electromyography Armband. InProc. AH ’18.

3. S. Mayer and G. Sörös. User Interface Beaming -Seamless Interaction with Smart Things using PersonalWearable Computers. In BSN ’14 Workshop.