1. 1. NATURAL LANGUAGE INTERACTION WITH THE WEB OF THINGS DeborahA.Dahl ConversationalTechnologies MobileVoice2015 April20-21 SanJose
2. 2. Connected things will be everywhere Everyday life home appliances, entertainment, fitness monitor, toys, pets Industry office, building, industrial robot, parts for manufacturing Public environments retail store, city services, highway, museum Medicine sensors, alarms Transportation -- car, train, bus, airplane
3. 3. Current interaction model Vendor provides an SDK for user devices Vendor provides an API for accessing the functions of their connected things Company and/or Developers create apps for various user devices using provided APIs Users buy connected things and use GUI apps to interact with things Whats wrong with this approach?
4. 4. Problem 1: UIs
5. 5. One Approach to UIs: the basket of remotes
6. 6. Hundreds of apps! Another approach: Apps
7. 7. UI Proliferation user Control/Interface Mobile device Wearable Ambient device (e.g. Amazon Echo/Jibo) Apps Apps Apps
8. 8. Users need a uniform way to interact Dont burden the user with too many interfaces No matter how well-designed a GUI interface is, its different from other GUIs User has to learn it Harder for users like older adults or people with cognitive disabilities UIs can be aggregated into categories (like one UI for the connected home) Natural language provides a uniform UI across connected things
9. 9. Problem 2: Too many APIs Many different APIs for different things Want to hide details of each thing API to simplify development
10. 10. The Answer: Standards! For natural language UIs: W3C EMMA For APIs: The W3C Multimodal Architecture
11. 11. EMMA: a standard format for natural language inputs user Mobile device Wearable Ambient device (e.g. Amazon Echo/Jibo) Natural language Understanding + graphical input EMMA
12. 12. Whats in an EMMA document? Standard metadata about utterance Confidence, input tokens, alternative interpretations, process, timing Interpretation of utterance in a structured format Key-value pairs, for example Groupings with related utterances A standard way to represent natural user inputs
13. 13. What about APIs? W3C Multimodal Architecture is generic API for interacting with encapsulated functionality (Modality Components) Life-cycle events like start, cancel, pause, resume Coordinated by an Interaction Manager Work on dynamic systems is starting
14. 14. MMI generic modality component Echonet Lite KNX Zigbee Mobile device Wearable Ambient device Command mapper MMI Interaction Manager Speech Recognitio n Natural language understandin g User devices Interpretation Modality Components EMMA over MMI EMMA Over MMI MMI Life-cycle events things W3C Standards Other standards Putting things together Car Audio over HTTP or Web Sockets
15. 15. Example: Light bulb Philips Hue Light Bulb LED light that can change colors Controlled through LAN via bridge Bridge communicates to light bulbs via Zigbee protocol Has own API for developers 45 apps on Google Play, 70 on iTunes Store, 3 Windows available
16. 16. Standards-based Control of Hue Light