Embedding Intelligence in Everyday Objects with TJBot
-
Upload
victor-dibia -
Category
Devices & Hardware
-
view
410 -
download
1
Transcript of Embedding Intelligence in Everyday Objects with TJBot
![Page 1: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/1.jpg)
IBM Research
1
IBM Research
Victor Dibia
Embedding Intelligence in Everyday Objects with TJBot.An open source DIY project powered by Watson Cognitive Services.
Human – Agent Collaboration Lab, IBM [email protected]@vykthur | github.com/victordibia
Feb 20, 2017
![Page 2: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/2.jpg)
IBM Research
2
TJBot : What and Why?- Open source DIY project to get you engaged
with Watson Services
![Page 3: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/3.jpg)
IBM Research
3What is TJBot?- A cardboard robot
- Simple, approachable- Open Source (design, code)- Cognitive (IBM Watson services)- Extensible (prototyping platform)
Components: Raspberry Pi, LED, Camera, Microphone, Speaker, Servo.
![Page 4: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/4.jpg)
IBM Research
4
3D Print Laser cut
![Page 5: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/5.jpg)
IBM Research
5ibm.biz/mytjbot
![Page 6: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/6.jpg)
IBM Research
6Recipes.Step by step instructions +Code (node.js) to help you prototype capabilities for TJBot powered by Watson services.
http://www.instructables.com/member/TJBot/
![Page 7: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/7.jpg)
IBM Research
7Project GoalsHow can we make it easier to engage a community of enthusiasts experimenting with embodied cognition – the idea of embedding intelligence in everyday objects within the physical world?
![Page 8: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/8.jpg)
IBM Research
8Project GoalsDesign principle – Approachable Design
- Use of familiar material (cardboard) that can be altered with ease.
- Simplified part assembly: no soldering or adhesive required.
- Simplified programming model and language interface (JavaScript).
![Page 9: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/9.jpg)
IBM Research
9Project OutcomeA prototyping platform to help democratize Embodied Cognition.
Target communities:
- Makers- Developers- Students (Education and Learning)
![Page 10: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/10.jpg)
IBM Research
10How Does Watson Enable TJBot?ListenWatson Speech to Text service converts spoken speech to text that can be analyzed
SpeakWatson Text to Speech service service converts text to sound using various voices.
Understand EmotionsWatson Tone Analyzer service can infer the emotion within text. E.g.. it can tell if a message contains emotions like happy , sad, angry
Understand ConversationsWatson Conversation Service can respond to users in a way that simulates a conversation between humans.
SeeWatson Visual Recognition service can understand the content of an image and describe it.
![Page 11: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/11.jpg)
IBM Research
11TJBotSensors
ExampleCapabilities
ExampleWatson Services
ExampleUse cases
LED
Speakers
Camera Servo Motor Arm
Microphone Listen
Speak
Shine
Show emotion
Wave
See
Speech to text
Tone Analyzer
Vision Recognition
Conversation
Text to speech
Sentiment Analysis
Virtual Agents (eldercare, home care)
Education (language learning)
![Page 12: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/12.jpg)
IBM Research
12
Demo.- Watson Services
- Speech to text- Text to speech- Conversation- Visual Recognition
![Page 13: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/13.jpg)
IBM Research
13
Overview of Watson Services
![Page 14: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/14.jpg)
IBM Research
14
IBM Watson Cognitive
Take your first step into the cognitive era with our variety of smart services.
Services.
- Natural interaction- Semi-structured data processing- Trained and continuously improved via machine learning and deep
learning.- Restful API services with SDKs for node.js, java, python.
![Page 15: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/15.jpg)
IBM Research
15
Language
![Page 16: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/16.jpg)
IBM Research
16
Speech
Vision
Data Insights
![Page 17: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/17.jpg)
IBM Research
17
Speech
Vision
Data Insights
![Page 18: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/18.jpg)
IBM Research
18Speech to TextConverts audio voice into written text.
• Transcription • Voice-controlled applications: allows for custom
models
https://speech-to-text-demo.mybluemix.net/
![Page 19: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/19.jpg)
IBM Research
19Text To SpeechConverts written text into natural sounding audio in a variety of languages and voices. • Customize and control the pronunciation of specific words to
deliver a seamless voice interaction that catered s to your audience.
• Interactive voice based applications.
https://text-to-speech-demo.mybluemix.net/
![Page 20: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/20.jpg)
IBM Research
20Tone AnalyzerUses linguistic analysis to detect three types of tones in written text: emotions, social tendencies, and writing style. • Understand emotional context in conversations or
communications• Taylor interaction based on sentiment.
https://tone-analyzer-demo.mybluemix.net/
![Page 21: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/21.jpg)
IBM Research
21Visual RecognitionUnderstands the contents of images - visual concepts tag the image, find human faces, approximate age and gender, and find similar images in a collection. • Train the service by creating your own custom concepts.
Use Visual Recognition to detect a dress type in retail, identify spoiled fruit in inventory, and more.
https://visual-recognition-demo.mybluemix.net/
![Page 22: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/22.jpg)
IBM Research
22AlchemyLanguage
Analyzes text to help you understand its concepts, entities, keywords, sentiment, and more.
• Additionally, you can create a custom model for some APIs to get specific results that are tailored to your domain.
https://alchemy-language-demo.mybluemix.net/
![Page 23: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/23.jpg)
IBM Research
23ConversationQuickly build, test and deploy a bot or virtual agent across mobile devices, messaging platforms like Slack or even on a physical robot.
• Visual dialog builder to help you create natural conversations between your apps and users, without any coding experience required.
https://conversation-demo.mybluemix.net/
![Page 24: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/24.jpg)
IBM Research
24
Programming TJBot- Getting started - Tying stuff together
![Page 25: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/25.jpg)
IBM Research
25Steps.
- Wifi Setup, - Raspberry Pi Software update - Hardware setup : LED, Servo, Microphone, etc- Credential Setup : Bluemix- Recipe software setup : Clone Github repo - Ready to go.
![Page 26: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/26.jpg)
IBM Research
26
http://www.instructables.com/member/TJBot/Instructions
![Page 27: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/27.jpg)
IBM Research
27Libraries UsedDepends on several npm packages.
- RGB LED – ws281x library- Servo – pigpio software PWM library- Microphone – mic library- Speaker – aplay library- Camera – raspistill wrapper
![Page 28: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/28.jpg)
IBM Research
28Code Walk through: Control LED on TJBot using voice.- http://www.instructables.com/i
d/Use-Your-Voice-to-Control-a-Light-With-Watson/
- Code Walk through
![Page 29: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/29.jpg)
IBM Research
29
TJBot Library [Beta]- Experimental work to encapsulate basic
functions of the bot.- https://github.com/ibmtjbot/tjbotlib
![Page 30: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/30.jpg)
IBM Research
30The TJBot Library
Encapsulate basic functions for TJBot such as listening, speaking, led color change, waving, seeing.
![Page 31: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/31.jpg)
IBM Research
31The TJBot Library
tj.listen(transcript callback)tj.speak(“text”)tj.converse()tj.see()tj.shine(“red”)
![Page 32: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/32.jpg)
IBM Research
32Code Walk through: Control LED using the TJBot library.
- https://github.com/ibmtjbot/recipes
- Code Walk through
![Page 33: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/33.jpg)
IBM Research
33
Open Issues- Improving accuracy- Bot “Interruptibility” - Gracefully managing latency
![Page 34: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/34.jpg)
IBM Research
34Improving AccuracyHow do we improve interaction (voice) accuracy? Improving Speech-to-Text models may not be enough!
- Customized language models?- Intent Matching?- Multi-turn conversations?
![Page 35: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/35.jpg)
IBM Research
35Bot “Interruptibility”When and how should the robot be interrupted (while performing an activity like speaking, waving etc.)?
- Vision? (monitoring a user’s facial expression, raised hand)
- Hardware button or sensor?
![Page 36: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/36.jpg)
IBM Research
36Latency ToleranceLatency can severely degrade quality of interaction. How do we minimize its effect?
- Managing and ordering service responses- Leverage cues to provide additional information- Balancing capabilities – cloud vs local
processing.
![Page 37: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/37.jpg)
IBM Research
37
Next Steps
![Page 38: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/38.jpg)
IBM Research
38Next Steps3 pronged
- Conduct basic research that address open issues.
- Make TJBot simpler and easier to use (tjbotlibrary, visual programming tool)
- Build and sustain the TJBot community.
![Page 39: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/39.jpg)
IBM Research
39Learn more?
- Ibm.biz/mytjbot- http://www.instructables.com/
member/TJBot/
![Page 40: Embedding Intelligence in Everyday Objects with TJBot](https://reader031.fdocuments.us/reader031/viewer/2022030319/58ecd4a61a28ab44478b457b/html5/thumbnails/40.jpg)
IBM Research
40
Thank You!