TouchNet preview at Numenta
Transcript of TouchNet preview at Numenta
TouchNet preview
by Jason Toy
TouchNet
• A touch simulator and dataset of touchable objects to teach AIs how to interact with their environments via motor-sensory neurons
history
• idea around 2015
• started coding September 2017
• initial release 2017 Q4
why TouchNet?
why TouchNet?
• On Intelligence
• future predicting machines
• neuroscience based principles
why TouchNet?
• Helen Keller
• intelligence through fingers
• consciousness “started” from I
Helen Keller• “Every object is associated in my mind with tactual
qualities which, combined in countless ways, give me a sense of power, of beauty.”
• “..with the fingers I see only a very small portion of a surface, and that I must pass my hand continually over it before my touch grasps the whole. It is still more important, however, to remember that my imagination is not tethered to certain points, locations, and distances. It puts all the parts together simultaneously as if it saw or knew instead of feeling them.”
where is AI now?• images, CNNs, LSTMs, GANs
• most business focus on static models; some work in RL
• where is time?
• motor sensory neurons?
• where is the agent compared to the world?
- John Locke, philosophy of mind
“An unknown, but potentially large, fraction of animal and human intelligence is a direct
consequence of the perceptual and physical richness of our environment, and is unlikely to
arise without it.”
TouchNet consists of 4 parts
• dataset of 3D objects to touch and interact with
• an evolving simulation
• a simple extendable API
• benchmarks to measure and track progress
Dataset
Dataset
• procedurally generated 3D objects
• common objects: cups, plates, bowls, etc
• starting with tool objects
• cifar100 …. imagenet
Simulator
Simulators and physics
• need a real physics engine
• needs to be fully programmable & modifiable
• tested all of them: MuJoCo, DeepMind Lab, Roboschool, project Malmo, VizDoom,Gazebo, bullet , Processing
Simulators and physics• bullet and pybullet are a secret weapon!
• robust physics engine used for many applications
• super easy to install
• movies: Shrek, Hancock,Sherlock Holmes….
• games: Toy Story 3, Grand Theft Auto 4,Hot Wheels…
pybullet• rigid and soft body simulations
• collision detection
• works with unity, blender,CUDA,OpenCL,…..
• developed by Erwin Coumans & community since ~2003
• now at Google and used for RL & Robotics
• http://pybullet.org
What we want to simulate
What we have
simulation demo
coevolving simulator & agent
• index finger with merkel touch sensors
• all sensor types in finger
• all fingers
• full arm
• full body?
API
• simple to get started and use
• easy to install
• modifiable and customizable
reinforcement learning api
• based off OpenAI’s gym and universe API
• can plug in almost any code for reinforcement learning
OpenAI’s Universe API
Same API
Benchmarks
“Blind” objection classification
• goal: classify the object
• CNN and actor critic DNN models trained concurrently
• pytorch and tensorflow versions
• 4% accuracy, please beat mine!
“blind” classification code walk through
Benchmarks
• HTM version?
• more benchmarks in the future e.g:
• moving objects, differentiating textures,…..
Help!
• very ambitious project, too big for me
• 100% open source
• I do a little of everything ( suck at everything? )
• looking for contributors and experts in physics,math, neuroscience, ml
Jason Toy [email protected]
twitter.com/jtoy
Thank you!
Questions and Feedback
http://github.com/jtoy/touchnet