Knowledge Processing for Autonomous Robots · 2012. 6. 13. · RoboEarth: concept, interface to...
Transcript of Knowledge Processing for Autonomous Robots · 2012. 6. 13. · RoboEarth: concept, interface to...
Knowledge Processing for AutonomousRobots
Moritz TenorthIntelligent Autonomous Systems GroupTechnische Universitat Munchen
June 13, 2012
Knowing where to look for something
◮ Naive:Fixed controller for opening everything with a handle
Knowing where to look for something
◮ Naive:Fixed controller for opening everything with a handle
◮ Parameterizable:Look up the positions of all containers in the map
Knowing where to look for something
◮ Naive:Fixed controller for opening everything with a handle
◮ Parameterizable:Look up the positions of all containers in the map
◮ Knowledge-based:Infer the right container based on the types of the object and thecontainer
Knowing where to look for something[Humanoids 2011]
storagePlaceFor
instanceOf
instanceOf
Knowledge-based decision making
◮ Formulate control decisions as inference tasks
◮ Separate control flow from execution context
◮ Increase re-usability of robot plans
CRAM: Cognitive Robot Abstract Machine
Outline
◮ Wednesday: Knowledge representation (KnowRob, ProbCog)
◮ Thursday morning: CRAM execution framework
◮ Thursday afternoon: Semantic perception methods
◮ Friday: Group discussions about specific project ideas
Outline: KnowRob tutorial
◮ Introduction: What can CRAM, KnowRob, etc be used for?
◮ Languages: Prolog, OWL, Java, etc
◮ Basics: launching KnowRob, loading files, querying, debugging
◮ Objects: class taxonomy, instances, components, environmentmodels, units and coordinates, qualitative spatial relations,computable properties
◮ Actions: class-level descriptions, properties, projection, planning,action dependencies
◮ RoboEarth: concept, interface to KnowRob, up- and download ofinformation, capability matching
◮ ProbCog: interface to statistical relation inference (Dominik)
What I won’t talk much about...
◮ Time: Reasoning about time points and intervals, object posesthat change over time
◮ Changing objects: Creation/destruction/modification of objectsthrough actions and processes
◮ Interfaces to external components: Perceptual methods, SQLdatabases, Web information
◮ Human activity data: segmentation/classification/abstraction/reasoning about observations of human activities
◮ Machine learning: clustering and classification using Weka andMallet libraries
KnowRob components
The KnowRob knowledge processing system
KnowRob: A practical robot knowledge base[IROS 2009, RAS 2010]
Representations for robot knowledge
Semantic object and environment models[Humanoids 2010, IROS 2011, RAS 2012]
◮ Recognized objects andenvironment map represented inthe knowledge base
◮ Pieces of furniture as objectinstances that inherit propertiesof their types
◮ Articulation models for openingcontainers
◮ Spatio-temporal representation ofobject poses
Image courtesy of Ulrich Klank
Actions and processes◮ Actions move, split, destroy, create, join,
switch on, open and close objects
◮ Processes are started as indirect effectsof actions
◮ Integrated representation to jointlyreason about actions and processes
◮ Prediction of their effects usingqualitative high-level projection rules
Composing actions to plans[ICRA 2012]
Web-enabled knowledge bases
Task instructions from the WWW[ICRA 2010]
semantic
parsing
WordNet
lexical
database
OpenCyc
ontology
word
sense
resolution
ontology
mapping
robot
plan
generation
S
VP
VB NP
IN NP
DT NN
DT NN
PP
place/PuttingSthSomewherecup/DrinkingMugtable/Table-PieceOfFurniture
place/ cup/ table/
N08664443,V01494310,V02392762,...N03147509,N13619168,...N08266235,N04379243
Object models
◮ Automatically created ontologyof >7500 objects from theonline shop germandeli.com
◮ Class hierarchy from categories+ perishability, weight, price,origin, ...
◮ SIFT recognition models fromproduct pictures (work byDejan Pangercic)
[RAM 2011a]
Understanding observations of human actions
Acquiring knowledge by observing humans[T&D 2010]
Image courtesy of Jan Bandouch
◮ Gather knowledge from observing human activities
◮ Interpret the data to obtain reusable information
Segmentation and abstraction of motions[ICCV THEMIS 2009]
◮ Conditional Random Fields-based motion segmentation
◮ Pose-related features and data from external sensors
Extracting knowledge by observing humans[ICRA 2011]
Integrated experiments
Completing underspecified instructions[Humanoids 2011]
Comparing observations with WWW instructions
Inferring missing objects[IROS 2010]
◮ Given: incomplete table setup◮ Objective: infer which objects are additionally needed◮ Integrates perception with statistical relational models of human
preferences(see also comp missingobj, prolog perception, mod probcog)
Outline: KnowRob tutorial
◮ Introduction: What can CRAM, KnowRob, etc be used for?
◮ Languages: Prolog, OWL, Java, etc
◮ Basics: launching KnowRob, loading files, querying, debugging
◮ Objects: class taxonomy, instances, components, environmentmodels, units and coordinates, qualitative spatial relations,computable properties
◮ Actions: class-level descriptions, properties, projection, planning,action dependencies
◮ RoboEarth: concept, interface to KnowRob, up- and download ofinformation, capability matching
◮ ProbCog: interface to statistical relation inference (Dominik)
Why Prolog for autonomous robots?
◮ Can be read declaratively
◮ Can be programmed (prodecurally)
◮ There are textbooks and tutorials
◮ Industrial-strength open-source implementations
◮ Foreign language interfaces
◮ Result interpretation: similar to a DB query language
KnowRob: based on SWI Prolog (http://www.swi-prolog.org)
Which language for what purpose?
OWL:
◮ Class taxonomy of objects, actions, events,...
◮ Instances of these classes (e.g. environment models, experiences)
◮ Robot capabilities/action requirements
Prolog:
◮ Internal representation (OWL parsed into Prolog triples)
◮ Query language
◮ Inference predicates (DL inference, computables,...)
Java:
◮ External interfaces (WWW, ROS,...)
◮ Library integration (ProbCog, Weka,...)
Prolog syntax
◮ Predicate names and constants start with a lowercase letter or areinside single quotes
◮ Variables start with an uppercase letter
◮ Formulas end with a full stop ’.’
◮ Stepping through all results of a query with ’;’
◮ Logical operators:
Operator Prolog equivalent Example∧ , type(A, ’Cup’), on(A, T).∨ ; type(A, ’Food’); type(A, ’Drink’).
Example
dairyproduct(milk).
perishable(Stuff) :- storedIn(Stuff, Ref), refrigerator(Ref).
Introduction to Prolog
milk.pl
cup(cup0).
dairyProduct(milk1).
meatProduct(ham2).
cupboard(cupboard3).
refrigerator(fridge4).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
storagePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).
storagePlaceFor(Loc, Item) :- cupboard(Loc), cup(Item).
searchForIn(Item,Loc) :- storagePlaceFor(Loc, Item).
Example queries
Two main types of queries:
◮ Query with unbound variables → variable bindings
◮ All variables bound → true/false result
Example queries
$ prolog -f milk.pl
?- refrigerator(A).
A = fridge4.
?- refrigerator(fridge4).
true.
?- searchForIn(milk1, A).
A = fridge4 .
Example queries
Example queries
?- storagePlaceFor(A, milk1).
A = fridge4 .
?- storagePlaceFor(fridge4, A).
A = milk1 ;
A = ham2 ;
?- storagePlaceFor(A, B).
A = fridge4,
B = milk1 ;
A = fridge4,
B = ham2 ;
A = cupboard3,
B = cup0.
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
milk.pl
searchForIn(Item, Loc) :- storagePlaceFor(Loc, Item).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
milk.pl
storagePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).
storagePlaceFor(Loc, Item) :- cupboard(Loc), cup(Item).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
milk.pl
refrigerator(fridge4).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
milk.pl
refrigerator(fridge4).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
milk.pl
dairyProduct(milk1).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
milk.pl
dairyProduct(milk1).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
milk.pl
dairyProduct(milk1).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
. . Exit: (9) perishable(milk1) ? creep
milk.pl
dairyProduct(milk1).
perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
. . Exit: (9) perishable(milk1) ? creep
. Exit: (8) storagePlaceFor(fridge4, milk1) ? creep
milk.pl
storagePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).
storagePlaceFor(Loc, Item) :- cupboard(Loc), cup(Item).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
. . Exit: (9) perishable(milk1) ? creep
. Exit: (8) storagePlaceFor(fridge4, milk1) ? creep
Exit: (7) searchForIn(milk1, fridge4) ? creep
milk.pl
searchForIn(Item, Loc) :- storagePlaceFor(Loc, Item).
Under the hood: the tracer
Tracing a query
?- trace, searchForIn(milk1, P).
Call: (7) searchForIn(milk1, G369) ? creep
. Call: (8) storagePlaceFor( G369, milk1) ? creep
. . Call: (9) refrigerator( G369) ? creep
. . Exit: (9) refrigerator(fridge4) ? creep
. . Call: (9) perishable(milk1) ? creep
. . . Call: (10) dairyProduct(milk1) ? creep
. . . Exit: (10) dairyProduct(milk1) ? creep
. . Exit: (9) perishable(milk1) ? creep
. Exit: (8) storagePlaceFor(fridge4, milk1) ? creep
Exit: (7) searchForIn(milk1, fridge4) ? creep
P = fridge4 .
milk.pl
searchForIn(Item, Loc) :- storagePlaceFor(Loc, Item).
Why is this useful for robotics?
◮ Separation of the inference task (what to infer) from theimplementation (how to infer it) and the data (what evidence toinfer it from)
◮ Search-based inference: easy integration of additional inferencemethods or knowledge sources as additional branches in the searchtree
Outline: KnowRob tutorial
◮ Introduction: What can CRAM, KnowRob, etc be used for?
◮ Languages: Prolog, OWL, Java, etc
◮ Basics: launching KnowRob, loading files, querying,debugging
◮ Objects: class taxonomy, instances, components, environmentmodels, units and coordinates, qualitative spatial relations,computable properties
◮ Actions: class-level descriptions, properties, projection, planning,action dependencies
◮ RoboEarth: concept, interface to KnowRob, up- and download ofinformation, capability matching
◮ ProbCog: interface to statistical relation inference (Dominik)
Launching KnowRob
◮ KnowRob is very modular: Dependencies are automatically loadedby rosprolog
◮ General syntax: rosrun rosprolog rosprolog <pkg>
◮ starts a Prolog environment◮ checks for dependencies of <pkg>◮ locates them on the disk◮ loads the prolog/init.pl file for each of these packages
◮ For today we use:rosrun rosprolog rosprolog knowrob tutorial
TBOX queries
◮ The TBOX contains terminological statements: abstractdescriptions of classes and properties
◮ owl subclass of(Sub, Super). reads all classes derived from Super
◮ rdfs subproperty of(SubProp, Prop). lists sub-properties of Prop
◮ owl individual of(Restr, owl:’Restriction’), owl subclass of(Class,
Restr). reads all restrictions that are defined on Class
◮ owl satisfies restriction(Res, Restr). verifies whether Res satisfiesthe restriction Restr
◮ owl description(RestrOWL, RestrProlog). translates the OWL identifierof a restriction into a Prolog term
(see also http://www.swi-prolog.org/pldoc/package/semweb.html)
Example queries: TBOX
Example
$ rosrun rosprolog rosprolog knowrob tutorial
?- owl subclass of(A, knowrob:’FoodOrDrink’).
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#FoodOrDrink’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drink’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Coffee-Beverage’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#InfusionDrink’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Tea-Beverage’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Tea-Iced’
ABOX queries
◮ The ABOX contains assertional statements: concrete instances ofthe abstract classes and relations between them
◮ Query predicates:◮ rdf(S, P, O) returns only exactly matching triples◮ rdf has(S, P, O) also takes the subPropertyOf relation into account,
returning matches for all specializations of P◮ rdf reachable(S, P, O) further considers transitivity of P◮ rdf triple(P, S, O) additionally includes computables◮ owl has(S, P, O) includes OWL inference results (e.g. inferred class
membership of an individual)
Important: Most predicates can be used with different combinations offree and bound variables:
◮ owl individual of(Ind, knowrob:’Cup’) reads all instances of class Cup
◮ owl individual of(knowrob:’cup0’, Class) reads all classes of cup0
Example queries: ABOX
Example
?- owl has(A, rdf:type, knowrob:’Drawer’).
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer1’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer103’ ;
?- owl has(knowrob:’Drawer1’, P, O).
P = ’http://www.w3.org/1999/02/22-rdf-syntax-ns#type’,
O = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer’ ;
P = ’http://ias.cs.tum.edu/kb/knowrob.owl#widthOfObject’,
O = literal(type(xsd:float, ’0.58045006’)) ;
P = ’http://ias.cs.tum.edu/kb/knowrob.owl#properPhysicalParts’,
O = ’http://ias.cs.tum.edu/kb/knowrob.owl#Door4’ ;
Querying KnowRob from your program
◮ Tell/ask interface via a ROS service◮ tell: add statements◮ ask: send a query
◮ Send queries as string like you would do from a console
◮ Result serialized using json
◮ Client libraries for C++, Python, Java, Lisp available
Example json prolog launch file
<launch>
<node name="knowrob" pkg="rosprolog" type="run with prolog env"
args="demo cotesys fall2010 $(find json prolog)/bin/json prolog" />
</launch>
(see also http://ros.org/wiki/json prolog)
Querying KnowRob from Python
knowrob/json prolog/examples/test json prolog.py
#/usr/bin/env python
import roslib; roslib.load manifest(’json prolog’)
import rospy
import json prolog
if name == ’ main ’:
rospy.init node(’test json prolog’)
prolog = json prolog.Prolog()
query = prolog.query("member(A, [1, 2, 3, 4]), B = [’x’, A]")
for sol in query.solutions():
print ’Found solution. A = %s, B = %s’ % (sol[’A’], sol[’B’])
query.finish()
Querying KnowRob from C++
knowrob/json prolog/examples/test json prolog.cpp
#include <string>
#include <iostream>
#include <ros/ros.h>
#include <json prolog/prolog.h>
using namespace std;
using namespace json prolog;
int main(int argc, char *argv[]) {
ros::init(argc, argv, "test json prolog");
Prolog pl;
PrologQueryProxy bdgs = pl.query("member(A, [1, 2, 3]), B = [’x’, A]");
for(PrologQueryProxy::iterator it=bdgs.begin(); it != bdgs.end(); it++) {
PrologBindings bdg = *it;
cout << "Found solution: " << (bool)(it == bdgs.end()) << endl;
cout << "A = "<< bdg["A"] << endl;
cout << "B = " << bdg["B"] << endl;
}return 0;
}
Workspace setup and debugging
◮ Global history: put the following .plrc file in your home
.plrc
rl write history :-
expand file name("∼/.pl-history", [File| ]),
rl write history(File).
:- ( current prolog flag(readline, true) ->
expand file name("∼/.pl-history", [File| ]),
(exists file(File) -> rl read history(File); true),
at halt(rl write history) ; true ).
◮ Debugging: graphical tracer available in SWI Prolog
Graphical tracer
?- guitracer.
?- trace.
(see also http://www.swi-prolog.org/pldoc/refman, Section 3.5)
Exercises (45min):
KnowRob ontology walkthrough: Overview of main classes andproperties in http://ias.in.tum.de/kb/knowrob.owl
Structure of a KnowRob ROS package
KnowRob basics tutorial:http://ias.in.tum.de/kb/wiki/index.php/Tutorial: KnowRob basics
Loading OWL files and ROS modules:http://ias.in.tum.de/kb/wiki/index.php/Loading files and ROS packages
Compile and run a json prolog example client in your favorite language(launch the KnowRob service with roslaunch knowrob tutorialtutorial.launch)
Further links
API documentation of KnowRob packages:http://ias.cs.tum.edu/kb/api/<pkg-name>/
Integration of common DL reasoners (e.g. for inferring class subsumption)http://ias.in.tum.de/kb/wiki/index.php/OWL reasoners via OWLAPIhttp://owlapi.sourceforge.net/documentation.html
PLDOC source code documentation:http://ias.in.tum.de/kb/wiki/index.php/Source code documentation with PLDOC
Prolog documentation (incl. semweb library and thea OWL library)http://www.swi-prolog.org/pldoc/refman/http://www.swi-prolog.org/pldoc/package/semweb.htmlhttp://www.semanticweb.gr/thea/index.html
Matlab interface (possibly outdated)http://ias.in.tum.de/kb/wiki/index.php/Using MATLAB from Prolog
Outline: KnowRob tutorial
◮ Introduction: What can CRAM, KnowRob, etc be used for?
◮ Languages: Prolog, OWL, Java, etc
◮ Basics: launching KnowRob, loading files, querying, debugging
◮ Objects: class taxonomy, instances, components,environment models, units and coordinates, qualitativespatial relations, computable properties
◮ Actions: class-level descriptions, properties, projection, planning,action dependencies
◮ RoboEarth: concept, interface to KnowRob, up- and download ofinformation, capability matching
◮ ProbCog: interface to statistical relation inference (Dominik)
Object representation
Objectives:
◮ Class-level knowledge about object types
◮ Instance-level representations of object configurations
◮ Metric as well as qualitative spatial information (+ conversion)
◮ Explicit representation of units and coordinates (+ conversion)
◮ Nominal object locations in an environment
◮ Changes of locations over time
(see also http://ias.in.tum.de/kb/wiki/index.php/Reasoning about objects)
Visualizing objects
◮ A java-based visualization canvas can display objects and theirlocations
◮ visualization canvas(C). starts the canvas and stores a handle in thevariable C
◮ $C refers to the last binding of the variable C
◮ add object(Identifier, $C). sends an object instance to the canvas inorder to visualize it
(see also API documentation: http://ias.cs.tum.edu/kb/api/mod vis/)
Example queries: Visualization
Example
?- owl has(A, rdf:type, knowrob:’Drawer’),
add object with children(A, $C).
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer1’ ;
A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer103’ ;
Semantic map representation
Abstract knowledgeabout object classes
Object instances and componenthierarchy
Poses in the environmentand their changes over time
Related: TBOX/SBOX, Galindo et al (RAS 2008)
Semantic map representation
Abstract knowledgeabout object classes
Object instances and componenthierarchy
Poses in the environment
and their changes over time
Spatio-temporal object representation
(see also
http://ias.in.tum.de/kb/wiki/index.php/Modeling perceived or inferred objects)
Integrating perception
(see also
http://ias.in.tum.de/kb/wiki/index.php/Modeling perceived or inferred objects)
Semantic map representation
Abstract knowledgeabout object classes
Object instances andcomponent hierarchy
Poses in the environment
and their changes over time
Representing articulated objects
◮ Poses and joint limits of hinges and prismatic joints
◮ Enable robot to open doors and drawers
Object part composition
◮ Part-of hierarchy: composition of objects from parts
◮ Hinged-to/fixed-to: kinematic properties
Integrating CAD models
◮ Visualization and computation of qualitative spatial relations
◮ Soon: segmentation and interpretation of object parts(see also http://ias.in.tum.de/kb/wiki/index.php/CAD Models)
Computing qualitative spatial relations
(see also http://ias.in.tum.de/kb/wiki/index.php/Tutorial: Computables)
Computing qualitative spatial relations
(see also http://ias.in.tum.de/kb/wiki/index.php/Tutorial: Computables)
Computables
◮ Procedural attachment to OWL classes and properties◮ Computable properties:
◮ Calculate if relation exists between two instances◮ Generate instances for which relation exists
◮ Computable classes:◮ Check for class subsumption◮ Generate individuals of a class
Defining computables
◮ Define OWL property (as usual incl. Domain and Range)Ob je c tP rope r t y : a f t e r
SubPropertyOf : t empo r a l l yR e l a t e dDomain : TimePointRange : TimePoint
◮ Write Prolog predicate that computes this relationcomp a f t e r ( Pre , A f t e r ) : -
ow l ha s ( Pre , type , ’ TimePoint ’ ) ,ow l ha s ( A f t e r , type , ’ TimePoint ’ ) ,t e rm to atom (P , Pre ) ,te rm to atom (A, A f t e r ) ,P<A.
◮ Attach this predicate to the OWL propertyI n d i v i d u a l : computeAfte r
Types : P r o l o gP rope r t yFac t s : t a r g e t a f t e r
command comp a f t e r(see also http://ias.in.tum.de/kb/wiki/index.php/Define computables)
Calling computables
◮ Computables can trigger complex calculations, include them only ifneeded
◮ Special predicate rdf triple(?P,?S,?O) that returns the union of◮ asserted knowledge (via owl has(?S,?P,?O))◮ results of computables (rdfs computable triple(?S,?P,?O))◮ provides a hook for custom predicates.
r d f t r i p l e ( Prop , Subj , Obj ) : -s u b p r o p e r t y o f ( SubProp , Prop ) ,( ow l ha s ( Subj , SubProp , Obj ) ;
r d f s c om p u t a b l e t r i p l e ( Subj , SubProp , Obj ) ;r d f t r i p l e h o o k ( Subj , SubProp , Obj ) ) .
(see also http://ias.in.tum.de/kb/wiki/index.php/Tutorial: Computables)
Semantic map representation
Abstract knowledgeabout object classes
Object instances and componenthierarchy
Poses in the environment
and their changes over time
Video: Unpacking a shopping basket
Thanks to Dejan Pangercic, Mathe Koppany, Zoltan-Csaba Marton, Lucian Goron,Monica Opris and Thomas Ruhr for making this live demonstration possible.
Infer storage location based on generic class
knowledge
storagePlaceFor
instanceOf
instanceOf
Object ontology
◮ Automatically created ontologyof >7500 objects from theonline shop germandeli.com
◮ Class hierarchy from categories+ perishability, weight, price,origin, ...
◮ SIFT recognition models fromproduct pictures (work byDejan Pangercic)
[RAM 2011a]
(see also http://www.ros.org/wiki/comp germandeli)
Infer storage locations based on semantic object
similarity
SpatialThing
DrinkingVessel
Cup
Thing
Glass
CoffeeCup SodaGlass
Root
LCS
C1 C2
depth(LCS)
= 2
depth(C1)
= 4
depth(C2)
= 4
Learning Organizational Principles in Human Environments.
Martin Schuster, Dominik Jain, Moritz Tenorth and Michael Beetz. ICRA 2012
(see also http://www.ros.org/wiki/comp orgprinciples)
Infer storage locations based on semantic object
similarity
?- highlight best location dtree(
orgprinciples:’CoffeeFilter1’, Canvas).
Best location: knowrob:Drawer7
Objects at location knowrob:Drawer7:
WUP similarity: object (class)
0.87500: orgprinciples:CoffeGround1
(germandeli:Dallmayr Classic Ground Coffee 250g)
0.75000: orgprinciples:EspressoBeans1
(germandeli:illy Espresso Whole Beans 88 oz)
0.70588: orgprinciples:Sugar1
(germandeli:Nordzucker Brauner Teezucker 500g)
0.66667: orgprinciples:Tea2
(germandeli:Teekanne Rotbusch Tee Vanille 20 Bags)
Open the drawer where cups are stored
◮ Infer most likely storage location
◮ Read articulation model from semantic map
Check if objects are placed correctly[ICRA SPME 2011]
◮ Infer most likely storage location
◮ Compare with actual locations of objects
Select objects based on their purpose
◮ Combine abstract object knowledge with map information
◮ Here: “appliance that can be used for washing dishes”
Integrate with human tracking data
◮ Observations of humans are represented in the KB
◮ Reason about their interaction with objects in the map
Scaling up to large environmentsSearching Objects in Large-Scale Indoor Environments: A Decision-Theoretic Approach.
Lars Kunze, Michael Beetz, Manabu Saito, Haseru Azuma, Kei Okada, Masayuki Inaba. ICRA 2012
Exercises (45min):
Walkthrough: object pose representation
KnowRob objects tutorial:http://ias.in.tum.de/kb/wiki/index.php/Reasoning about objects
Additional information:http://ias.in.tum.de/kb/wiki/index.php/Measurement unitshttp://ias.in.tum.de/kb/wiki/index.php/Coordinate systemshttp://ias.in.tum.de/kb/wiki/index.php/Map data representationhttp://ias.in.tum.de/kb/wiki/index.php/CAD Modelshttp://ias.in.tum.de/kb/wiki/index.php/Mod semantic map
Outline: KnowRob tutorial
◮ Introduction: What can CRAM, KnowRob, etc be used for?
◮ Languages: Prolog, OWL, Java, etc
◮ Basics: launching KnowRob, loading files, querying, debugging
◮ Objects: class taxonomy, instances, components, environmentmodels, units and coordinates, qualitative spatial relations,computable properties
◮ Actions: class-level descriptions, properties, projection,planning, action dependencies
◮ RoboEarth: concept, interface to KnowRob, up- and download ofinformation, capability matching
◮ ProbCog: interface to statistical relation inference (Dominik)
Action representation
Objectives:
◮ Different hierarchies: temporal composition, class taxonomy
◮ Action properties: toLocation, objectActedOn, bodyPartUsed,...
◮ Partial ordering of actions in a task
◮ Reason about effects of actions on objects
◮ Support projection and (simple) planning
◮ Generate executable robot plan from specification
◮ Specify requirements on robot capabilities or components
(see also http://ias.in.tum.de/kb/wiki/index.php/Reasoning about actions)
Reasoning about action sequences
◮ Generate initial task description from Web instructions
Reasoning about action sequences
◮ Project action effects to infer interaction with objects
◮ Specific properties exactly describe object-action relation
◮ Reason about changes induced on objects
Reasoning about action sequences
◮ Complete action sequences using background knowledge
◮ Infer storage location, opening trajectory, etc
Reasoning about action sequences
◮ Detect knowledge gaps by comparing projected world state withrequired action inputs
◮ Apply planning to add required actions (see also knowrob actions package)
Reasoning about action sequences
◮ Include physics-based reasoning to infer action parameterizations
◮ Work by Lars Kunze, Mihai Dolha
Action class representation
◮ Generic action class definition, incl. super-classes and sub-actions
C l a s s : Putt ingSomethingSomewhereSubClassOf :
Movement - T r a n s l a t i o nEv en tT r an spo r t a t i o nEven tsubAct i on some Pick ingUpAnObjectsubAct i on some Car ry ingWhi l eLocomot ingsubAct i on some PuttingDownAnObjecto r d e r i n gC o n s t r a i n t s v a l u e SubEventOrder ing1o r d e r i n gC o n s t r a i n t s v a l u e SubEventOrder ing2
◮ Specification of sub-actions and ordering constraints
I n d i v i d u a l : SubEventOrder ing1Types :
P a r t i a lO r d e r i n g - S t r i c tFac t s :
o c c u r sB e f o r e I nO r d e r i n g Pick ingUpAnObjecto c c u r sA f t e r I nO r d e r i n g Car ry ingWhi l eLocomot ing
Composing actions to tasks
C l a s s : SetATableAnnota t i on s : l a b e l ” s e t a t a b l e ”SubClassOf : Ac t i onEqu i va l en tTo :
subAct i on some PutP laceMat InFrontOfCha i rsubAct i on some PutP la te InCente rOfP laceMatsubAct i on some PutKn i f eR i gh tO fP l a t esubAct i on some [ . . . ]o r d e r i n gC o n s t r a i n t s v a l u e [ . . . ]
C l a s s : PutP laceMat InFrontOfCha i rEqu i va l en tTo :
Putt ingSomethingSomewhereob jectActedOn va l u e PlaceMat1t oLoca t i o n some Place1
C l a s s : P lace1Equ i va l en tTo :
i nFrontOf - G e n e r a l l y some Chai r - P i e c eO fFu r n i t u r e
I n d i v i d u a l : PlaceMat1Types : PlaceMat
Task specification
CRAM plan generation
( d e f - t o p - l e v e l - p l a n s e r v e - a - d r i n k ( )( w i t h - d e s i g n a t o r s (
( b o t t l e 1 ( ob j e c t ‘ ( ( name b o t t l e 1 )( type d r i n k i n g - b o t t l e ) ) ) )
( bed1 ( ob j e c t ‘ ( ( name bed1 )( type b e d p i e c e - o f - f u r n i t u r e ) ) ) )
( hand -pose -handove r1 ( l o c a t i o n ‘ ( ( on , bed1 ) ) ) )( r o bo t - p o s e - h andov e r 1 ( l o c a t i o n ‘ ( ( to r each )
( s i d e : r i g h t )( l o c , hand -pose -handove r1 ) ) ) )
( a rms-a t101 ( a c t i o n ‘ ( ( t ype t r a j e c t o r y )( pose , hand -pose -handove r1 )( s i d e : r i g h t ) ) ) )
( unhand -ac t i on102 ( a c t i o n ‘ ( ( t ype o p e n - g r i p p e r )( s i d e : r i g h t ) ) ) )
)
( a ch i e v e ‘ ( o b j e c t - i n - h a n d , b o t t l e 1 : r i g h t ) )( a t - l o c a t i o n ( r o bo t - p o s e - h andov e r 1 ) )( a ch i e v e ‘ ( a rms -a t , a rms-a t101 ) )( a ch i e v e ‘ ( a rms -a t , unhand -ac t i on102 ) ) ) )
Requirements of actions on robot capabilities
Objectives:
◮ Find out if robot can do a job or, if not, what is missing
◮ Technically: flexible matching between robot capabilities andaction requirements
Complex problem because of
◮ hierarchical structure of robot components
◮ temporal action hierarchy (task composed of sub-actions)
◮ inheritance hierarchy (requirements defined on higher levels)
◮ inter-dependencies (capabilities depend on other components orcapabilities)
◮ tasks do not usually come with requirement specs, they need to beinferred using prior knowledge
SRDL: Robot description
◮ Kinematic robot model imported from URDF◮ Links and joints, incl. coordinate transformations◮ linked by sub-properties of transitive subComponent property
◮ Manual semantic annotations: ’these links form a gripper’
◮ Robot capabilities, e.g. 2D navigation(see also http://ias.in.tum.de/kb/wiki/index.php/SRDL2 Tutorial)
SRDL: Query for available robot components
sub component ( Super , Sub ) : -ow l ha s ( Super , s rd l2comp : ’ subComponent ’ , Sub ) .
sub component ( Super , Sub ) : -ow l ha s ( Sub , s rd l2comp : ’ baseL inkOfCompos i t i on ’ , Base ) ,
sub component ( Super , Base ) ,ow l ha s ( Sub , s rd l2comp : ’ endL inkOfCompos i t i on ’ , End ) ,
sub component ( Super , End ) .
c omp t y p e a v a i l a b l e ( Super , SubT) : -sub component ( Super , Sub ) ,o w l i n d i v i d u a l o f ( Sub , SubT ) .
◮ Very simple queries for reading components and checkingavailability
◮ Query exploits transitivity of subComponent property
SRDL: Query for available capabilities
% c a p a b i l i t y a s s e r t e d f o r r obo t i n s t a n c ec a p a v a i l a b l e o n r o b o t (Cap , Robot ) : -ow l ha s ( Robot , s r d l 2 c a p : ’ h a s C a p a b i l i t y ’ , SubCap ) ,o w l s u b c l a s s o f ( SubCap , Cap ) .
% c a p a b i l i t y a s s e r t e d f o r r obo t c l a s sc a p a v a i l a b l e o n r o b o t (Cap , Robot ) : -r d f s i n d i v i d u a l o f ( Robot , RobotC la s s ) ,c l a s s p r o p e r t i e s ( RobotClass , s r d l 2 c a p : ’ h a s C a p a b i l i t y ’ , SubCap ) ,o w l s u b c l a s s o f ( SubCap , Cap ) .
% c a p a b i l i t y depends on l y on a v a i l a b l e components or c a p a b i l i t i e sc a p a v a i l a b l e o n r o b o t (Cap , Robot ) : -
r d f s s u b c l a s s o f (Cap , s r d l 2 c a p : ’ C a p a b i l i t y ’ ) ,
f o r a l l ( c l a s s p r o p e r t i e s (Cap , s rd l2comp : ’ dependsOnComponent ’ ,CompT) ,c omp t y p e a v a i l a b l e ( Robot ,CompT) ) ,
f o r a l l ( c l a s s p r o p e r t i e s (Cap , s r d l 2 c a p : ’ d epend sOnCapab i l i t y ’ , SubCap ) ,c a p a v a i l a b l e o n r o b o t ( SubCap , Robot ) ) .
SRDL: Query for required components
◮ Next step: read required components from action specification
◮ Recursively read dependencies for sub-actions
◮ Inherit dependencies from super-classes
◮ Equivalent for capability dependencies
% d i r e c t l y r e q u i r e d by an a c t i o n and i t s sub - a c t i o n sr e q u i r e d c omp f o r a c t i o n ( Act ion ,Comp) : -
c l a s s p r o p e r t i e s ( Act ion , s rd l2comp : ’ dependsOnComponent ’ ,Comp ) .
% r e q u i r e d by any o f i t s sub - a c t i o n sr e q u i r e d c omp f o r a c t i o n ( Act ion ,Comp) : -
p l a n s u b e v e n t s r e c u r s i v e ( Act ion , SubAct ion ) ,c l a s s p r o p e r t i e s ( SubAction , s rd l2comp : ’ dependsOnComponent ’ ,Comp ) .
% i n d i r e c t l y r e q u i r e d by r e q u i r e d c a p a b i l i t i e sr e q u i r e d c omp f o r a c t i o n ( Act ion ,Comp) : -
r e q u i r e d c a p f o r a c t i o n ( Act ion , Cap ) ,c l a s s p r o p e r t i e s (Cap , s rd l2comp : ’ dependsOnComponent ’ ,Comp ) .
SRDL: Determine missing components
◮ Next: determine missing components (required but not available)
◮ Availability is recursively evaluated: available capabilities aredefined as ’depend only on available ones’
◮ Empty result: nothing missing → action feasible
m i s s i n g c omp f o r a c t i o n ( Act ion , Robot , Comp) : -r e q u i r e d c omp f o r a c t i o n ( Act ion , Comp) ,\+ comp t y p e a v a i l a b l e ( Robot , Comp ) .
m i s s i n g c a p f o r a c t i o n ( Act ion , Robot , Cap ) : -r e q u i r e d c a p f o r a c t i o n ( Act ion , Cap ) ,\+ c a p a v a i l a b l e o n r o b o t (Cap , Robot ) .
a c t i o n f e a s i b l e o n r o b o t ( Act ion , Robot ) : -\+ m i s s i n g c a p f o r a c t i o n ( Act ion , Robot , ) ,\+ m i s s i n g c omp f o r a c t i o n ( Act ion , Robot , ) .
SRDL: Lessons learned
◮ Inference is supported by modeling:◮ Transitivity: read whole branches of kinematic tree at once◮ Property hierarchy: describe specifically, query generically◮ Class taxonomy: simply inherit dependencies from superclasses
◮ Task specifications do not need any special annotation◮ Dependencies inherited from generic action classes◮ Action dependencies are stored locally and are thus reusable
◮ Prolog facilitates definition of additional solutions:◮ E.g. determine available capabilities by looking at the ROS network
◮ Easy implementation of complex behavior◮ Few simple Prolog rules and a bit of OWL modeling◮ Robot model largely imported automatically
Exercises (30min):
KnowRob actions tutorial:http://ias.in.tum.de/kb/wiki/index.php/Reasoning about actions
Additional information:http://ias.in.tum.de/kb/wiki/index.php/Modeling tasks and actionshttp://ias.in.tum.de/kb/wiki/index.php/Modeling changing objects
Outline: KnowRob tutorial
◮ Introduction: What can CRAM, KnowRob, etc be used for?
◮ Languages: Prolog, OWL, Java, etc
◮ Basics: launching KnowRob, loading files, querying, debugging
◮ Objects: class taxonomy, instances, components, environmentmodels, units and coordinates, qualitative spatial relations,computable properties
◮ Actions: class-level descriptions, properties, projection, planning,action dependencies
◮ RoboEarth: concept, interface to KnowRob, up- anddownload of information, capability matching
◮ ProbCog: interface to statistical relation inference (Dominik)
Information exchange among humans
Information exchange among robots
Information exchange among robots
Information exchange among robots
Information exchange among robots
RoboEarth
(see also
http://ias.in.tum.de/kb/wiki/index.php/Exchanging information via RoboEarth)
Application of SRDL in RoboEarth
Video: Downloading recipes, objects, and maps
Object models
Finding maps for an environment
Exercises (45min):
KnowRob RoboEarth tutorial:http://ias.in.tum.de/kb/wiki/index.php/Exchanging information via RoboEarth
Additional information:http://www.ros.org/wiki/roboearthhttp://www.ros.org/wiki/re comm
Outline: KnowRob tutorial
◮ Introduction: What can CRAM, KnowRob, etc be used for?
◮ Languages: Prolog, OWL, Java, etc
◮ Basics: launching KnowRob, loading files, querying, debugging
◮ Objects: class taxonomy, instances, components, environmentmodels, units and coordinates, qualitative spatial relations,computable properties
◮ Actions: class-level descriptions, properties, projection, planning,action dependencies
◮ RoboEarth: concept, interface to KnowRob, up- and download ofinformation, capability matching
◮ ProbCog: interface to statistical relation inference (Dominik)
Interface to statistical relational learning and
inference
Objectives:
◮ Integrate probabilistic logical models to account for uncertaininformation
◮ Query for results generated by statistical relational models
◮ Provide evidence for statistical inference to ProbCog
(see also http://ias.in.tum.de/research/probcog)
ProbCog interface
Thank you for your attention