Deep Learning For Detecting Robotic Grasps · De nition Deep Learning for Detecting Robotic Grasps...

University of Hamburg

MIN Faculty

Department of Informatics

Deep Learning for Detecting Robotic Grasps

Deep Learning For Detecting Robotic Grasps

Waleed Mustafa

University of HamburgFaculty of Mathematics, Informatics and Natural SciencesDepartment of Informatics

Technical Aspects of Multimodal Systems

4. Januar 2016

W. Mustafa 1

MIN Faculty

Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

3. Overview of Grasp Process

4. Grasp Representation

5. Detect Grasp from Image

6. Learning Grasp Ranking Function

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 2

MIN Faculty

Definition Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 3

MIN Faculty

Definition Deep Learning for Detecting Robotic Grasps

Definition

”A grasp is commonly defined as a set of contacts on the surfaceof the object, which purpose is to constrain the potentialmovements of the object in the event of externaldisturbances”Leon et al. (2014)

Grasp Synthesis

”Grasp synthesis is the problem of finding a suitable set ofcontacts given an object and some constraints on the allowablecontacts”Leon et al. (2014)

W. Mustafa 4

MIN Faculty

Motivation Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 5

MIN Faculty

Motivation Deep Learning for Detecting Robotic Grasps

MotivationWhy do we need robotic grasp?

I Almost all robotic applications include manipulation of objects

I In order to manipulate an object you need first to grasp itI Applications include:

I Exploration

I Household

I Industry robotic hands

W. Mustafa 6

MIN Faculty

Overview of Grasp Process Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 7

MIN Faculty

Overview of Grasp Process Deep Learning for Detecting Robotic Grasps

Overview of Grasp Process

Predict Gripper configuration (i.e., Gripper Location,Orientation, and Gripper Opening Width)

Input:

2-D Image, and Depth Map

Output:

Grasp Representation? We need parameters that represent thegripper configuration

W. Mustafa 8

MIN Faculty

Grasp Representation Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 9

MIN Faculty

Grasp Representation

I A Grasp is:I Gripper LocationsI Gripper PoseI Gripper Opening Width

W. Mustafa 10

MIN Faculty

Grasp Representation (cont.)

I A good representation is:I Easily predicted it from sensory dataI The full grasp parameters can be retrieved from it

I Saxena et al. (2008) proposed one point as a representation ofa graspI Easy to predictI Procedure for retrieving Grasp parameters hard and faulty

I Jiang et al. (2011) Represented grasps as an oriented rectangle

W. Mustafa 11

MIN Faculty

Grasp Representation (cont.)

I Grasp is defined by:I rG ,cG position in image plan, mG , nG width and Height of

rectangle θ is the angle of rectangle with respect to X -axis

W. Mustafa 12

MIN Faculty

Grasp Representation

I Positions, Opening Width, and pose around camera axis aredirectly defined

I Other two angles is computed by:I Select the point with lower depth in the middle thirdI Compute average surface norm around this point Jiang et al.

(2011)

W. Mustafa 13

MIN Faculty

Detect Grasp from Image Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 14

MIN Faculty

Detect Grasp from Image

1. Generate every possible grasp rectangle

W. Mustafa 15

MIN Faculty

Detect Grasp from Image (cont.)

2. Using a function f (x |Θ) : Rn → [0, 1] rank the rectangles,where x is features computed from rectangles

3. Choose the rectangle with highest rank

W. Mustafa 16

MIN Faculty

Learning Grasp Ranking Function Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 17

MIN Faculty

Learning Grasp Ranking Function

I Given a training set (Images with Human marking) we cantrain the function f (x |Θ)

I Jiang et al. (2011) proposed extracting feature from potentialrectangle and build an SVM classifierI Their features was histogram of different filters

I Lenz et al. (2015) Used sparse auto-encoder to automaticallylearn features Goodfellow et al. (2009)I Used neural network to learn the rank functionI The input to the network

W. Mustafa 18

MIN Faculty

W ∗ = arg minW

∑Mt=1 (‖x − x‖2

2 + λ∑K

j=1 g(h(t))) + βf (W )

h(t)j = σ(

∑Ni=1 x

(t)i Wij)

x(t)i =

∑Kj=1 h

(t)j Wij

Lenz et al. (2015)W. Mustafa 19

MIN Faculty

I We can repeat the above process to learn N layersI Finally we stack learned layers together with an output decision

layersI Complete the learning with BP

Lenz et al. (2015)

W. Mustafa 20

MIN Faculty

I It is very slow to run the huge network on all the rectangles

I Lenz et al. (2015) propose a cascaded system

Lenz et al. (2015)

W. Mustafa 21

MIN Faculty

Multi-modal Input

I Concatenate the data from different modes

Lenz et al. (2015)

W. Mustafa 22

MIN Faculty

Multi-modal Input (cont.)

I Separate modes at the first layer

Lenz et al. (2015)

W. Mustafa 23

MIN Faculty

Multi-modal Input (cont.)

I Or a mix

Lenz et al. (2015)

W. Mustafa 24

MIN Faculty

Resutls Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 25

MIN Faculty

Resutls Deep Learning for Detecting Robotic Grasps

Resutls

Lenz et al. (2015)

W. Mustafa 26

MIN Faculty

Demo Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 27

MIN Faculty

Demo Deep Learning for Detecting Robotic Grasps

W. Mustafa 28

MIN Faculty

Conclusion Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 29

MIN Faculty

Conclusion Deep Learning for Detecting Robotic Grasps

Conclusion

I We introduced a grasp detection system based on deep learning

I Results shows that it outperforms systems that are based onhuman designed features

I The problem have a lot on common with object detection

I A lot of methods in object detection can be used

I Overfeat will be fasterI CNN might be more suitable for Images

W. Mustafa 30

MIN Faculty

References Deep Learning for Detecting Robotic Grasps

Outline

1. Definition

2. Motivation

7. Resutls

8. Demo

9. Conclusion

10. References

W. Mustafa 31

MIN Faculty

References

I Goodfellow, H Lee, and QV Le. Measuring invariances in deepnetworks. Advances in neural . . . , 2009. URLhttp://papers.nips.cc/paper/

3790-measuring-invariances-in-deep-networks.

Y Jiang, S Moseson, and A Saxena. Efficient grasping from rgbdimages: Learning using a new rectangle representation. Roboticsand Automation ( . . . , 2011. URL http://ieeexplore.ieee.

org/xpls/abs_all.jsp?arnumber=5980145.

I Lenz, H Lee, and A Saxena. Deep learning for detecting roboticgrasps. The International Journal of Robotics, 2015. URLhttp://ijr.sagepub.com/content/34/4-5/705.short.

W. Mustafa 32

MIN Faculty

References (cont.)

B Leon, A Morales, and J Sancho-Bru. From robot to humangrasping simulation. 2014. URL http://link.springer.com/

content/pdf/10.1007/978-3-319-01833-1.pdf.

A Saxena, J Driemeyer, and AY Ng. Robotic grasping of novelobjects using vision. International Journal of Robotics, 2008.URL http://ijr.sagepub.com/content/27/2/157.short.

W. Mustafa 33

Deep Learning For Detecting Robotic Grasps · De nition Deep Learning for Detecting Robotic Grasps...

Documents

Transcript of Deep Learning For Detecting Robotic Grasps · De nition Deep Learning for Detecting Robotic Grasps...

GRASPS and ROLES

Efﬁcient GRASP+VND and GRASP+VNS metaheuristics for the ... · Efﬁcient GRASP+VND and GRASP+VNS metaheuristics for the traveling repairman problem ... 1 Introduction The traveling

Synthesizing Diverse and Physically Stable Grasps with ...

Grasp planning for thin-walled deformable objects · Grasp planning for thin-walled deformable objects ... We limit our discussion to grasps with two contact points ... corresponds

Grasps powerpoint final

Generating Multi-Fingered Robotic Grasps via Deep Learningallen/PAPERS/iros15_grasp_varley.pdfsystem efﬁcient, allow the use of a large context window for determining grasp quality,

Energy sources(Grade 9 GRASPS)

South Mansfield College - smc.edu.ph 7 Altruism - 10.pdf · Music: Lowland Luzon Tetratonic Songs Art: GRASPS Computer: GRASPS Health: Management of Health Concerns During Adolescence

general-purpose simple hand © The Author(s) 201 · 2018. 3. 29. · Grasp taxonomies and in-hand manipulation are cou-pled. Grasp taxonomies often identify two broad classes of grasps:

The GRASP Taxonomy of Human Grasp Typesgrasp.xief.net/documents/THMS_taxonomy.pdf · The GRASP Taxonomy of Human Grasp Types 1 ... classification, the 33 grasp types might be reduced

slpemad.files.wordpress.com · outae.ynes ror sconng nanã runct/on: minimai independent hand grasps, no active release, unable to combine reach and grasp spontaneous reach, grasp

Finding Enveloping Grasps by Matching Continuous Surfaces

GRASPS K-12 Writing

Grasps examples

Modeling of grasps in judo competition contests...1 Modeling of grasps in judo competition contests Michel Calmet1,2, Bianca Miarka 2, Emerson Franchini 1 – Montpellier University,

SURJ Learning to Select Robotic Grasps Usingpeople.csail.mit.edu/lsw/papers/surj08-graspinglearn.pdf · While in theory this fully determines the result of the grasp, this approach

Robust proprioceptive grasping with a soft robot handhand capable of dexterous grasps (Deimel and Brock 2014, 2016).Stokesetal.(2014)usedasoftelastomerquadrupedal robot to grasp objects

Grasp planning via hand-object geometric fitting · Grasp planning via hand-object geometric ﬁtting ... mate the shape of a 3D object, which could be tedious. We observe that grasps

Primitive Reflexes That Influence Grasp/media/book/occupational-therapy/hand-grasps-2e/... · and proprioceptive input (Case-Smith, 1995). Primitive reflexes that influence grasp

One shot learning and generation of dexterous grasps for ... · This paper presents a method for one-shot learning of dexterous grasps, and grasp generation for novel objects. A model