Smart Robotic Assistants for Manufacturing Applications

through Advances in Artificial Intelligence

Satyandra K. Gupta

Director, Center for Advanced Manufacturing (CAM)

Smith International Professor

Aerospace and Mechanical Engineering Department

Computer Science Department

Viterbi School of Engineering

University of Southern California

Robots are physically capable of performing highly complex tasks

Today’s Industrial Robots

Source: https://www.youtube.com/watch?v=8_lfxPI5ObM Source: https://www.youtube.com/watch?v=SOESSCXGhFo

Source: https://www.youtube.com/watch?v=NAqm5_jk2ws Source: https://www.youtube.com/watch?v=CigVLW4M4-I

• Need human experts to program robots

• Configuring a robotic cell for a new task takes significant time and effort

• Robots repeat preprogrammed motions and cannot automatically adapt to changes in the workspace or tasks

• Recovering from errors requires significant downtime and can be very expensive

Characteristics of Traditional Industrial Robots

(Image Source: http://www.fanuc.eu)

Robots are largely used in mass production applications.

Less than 2 robots for every 100 manufacturing workers in US.

Recent Advances in Industrial Robots

Multi-Arm

Manipulation

Stereo Vision

Force Sensing

Tactile Sensing

Impedance Control

Visual Servo

Shape Control

Human Safe Robots

Mobile

Manipulation

Relying on humans to program industrial robots is not a viable option as robotic cells get more complex

Physics-Aware Artificial Intelligence (AI) to realize Smart Robotic Assistants

Our Approach

Capability #1: Robots that Program Themselves

• Agility and reconfigurability

requires robots to program

themselves from task descriptions

─ Automated path planning and

instruction generation

• Real-time computation of near

optimal trajectories for high degree

of freedom problems

─ Generate trajectories by minimizing

length/time/torque/jerk

─ Satisfy position and orientation

constraints along the trajectory

─ Avoid collisions

─ Satisfy task specific constraints along

the path

• Task and motion planning integration – Accounting for interactions among

agents during task assignment

• Part placement planning for robotic cells

– Successive refinement strategies for satisfying process requirements

• Point-to-point trajectory planning for manipulators

– Context-dependent search strategy switching for efficiently searching different regions of the search space

– Context identification based on the search tree configuration and progress

• Path constrained trajectory planning for manipulators

– Adaptive selection of configuration space trajectory representations for efficient planning

– Successive refinement strategies to find solution for optimization problem with complex interaction among constraints

https://youtu.be/bD7FItjOWY0

https://youtu.be/38746EJLqOI

Planning Technologies Developed at USC CAM

• Task performance model may not be known a priori

– Successfully completing a task requires meeting performance constraints

• Robots need to conduct experiments on their own to learn and use task performance model during task planning and execution

• Self-directed learning is needed to efficiently complete the given task

– Need to find the right tradeoff between model building experiments and task execution

Capability #2: Robots that Learn by Conducting Experiments

Robotic Sanding

• Aim to achieve the right balance

between exploration and exploitation

• Select trajectory parameters for

exploratory experiment(s) ─ Based on expected reduction in model

uncertainty

─ Hierarchical sampling

• Update meta models for constraint

satisfaction prediction after each

experiment

• Reduce prediction uncertainty by

conducting additional experiments ─ Selection of trajectory parameters for

refining experiment based on

probability of success

Self-Directed Learning Technology Developed at USC CAM

• Vision-based localization leads to significant uncertainties

– Agile and rapidly reconfigurable robot cells cannot use custom fixtures to reduce uncertainty

• Need contact based probing to improve uncertainty in part localization

– Provide safety guarantees during contact-based probing to reduce model uncertainty

• Robot should not apply excessive force on objects or cause excessive deformation

• Robot should not make unwanted contact with the objects in the environment

Capability #3: Safe Execution Under Uncertainty

• Contact-based probing with safety

constraints – Use heuristics search-based methods to

select contact points and tracing curves that

are expected to improve part pose

estimation accuracy with least effort

– Generate trajectories to reach contact points

safely under the estimated uncertainty Use contact-based probing data to accurately

determine part pose

• Safe insertion under large uncertainties – Select parameters of impedance controller to

avoid contact with delicate part features

– Use machine learning on force feedback from robot to generate optimal search strategy for determining the insertion location

Human-robot collaborative cell

for fixtureless assemblies

Assembly of a rectangular peg

in a hole problem

Safety-Aware Execution Technologies Developed at USC CAM

• Robot should estimate its task completion confidence and consequence of task completion failure

– Using uncertainty model

– Availability of contingency maneuvers

– Utilizing smart tools that can monitor tool state and task performance

• If the task completion confidence is low, then the system should seek help from the human

– Expert-on-Call Concept

Capability # 4: Robots that Seek Help from Humans

• Real-time tool-state monitoring based on task performance changes

– Request for tool change

• Identification of infeasible motion planning problem instances – Request for setup change

• Prediction of trajectory/setup planning failure – Request for human guidance during planning

• Estimation of plan execution risk – Request for human help to reduce part uncertainty

Sub-goal

Configuration

provided by human

Introspection Technologies Developed at USC CAM

• Robots need to be tasked by humans

• Robots need to elicit help from humans to handle challenging tasks

• Humans need to perform tasks such as tool and setup changes in robotic cells

• Interaction between humans and robots should ensure human safety

• Efficient human robot communication is a key to safe and efficient operation of robotic cells

Capability #5: Robots that Communicate Efficiently with Humans

• Augmented reality-based interfaces for robotic cells to facilitate information exchange between humans and robots

– Display safety zones

– Confirm instructions given by humans

– Display errors and warnings

– Display internal system states and enable humans to diagnose problems

• Context-dependent level of detail control to prevent cognitive overload for humans

Context-Aware Information Exchange Technologies Developed at USC CAM

• Domains: Sanding, Grinding, and Polishing

• One of the top 5 finalists in KUKA Innovation Award 2017

• On-line Planning – Setup planning to determine optimal

number of setups

– Tool and robot trajectory planning for each setup

• Self-directed learning – Selection of tool and robot trajectory

parameters using self-directed learning

– Combination of data-driven and model-driven approach to learn finishing process model

• Safe execution – Impedance control and on-line perception

for task safety

Application # 1: Robotic Finishing

Ra: 4.7021 um

Before Polishing Polished Surface

Robotic Finishing: Result on 3D Printed Metal Part

Ra: 0.4491 um

Robotic Finishing Videos

• https://youtu.be/0HuwavBxqp8

• https://youtu.be/DbCDVHBuOwI

• https://youtu.be/JcsQF19Xe9I

• https://youtu.be/HJGdn1Wrg3s

• Multi-arm manipulation of deformable sheets

– Holding

– Heating

– Stretching

– Deformation

• Design of end effectors for material dispensing and layup on complex molds

• Trajectory planning

• Self-directed learning

• Safe trajectory execution using impedance control

Application 2: Robotic Composite Layup

Robotic Composite Layup Videos

• https://youtu.be/yUAUaATqINw

• https://youtu.be/3XT9nzCwlOs

• Task planning and resource

allocation to explicitly account for

managing contingencies

• Trajectory planning

• Real-time monitoring of the task

progress during the execution

• Real-time replanning algorithms

to handle contingencies

• Human robot interaction to deal

with contingencies

Application 3: Collaborative Assembly

Collaborative Assembly Videos

• https://youtu.be/yqT2XHwkpOo

• https://youtu.be/E9IUc5p9Qck

Application 4: Robotic Bin Picking and Kitting

• Bin Picking – 3D imaging for object detection

– Tangle prediction

– Trajectory planning for transport

and singulation operations

• Kitting – Won ARIAC competition

organized by NIST in 2017

– Task and trajectory planning

– Real-time monitoring of the task

progress

– Real-time replanning to handle

contingencies

Bin Picking and Kitting Videos

• https://youtu.be/DKuR1KxSu5Y • https://youtu.be/95fC6mh34jI

Application 5: Machine Tending

• Robot arms integrated with mobile base

– Navigate to task locations in shop floors

– Perform pick and place operations using the manipulator

• Trajectory planning for concurrent mobility and manipulation

– Search for time optimal platform trajectory that causes least slow down in the arm during the pick up

– Automatically compute grasping location

Mobile Manipulation Video

• https://youtu.be/8fWLFA97TCo

• https://youtu.be/B4SumiabVus

Application 6: Robot Assisted Additive Manufacturing

• Use 6DOF robot to perform non-planar layer deposition

– Minimize number of layers

– Improve strength

– Reduce support material

– Enable use of continuous fibers in composite materials

• Use two collaborating robots to perform support less printing

• Trajectory planning and velocity regulation

Additive Manufacturing Videos

• https://youtu.be/HaOvKdOUBPE

• https://youtu.be/hxdW7sj0cmk

• https://www.youtube.com/watch?v=kl4scG38DFs

Summary

• Automated decision making is a key to realizing smart robotic assistants

– Need Physics-Aware AI

– Application context is important

• Smart robotic assistants enable

– Rapid reconfigurability

– Improved productivity of human operators

– Enhanced process efficiency

• Demonstrated feasibility in several applications

─ Robotic Finishing

─ Collaborative Assembly

─ Automated Composite Layup

─ Robot Assisted Additive Manufacturing

─ Machine Tending Using Mobile

Manipulators

─ Robotic Bin Picking and Kitting

Contributors

SK Gupta Rohil Aggarwal Abdullah Alsharhan Vivek Annem Prahar Bhatt Dr. Timotei Centea Jingsong Chu

Liwei Fang Ariyan Kabir Dr. Krishna Kaipa Alec Kanyuck Anirudh Kulkarni Nithyanand Kumbla Dr. Josh Langsfeld

Keng-Yu Lin Binwei Liu Rishi Malhan Dr. Carlos Morato Siddhant Nadkarni Pradeep Rajendran Dr. Brual Shah

Aniruddha Shembekar Shantanu Thakar Dennis Wang Yeo Jung Yoon Yash Shahapurkar

• S. Shriyam and S.K. Gupta. Modeling and analysis of subsystem interactions in robotic assembly. ASME Computers and

Information in Engineering Conference, Anaheim, CA, August 2019.

• Y.J. Yoon, M. Yon, S.E. Jung, and S.K. Gupta. Development of three-nozzle extrusion system for conformal multi-resolution

3D printing with a robotic manipulator. ASME Computers and Information in Engineering Conference, Anaheim, CA, August

2019.

• A. V. Shembekar, Y. J. Yoon, A. Kanyuck and S.K. Gupta. Generating robot trajectories for conformal 3D printing using non-

planar layers. ASME Journal of Computing and Information Science in Engineering, Accepted for publication.

• P. M. Bhatt, A. M. Kabir, R. K. Malhan, B. C. Shah, A. V. Shembekar, Y. J. Yoon, and S.K. Gupta. A robotic cell for multi-

resolution additive manufacturing. IEEE International Conference on Robotics and Automation, Montreal, Canada, May 2019.

• S. Thakar, V. Annem, A. Kabir, P. Rajendran, and S. K. Gupta. Accounting for part pose estimation uncertainties during

trajectory generation for part pick-up using mobile manipulators. IEEE International Conference on Robotics and Automation,

Montreal, Canada, May 2019.

• R. K. Malhan, A. M. Kabir, B. C. Shah, and S. K. Gupta. Identifying feasible workpiece placement with respect to redundant

manipulator for complex manufacturing tasks. IEEE International Conference on Robotics and Automation, Montreal,

Canada, May 2019.

• A. M. Kabir, A. Kanyuck, R. K. Malhan, A. V. Shembekar, S. Thakar, B. C. Shah, and S. K. Gupta. Generation of

synchronized configuration space trajectories of multi-robot systems. IEEE International Conference on Robotics and

Automation, Montreal, Canada, May 2019.

• P. M. Bhatt, P. Rajendran, K. McKay, and S. K. Gupta. Context-dependent compensation scheme to reduce trajectory

execution errors for industrial manipulators. IEEE International Conference on Robotics and Automation, Montreal, Canada,

May 2019.

• P.M. Bhatt, R.K. Malhan, and S.K. Gupta. Computational foundations for using three degrees of freedom build platforms to

enable supportless extrusion-based additive manufacturing. ASME Manufacturing Science and Engineering Conference,

Erie, PA, June 2019.

• R.K. Malhan, A.M. Kabir, B. Shah, T. Centea, and S.K. Gupta. Determining feasible robot placements in robotic cells for

composite prepreg sheet layup. ASME Manufacturing Science and Engineering Conference, Erie, PA, June 2019.

• V. Annem, P. Rajendran, S. Thakar, and S.K. Gupta. Towards remote teleoperation of a semi-autonomous mobile manipulator

system in machine tending tasks. ASME Manufacturing Science and Engineering Conference, Erie, PA, June 2019.

References

• J.D. Langsfeld, A.M. Kabir, K.N. Kaipa, and S.K. Gupta. Integration of planning and deformation model

estimation for robotic cleaning of elastically deformable objects. IEEE Robotics and Automation Letters,

3(1): 352-359, 2018.

• K.N. Kaipa, C.W. Morato, and S.K. Gupta. Design of hybrid cells to facilitate safe and efficient human-robot

collaboration during assembly operations. ASME Journal of Computing and Information Science in

Engineering, 18(3):031004, 2018.

• A.M. Kabir, J.D. Langsfeld, K.N. Kaipa, and S.K. Gupta. Identifying optimal trajectory parameters in robotic

finishing operations using minimum number of physical experiments. Integrated Computer-Aided

Engineering Journal, 25(2):111-135, 2018.

• N.B. Kumbla, S. Thakar, K. N. Kaipa, J.A. Marvel, and S.K. Gupta. Handling perception uncertainty in

simulation based singulation planning in robotic bin picking. ASME Journal of Computing and Information

Science in Engineering, 18(2):021004-021004-10, 2018.

• S. Thakar, L. Fang, B.C. Shah, and S.K. Gupta. Towards time-optimal trajectory planning for pick-and-

transport operation with a mobile manipulator. IEEE International Conference on Automation Science and

Engineering, Munich, Germany, Aug 2018.

• N.B. Kumbla, J.A. Marvel, and S.K. Gupta. Enabling fixtureless assemblies in human-robot collaborative

workcells by reducing uncertainty in the part pose estimate. IEEE International Conference on Automation

Science and Engineering, Munich, Germany, Aug 2018.

• A.M. Kabir, B.C. Shah, and S.K. Gupta. Trajectory Planning for manipulators operating in confined

workspaces. IEEE International Conference on Automation Science and Engineering, Munich, Germany,

Aug 2018.

References (Cont.)

• R. K. Malhan, A. M. Kabir, A. V. Shembekar, B. C. Shah, T. Centea, and S.K. Gupta. Hybrid cells for multi-layer prepreg composite sheet layup. IEEE International Conference on Automation Science and Engineering, Munich, Germany, Aug 2018.

• A.V. Shembekar, Y. J. Yoon, A. Kanyuck and S.K. Gupta. Trajectory planning for conformal 3d printing using non-planar layers. ASME Computers and Information in Engineering Conference, Montreal, Quebec City, Canada, August 2018.

• P.M. Bhatt, M. Peralta, H.A. Bruck, and S.K. Gupta. Robot assisted additive manufacturing of thin multifunctional structures. ASME Manufacturing Science and Engineering Conference, College Station, TX, June 2018.

• A.M. Kabir, A.V. Shembekar, R.K. Malhan, R.S. Aggarwal, J.D. Langsfeld, B.C. Shah, and S.K. Gupta. Robotic finishing of interior regions of geometrically complex parts. ASME Manufacturing Science and Engineering Conference, College Station, TX, June 2018.

• R.K. Malhan, Y. Shahapurkar, A.M. Kabir, B.C. Shah, and S.K. Gupta. Integrating impedance control and learning based search scheme for robotic assemblies under uncertainty. ASME Manufacturing Science and Engineering Conference, College Station, TX, June 2018.

• R.K. Malhan, A.M. Kabir, B. Shah, T. Centea, S.K. Gupta. Automated prepreg sheet placement using collaborative robotics. SAMPE, Long Beach, CA, May 2018.

• A.M. Kabir, K.N. Kaipa, J. Marvel, S.K. Gupta. Automated planning for robotic cleaning using multiple setups and oscillatory tool motions. IEEE Transaction on Automation Science and Engineering, 14(3):1364-1377, July 2017.

• C.W. Morato, K.N. Kaipa, and S,K. Gupta. System state monitoring to facilitate safe and efficient human-robot collaboration in hybrid assembly cells. ASME Computers and Information in Engineering Conference, Cleveland, OH, August 2017.

References (Cont.)

• N. Bhatt, S. Thakar, K.N. Kaipa, J. Marvel, and S.K. Gupta. Simulation-based on-line evaluation of singulation plans to handle perception uncertainty in robotic bin-picking. ASME Manufacturing Science and Engineering Conference, Los Angeles, CA, USA, June 2017.

• A. Alsharhan, T. Centea, and S.K. Gupta. Enhancing the mechanical properties of thin-walled structures using non-planar extrusion-based additive manufacturing. ASME Manufacturing Science and Engineering Conference, Los Angeles, CA, USA, June 2017.

• A.M. Kabir, J.D. Langsfeld, C. Zhuang, K.N. Kaipa, and S.K. Gupta. A systematic approach for minimizing physical experiments to identify optimal trajectory parameters for robots. IEEE International Conference on Robotics and Automation, Singapore, May 2017.

• K.N. Kaipa, C. Morato, J. Liu, and S.K. Gupta. Towards automated generation of multimodal assembly instructions for human operators. International Conference on Systems Engineering Research, Redondo Beach, CA, March 2017.

• K.N. Kaipa, A.S. Kankanhalli-Nagendra, N.B. Kumbla, S. Shriyam, S. S. Thevendria-Karthic, J.A. Marvel, and S.K. Gupta. Addressing perception uncertainty induced failure modes in robotic bin-picking. Robotics and Computer Integrated Manufacturing, 42(12):17-38, 2016.

• A.M. Kabir, J.D. Langsfeld, S. Shriyam, V. Rachakonda, C. Zhaung, K.N. Kaipa, J. Marvel, and S.K. Gupta. Planning algorithms for multi-setup multi-pass robotic cleaning with oscillatory moving tools. IEEE International Conference on Automation Science and Engineering, Fort Worth, TX, August, 2016.

• J.D. Langsfeld, A.M. Kabir, K.N. Kaipa, and S.K. Gupta. Robotic bimanual cleaning of deformable objects with online learning of part and tool models. IEEE International Conference on Automation Science and Engineering, Fort Worth, TX, August 2016.

References (Cont.)

• K. N. Kaipa, A. S. Kankanhalli-Nagendra, N. B. Kumbla, S. Shriyam, S. S. Thevendria-Karthic, J. A. Marvel, and S. K. Gupta. Enhancing robotic unstructured bin-picking performance by enabling remote human interventions in challenging perception scenarios. IEEE International Conference on Automation Science and Engineering, Fort Worth, TX, August 2016.

• J.D. Langsfeld, A. M. Kabir, K.N. Kaipa, and S.K. Gupta. Online learning of part deformation models in robotic cleaning. ASME Manufacturing Science and Engineering Conference, Blacksburg, USA, June 2016.

• K.N. Kaipa, S. Shriyam, N. Kumbla, and S.K. Gupta. Resolving occlusions through simple extraction motions in robotic bin picking. ASME Manufacturing Science and Engineering Conference, Blacksburg, USA, June 2016.

• A.M. Kabir, J.D. Langsfeld, C. Zhuang, K.N. Kaipa, and S.K. Gupta. Automated leaning of operation parameters for robotic cleaning by mechanical scrubbing. ASME Manufacturing Science and Engineering Conference, Blacksburg, USA, June 2016.

References (Cont.)