Smart Robotic Assistants for Manufacturing Applications
through Advances in Artificial Intelligence
Satyandra K. Gupta
Director, Center for Advanced Manufacturing (CAM)
Smith International Professor
Aerospace and Mechanical Engineering Department
Computer Science Department
Viterbi School of Engineering
University of Southern California
Robots are physically capable of performing highly complex tasks
Today’s Industrial Robots
Source: https://www.youtube.com/watch?v=8_lfxPI5ObM Source: https://www.youtube.com/watch?v=SOESSCXGhFo
Source: https://www.youtube.com/watch?v=NAqm5_jk2ws Source: https://www.youtube.com/watch?v=CigVLW4M4-I
• Need human experts to program robots
• Configuring a robotic cell for a new task takes significant time and effort
• Robots repeat preprogrammed motions and cannot automatically adapt to changes in the workspace or tasks
• Recovering from errors requires significant downtime and can be very expensive
Characteristics of Traditional Industrial Robots
(Image Source: http://www.fanuc.eu)
Robots are largely used in mass production applications.
Less than 2 robots for every 100 manufacturing workers in US.
Recent Advances in Industrial Robots
Multi-Arm
Manipulation
Stereo Vision
Force Sensing
Tactile Sensing
Impedance Control
Visual Servo
Shape Control
Human Safe Robots
Mobile
Manipulation
Relying on humans to program industrial robots is not a viable option as robotic cells get more complex
Physics-Aware Artificial Intelligence (AI) to realize Smart Robotic Assistants
Our Approach
Capability #1: Robots that Program Themselves
• Agility and reconfigurability
requires robots to program
themselves from task descriptions
─ Automated path planning and
instruction generation
• Real-time computation of near
optimal trajectories for high degree
of freedom problems
─ Generate trajectories by minimizing
length/time/torque/jerk
─ Satisfy position and orientation
constraints along the trajectory
─ Avoid collisions
─ Satisfy task specific constraints along
the path
• Task and motion planning integration – Accounting for interactions among
agents during task assignment
• Part placement planning for robotic cells
– Successive refinement strategies for satisfying process requirements
• Point-to-point trajectory planning for manipulators
– Context-dependent search strategy switching for efficiently searching different regions of the search space
– Context identification based on the search tree configuration and progress
• Path constrained trajectory planning for manipulators
– Adaptive selection of configuration space trajectory representations for efficient planning
– Successive refinement strategies to find solution for optimization problem with complex interaction among constraints
https://youtu.be/bD7FItjOWY0
https://youtu.be/38746EJLqOI
Planning Technologies Developed at USC CAM
• Task performance model may not be known a priori
– Successfully completing a task requires meeting performance constraints
• Robots need to conduct experiments on their own to learn and use task performance model during task planning and execution
• Self-directed learning is needed to efficiently complete the given task
– Need to find the right tradeoff between model building experiments and task execution
Capability #2: Robots that Learn by Conducting Experiments
Robotic Sanding
• Aim to achieve the right balance
between exploration and exploitation
• Select trajectory parameters for
exploratory experiment(s) ─ Based on expected reduction in model
uncertainty
─ Hierarchical sampling
• Update meta models for constraint
satisfaction prediction after each
experiment
• Reduce prediction uncertainty by
conducting additional experiments ─ Selection of trajectory parameters for
refining experiment based on
probability of success
Self-Directed Learning Technology Developed at USC CAM
• Vision-based localization leads to significant uncertainties
– Agile and rapidly reconfigurable robot cells cannot use custom fixtures to reduce uncertainty
• Need contact based probing to improve uncertainty in part localization
– Provide safety guarantees during contact-based probing to reduce model uncertainty
• Robot should not apply excessive force on objects or cause excessive deformation
• Robot should not make unwanted contact with the objects in the environment
Capability #3: Safe Execution Under Uncertainty
• Contact-based probing with safety
constraints – Use heuristics search-based methods to
select contact points and tracing curves that
are expected to improve part pose
estimation accuracy with least effort
– Generate trajectories to reach contact points
safely under the estimated uncertainty Use contact-based probing data to accurately
determine part pose
• Safe insertion under large uncertainties – Select parameters of impedance controller to
avoid contact with delicate part features
– Use machine learning on force feedback from robot to generate optimal search strategy for determining the insertion location
Human-robot collaborative cell
for fixtureless assemblies
Assembly of a rectangular peg
in a hole problem
Safety-Aware Execution Technologies Developed at USC CAM
• Robot should estimate its task completion confidence and consequence of task completion failure
– Using uncertainty model
– Availability of contingency maneuvers
– Utilizing smart tools that can monitor tool state and task performance
• If the task completion confidence is low, then the system should seek help from the human
– Expert-on-Call Concept
Capability # 4: Robots that Seek Help from Humans
• Real-time tool-state monitoring based on task performance changes
– Request for tool change
• Identification of infeasible motion planning problem instances – Request for setup change
• Prediction of trajectory/setup planning failure – Request for human guidance during planning
• Estimation of plan execution risk – Request for human help to reduce part uncertainty
Sub-goal
Configuration
provided by human
Introspection Technologies Developed at USC CAM
• Robots need to be tasked by humans
• Robots need to elicit help from humans to handle challenging tasks
• Humans need to perform tasks such as tool and setup changes in robotic cells
• Interaction between humans and robots should ensure human safety
• Efficient human robot communication is a key to safe and efficient operation of robotic cells
Capability #5: Robots that Communicate Efficiently with Humans
• Augmented reality-based interfaces for robotic cells to facilitate information exchange between humans and robots
– Display safety zones
– Confirm instructions given by humans
– Display errors and warnings
– Display internal system states and enable humans to diagnose problems
• Context-dependent level of detail control to prevent cognitive overload for humans
Context-Aware Information Exchange Technologies Developed at USC CAM
• Domains: Sanding, Grinding, and Polishing
• One of the top 5 finalists in KUKA Innovation Award 2017
• On-line Planning – Setup planning to determine optimal
number of setups
– Tool and robot trajectory planning for each setup
• Self-directed learning – Selection of tool and robot trajectory
parameters using self-directed learning
– Combination of data-driven and model-driven approach to learn finishing process model
• Safe execution – Impedance control and on-line perception
for task safety
Application # 1: Robotic Finishing
Ra: 4.7021 um
Before Polishing Polished Surface
Robotic Finishing: Result on 3D Printed Metal Part
Ra: 0.4491 um
Robotic Finishing Videos
• https://youtu.be/0HuwavBxqp8
• https://youtu.be/DbCDVHBuOwI
• https://youtu.be/JcsQF19Xe9I
• https://youtu.be/HJGdn1Wrg3s
• Multi-arm manipulation of deformable sheets
– Holding
– Heating
– Stretching
– Deformation
• Design of end effectors for material dispensing and layup on complex molds
• Trajectory planning
• Self-directed learning
• Safe trajectory execution using impedance control
Application 2: Robotic Composite Layup
Robotic Composite Layup Videos
• https://youtu.be/yUAUaATqINw
• https://youtu.be/3XT9nzCwlOs
• Task planning and resource
allocation to explicitly account for
managing contingencies
• Trajectory planning
• Real-time monitoring of the task
progress during the execution
• Real-time replanning algorithms
to handle contingencies
• Human robot interaction to deal
with contingencies
Application 3: Collaborative Assembly
Collaborative Assembly Videos
• https://youtu.be/yqT2XHwkpOo
• https://youtu.be/E9IUc5p9Qck
Application 4: Robotic Bin Picking and Kitting
• Bin Picking – 3D imaging for object detection
– Tangle prediction
– Trajectory planning for transport
and singulation operations
• Kitting – Won ARIAC competition
organized by NIST in 2017
– Task and trajectory planning
– Real-time monitoring of the task
progress
– Real-time replanning to handle
contingencies
Bin Picking and Kitting Videos
• https://youtu.be/DKuR1KxSu5Y • https://youtu.be/95fC6mh34jI
Application 5: Machine Tending
• Robot arms integrated with mobile base
– Navigate to task locations in shop floors
– Perform pick and place operations using the manipulator
• Trajectory planning for concurrent mobility and manipulation
– Search for time optimal platform trajectory that causes least slow down in the arm during the pick up
– Automatically compute grasping location
Mobile Manipulation Video
• https://youtu.be/8fWLFA97TCo
• https://youtu.be/B4SumiabVus
Application 6: Robot Assisted Additive Manufacturing
• Use 6DOF robot to perform non-planar layer deposition
– Minimize number of layers
– Improve strength
– Reduce support material
– Enable use of continuous fibers in composite materials
• Use two collaborating robots to perform support less printing
• Trajectory planning and velocity regulation
Additive Manufacturing Videos
• https://youtu.be/HaOvKdOUBPE
• https://youtu.be/hxdW7sj0cmk
• https://www.youtube.com/watch?v=kl4scG38DFs
Summary
• Automated decision making is a key to realizing smart robotic assistants
– Need Physics-Aware AI
– Application context is important
• Smart robotic assistants enable
– Rapid reconfigurability
– Improved productivity of human operators
– Enhanced process efficiency
• Demonstrated feasibility in several applications
─ Robotic Finishing
─ Collaborative Assembly
─ Automated Composite Layup
─ Robot Assisted Additive Manufacturing
─ Machine Tending Using Mobile
Manipulators
─ Robotic Bin Picking and Kitting
Contributors
SK Gupta Rohil Aggarwal Abdullah Alsharhan Vivek Annem Prahar Bhatt Dr. Timotei Centea Jingsong Chu
Liwei Fang Ariyan Kabir Dr. Krishna Kaipa Alec Kanyuck Anirudh Kulkarni Nithyanand Kumbla Dr. Josh Langsfeld
Keng-Yu Lin Binwei Liu Rishi Malhan Dr. Carlos Morato Siddhant Nadkarni Pradeep Rajendran Dr. Brual Shah
Aniruddha Shembekar Shantanu Thakar Dennis Wang Yeo Jung Yoon Yash Shahapurkar
• S. Shriyam and S.K. Gupta. Modeling and analysis of subsystem interactions in robotic assembly. ASME Computers and
Information in Engineering Conference, Anaheim, CA, August 2019.
• Y.J. Yoon, M. Yon, S.E. Jung, and S.K. Gupta. Development of three-nozzle extrusion system for conformal multi-resolution
3D printing with a robotic manipulator. ASME Computers and Information in Engineering Conference, Anaheim, CA, August
2019.
• A. V. Shembekar, Y. J. Yoon, A. Kanyuck and S.K. Gupta. Generating robot trajectories for conformal 3D printing using non-
planar layers. ASME Journal of Computing and Information Science in Engineering, Accepted for publication.
• P. M. Bhatt, A. M. Kabir, R. K. Malhan, B. C. Shah, A. V. Shembekar, Y. J. Yoon, and S.K. Gupta. A robotic cell for multi-
resolution additive manufacturing. IEEE International Conference on Robotics and Automation, Montreal, Canada, May 2019.
• S. Thakar, V. Annem, A. Kabir, P. Rajendran, and S. K. Gupta. Accounting for part pose estimation uncertainties during
trajectory generation for part pick-up using mobile manipulators. IEEE International Conference on Robotics and Automation,
Montreal, Canada, May 2019.
• R. K. Malhan, A. M. Kabir, B. C. Shah, and S. K. Gupta. Identifying feasible workpiece placement with respect to redundant
manipulator for complex manufacturing tasks. IEEE International Conference on Robotics and Automation, Montreal,
Canada, May 2019.
• A. M. Kabir, A. Kanyuck, R. K. Malhan, A. V. Shembekar, S. Thakar, B. C. Shah, and S. K. Gupta. Generation of
synchronized configuration space trajectories of multi-robot systems. IEEE International Conference on Robotics and
Automation, Montreal, Canada, May 2019.
• P. M. Bhatt, P. Rajendran, K. McKay, and S. K. Gupta. Context-dependent compensation scheme to reduce trajectory
execution errors for industrial manipulators. IEEE International Conference on Robotics and Automation, Montreal, Canada,
May 2019.
• P.M. Bhatt, R.K. Malhan, and S.K. Gupta. Computational foundations for using three degrees of freedom build platforms to
enable supportless extrusion-based additive manufacturing. ASME Manufacturing Science and Engineering Conference,
Erie, PA, June 2019.
• R.K. Malhan, A.M. Kabir, B. Shah, T. Centea, and S.K. Gupta. Determining feasible robot placements in robotic cells for
composite prepreg sheet layup. ASME Manufacturing Science and Engineering Conference, Erie, PA, June 2019.
• V. Annem, P. Rajendran, S. Thakar, and S.K. Gupta. Towards remote teleoperation of a semi-autonomous mobile manipulator
system in machine tending tasks. ASME Manufacturing Science and Engineering Conference, Erie, PA, June 2019.
References
• J.D. Langsfeld, A.M. Kabir, K.N. Kaipa, and S.K. Gupta. Integration of planning and deformation model
estimation for robotic cleaning of elastically deformable objects. IEEE Robotics and Automation Letters,
3(1): 352-359, 2018.
• K.N. Kaipa, C.W. Morato, and S.K. Gupta. Design of hybrid cells to facilitate safe and efficient human-robot
collaboration during assembly operations. ASME Journal of Computing and Information Science in
Engineering, 18(3):031004, 2018.
• A.M. Kabir, J.D. Langsfeld, K.N. Kaipa, and S.K. Gupta. Identifying optimal trajectory parameters in robotic
finishing operations using minimum number of physical experiments. Integrated Computer-Aided
Engineering Journal, 25(2):111-135, 2018.
• N.B. Kumbla, S. Thakar, K. N. Kaipa, J.A. Marvel, and S.K. Gupta. Handling perception uncertainty in
simulation based singulation planning in robotic bin picking. ASME Journal of Computing and Information
Science in Engineering, 18(2):021004-021004-10, 2018.
• S. Thakar, L. Fang, B.C. Shah, and S.K. Gupta. Towards time-optimal trajectory planning for pick-and-
transport operation with a mobile manipulator. IEEE International Conference on Automation Science and
Engineering, Munich, Germany, Aug 2018.
• N.B. Kumbla, J.A. Marvel, and S.K. Gupta. Enabling fixtureless assemblies in human-robot collaborative
workcells by reducing uncertainty in the part pose estimate. IEEE International Conference on Automation
Science and Engineering, Munich, Germany, Aug 2018.
• A.M. Kabir, B.C. Shah, and S.K. Gupta. Trajectory Planning for manipulators operating in confined
workspaces. IEEE International Conference on Automation Science and Engineering, Munich, Germany,
Aug 2018.
References (Cont.)
• R. K. Malhan, A. M. Kabir, A. V. Shembekar, B. C. Shah, T. Centea, and S.K. Gupta. Hybrid cells for multi-layer prepreg composite sheet layup. IEEE International Conference on Automation Science and Engineering, Munich, Germany, Aug 2018.
• A.V. Shembekar, Y. J. Yoon, A. Kanyuck and S.K. Gupta. Trajectory planning for conformal 3d printing using non-planar layers. ASME Computers and Information in Engineering Conference, Montreal, Quebec City, Canada, August 2018.
• P.M. Bhatt, M. Peralta, H.A. Bruck, and S.K. Gupta. Robot assisted additive manufacturing of thin multifunctional structures. ASME Manufacturing Science and Engineering Conference, College Station, TX, June 2018.
• A.M. Kabir, A.V. Shembekar, R.K. Malhan, R.S. Aggarwal, J.D. Langsfeld, B.C. Shah, and S.K. Gupta. Robotic finishing of interior regions of geometrically complex parts. ASME Manufacturing Science and Engineering Conference, College Station, TX, June 2018.
• R.K. Malhan, Y. Shahapurkar, A.M. Kabir, B.C. Shah, and S.K. Gupta. Integrating impedance control and learning based search scheme for robotic assemblies under uncertainty. ASME Manufacturing Science and Engineering Conference, College Station, TX, June 2018.
• R.K. Malhan, A.M. Kabir, B. Shah, T. Centea, S.K. Gupta. Automated prepreg sheet placement using collaborative robotics. SAMPE, Long Beach, CA, May 2018.
• A.M. Kabir, K.N. Kaipa, J. Marvel, S.K. Gupta. Automated planning for robotic cleaning using multiple setups and oscillatory tool motions. IEEE Transaction on Automation Science and Engineering, 14(3):1364-1377, July 2017.
• C.W. Morato, K.N. Kaipa, and S,K. Gupta. System state monitoring to facilitate safe and efficient human-robot collaboration in hybrid assembly cells. ASME Computers and Information in Engineering Conference, Cleveland, OH, August 2017.
References (Cont.)
• N. Bhatt, S. Thakar, K.N. Kaipa, J. Marvel, and S.K. Gupta. Simulation-based on-line evaluation of singulation plans to handle perception uncertainty in robotic bin-picking. ASME Manufacturing Science and Engineering Conference, Los Angeles, CA, USA, June 2017.
• A. Alsharhan, T. Centea, and S.K. Gupta. Enhancing the mechanical properties of thin-walled structures using non-planar extrusion-based additive manufacturing. ASME Manufacturing Science and Engineering Conference, Los Angeles, CA, USA, June 2017.
• A.M. Kabir, J.D. Langsfeld, C. Zhuang, K.N. Kaipa, and S.K. Gupta. A systematic approach for minimizing physical experiments to identify optimal trajectory parameters for robots. IEEE International Conference on Robotics and Automation, Singapore, May 2017.
• K.N. Kaipa, C. Morato, J. Liu, and S.K. Gupta. Towards automated generation of multimodal assembly instructions for human operators. International Conference on Systems Engineering Research, Redondo Beach, CA, March 2017.
• K.N. Kaipa, A.S. Kankanhalli-Nagendra, N.B. Kumbla, S. Shriyam, S. S. Thevendria-Karthic, J.A. Marvel, and S.K. Gupta. Addressing perception uncertainty induced failure modes in robotic bin-picking. Robotics and Computer Integrated Manufacturing, 42(12):17-38, 2016.
• A.M. Kabir, J.D. Langsfeld, S. Shriyam, V. Rachakonda, C. Zhaung, K.N. Kaipa, J. Marvel, and S.K. Gupta. Planning algorithms for multi-setup multi-pass robotic cleaning with oscillatory moving tools. IEEE International Conference on Automation Science and Engineering, Fort Worth, TX, August, 2016.
• J.D. Langsfeld, A.M. Kabir, K.N. Kaipa, and S.K. Gupta. Robotic bimanual cleaning of deformable objects with online learning of part and tool models. IEEE International Conference on Automation Science and Engineering, Fort Worth, TX, August 2016.
References (Cont.)
• K. N. Kaipa, A. S. Kankanhalli-Nagendra, N. B. Kumbla, S. Shriyam, S. S. Thevendria-Karthic, J. A. Marvel, and S. K. Gupta. Enhancing robotic unstructured bin-picking performance by enabling remote human interventions in challenging perception scenarios. IEEE International Conference on Automation Science and Engineering, Fort Worth, TX, August 2016.
• J.D. Langsfeld, A. M. Kabir, K.N. Kaipa, and S.K. Gupta. Online learning of part deformation models in robotic cleaning. ASME Manufacturing Science and Engineering Conference, Blacksburg, USA, June 2016.
• K.N. Kaipa, S. Shriyam, N. Kumbla, and S.K. Gupta. Resolving occlusions through simple extraction motions in robotic bin picking. ASME Manufacturing Science and Engineering Conference, Blacksburg, USA, June 2016.
• A.M. Kabir, J.D. Langsfeld, C. Zhuang, K.N. Kaipa, and S.K. Gupta. Automated leaning of operation parameters for robotic cleaning by mechanical scrubbing. ASME Manufacturing Science and Engineering Conference, Blacksburg, USA, June 2016.
References (Cont.)
Top Related