SAPIEN - slides.games-cn.org · SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang1...
86
SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 Yuzhe Qin 1 Kaichun Mo 2 Yikuan Xia 1 Hao Zhu 1 Fangchen Liu 1 Minghua Liu 1 Hanxiao Jiang 3 Yifu Yuan 5 He Wang 2 Li Yi 4 Angel X. Chang 3 Leonidas J. Guibas 2 Hao Su 1 1 UC San Diego 2 Stanford University 3 Simon Fraser University 4 Google Research 5 UC Los Angeles
Transcript of SAPIEN - slides.games-cn.org · SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang1...
SAPIENA SimulAted Part-based Interactive ENvironment
Fanbo Xiang1 Yuzhe Qin1 Kaichun Mo2 Yikuan Xia1 Hao Zhu1 Fangchen Liu1 Minghua Liu1 Hanxiao Jiang3 Yifu Yuan5 He Wang2 Li Yi4
Angel X. Chang3 Leonidas J. Guibas2 Hao Su1
1UC San Diego 2Stanford University 3Simon Fraser University 4Google Research 5UC Los Angeles
About Me
UIUCBS CS, BS MathUCSD: MS CS
Fanbo Xiang, UCSD
Advisor: Hao Su, UCSD
Research: Graphics, Vision, Robotics
Outline
● Background○ Intelligence, vision and robotics○ Simulated environment
● SAPIEN architecture○ Physics and robotics○ Renderer○ Assets
● Future research problems
Intelligent Agent
Intelligent Agent
Perceive information
Intelligent Agent
Perceive information
Adapt behavior
Perceive information
Adapt behavior
Perceive information
Adapt behavior
Computer vision
Perceive information
Adapt behavior
Computer vision
Robotics control
Perceive information
Adapt behavior
Computer vision
Robotics control
Perceive information
Adapt behavior
Computer vision
Robotics control
Planning?Active perception?
Data driven approaches
Learn from data?
Real-world data collection
Reinforcement Learning?Manual data collection?
Reinforcement Learning
RL Agent
Observation
RL Agent
Observation
RL Agent
Perception
Observation
RL Agent
Perception
Action
Observation
RL Agent
Perception
Control
Action
Observation
RL Agent
Perception
Control
Action
Reward
State update
RL Problems● Expensive hardware● Slow data collection● Low sample efficiency● Overfit to specific agent
Imitation Learning (Manual Data Collection)
Observation
Human
Perception
Control
Action
State update
Observation
Human
Perception
Control
Action
State update IL Agent
Training
IL Problems● Unintuitive control● May not be optimal● Human expertise
Simulated Environment
Simulated Environment
Manual data collection● More freedom in controller
design● Less expertise required
RL● Will not break hardware● Easy to scale
Simulated Environment
Simulated Environment
Real World
?
Physics
Simulated Environment
Physics Robotics
Simulated Environment
Physics Robotics Rendering
Simulated Environment
Physics Robotics Rendering Simulation Content
Simulated Environment
SAPIENPhysics Robotics Rendering Simulation Content
SAPIENAsset
SAPIEN Renderer
SAPIENEngine
Physics Robotics Rendering Simulation Content
SAPIEN Engine
SAPIEN Engine
SAPIEN Engine
PhysX Physical Simulator
SAPIEN Engine
SAPIEN Engine
PhysX Physical Simulator
World Interface
Articulation Interface
SAPIEN Engine
SAPIEN Engine
PhysX Physical Simulator
World Interface
Articulation Interface
ROS Interface
Sensor Interface
Controller Interface
SAPIEN Engine
SAPIEN Engine
PhysX Physical Simulator
World Interface
Articulation Interface
Sensor Interface
Controller Interface
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
Client API
ROS Interface
SAPIEN Engine
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
Client API
SAPIEN Engine
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
Client API
Reinforcement Learning
SAPIEN Engine
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
Client API
Reinforcement Learning Robot Tasks
SAPIEN Renderer
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
Client API
SAPIEN Engine
PhysX Physical Simulator
World Interface
Articulation Interface
Sensor Interface
Controller Interface
ROS Interface
SAPIEN Renderer
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
Client API
SAPIEN Engine
PhysX Physical Simulator
World Interface
Articulation Interface
Sensor Interface
Controller Interface
ROS Interface
SAPIEN Renderer
SAPIEN Renderer
SAPIEN Renderer
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
Client API
SAPIEN Engine
PhysX Physical Simulator
World Interface
Articulation Interface
Sensor Interface
Controller Interface
Renderer Interface
ROS Interface
SAPIEN Engine
ROS Interface
SAPIEN Renderer
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
Client API
SAPIEN Renderer
PhysX Physical Simulator
World Interface
Articulation Interface
Sensor Interface
Controller Interface
Renderer Interface
Rendered Images
SAPIEN Renderer
SAPIEN Renderer
GLSL Shaders
Renderer Interface
Normal
RGBD
Segmentation
SAPIEN Renderer
SAPIEN Renderer
OptiX ShadersRay Tracing
GLSL Shaders
Renderer Interface
Normal
RGBD
Segmentation
SAPIEN Renderer
SAPIEN Renderer
OptiX ShadersRay Tracing
GLSL Shaders
Renderer Interface
Normal
RGBD
Segmentation
4 spp, 3 bounce, 512x512OptiX denoise, < 20 FPS
CustomizableRenderer/Visualizer
SAPIEN Renderer
SAPIEN Renderer
OptiX ShadersRay Tracing
GLSL Shaders
Renderer Interface
Normal
RGBD
Segmentation
CustomizableRenderer/Visualizer
SAPIEN Renderer
SAPIEN Renderer
OptiX ShadersRay Tracing
GLSL Shaders
Renderer Interface
Normal
RGBD
Segmentation
Considerations● Needs to run at real time (~100 FPS)○ Rasterizer○ Ray tracer denoise? (1spp max)○ GPU-CPU transfer? (>10 ms)○ Prebaking? (interaction)
SAPIEN Renderer
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
CustomizableRenderer/Visualizer
Client API
SAPIEN Engine
ROS Interface
PhysX Physical Simulator
World Interface
Articulation Interface
Sensor Interface
Controller Interface
CustomizableRenderer/Visualizer
SAPIEN Renderer
OptiX ShadersRay Tracing
GLSL Shaders
Renderer Interface
Normal
RGBD
Segmentation
SAPIEN Asset
PartNet-Mobility Dataset
SAPIEN Asset
ShapeNet
PartNetMotion Annotation2K models, 14K parts
PartNet-Mobility Annotation
Shape cleaning (ShapeNet, 3D Warehouse)
Motion annotation(PartNet-Mobility)
Part segmentation (PartNet)
Training Testing
Generalize
SAPIEN Asset
SAPIEN Asset
PartNet-Mobility Dataset
SAPIEN Asset
SAPIEN Asset
PartNet-Mobility Dataset
Robot Model
SAPIEN Asset
SAPIEN Asset
PartNet-Mobility Dataset
Robot Model
Object Layout
SAPIEN Asset
Robot/Scene Builder
Standard URDF format
Python API
SAPIEN Asset
PartNet-Mobility Dataset
Robot Model
Object Layout
SAPIEN
3D/IMUSensor
Force/Joint/VelocityController
TrajectoryController
Inverse KinematicsMotion Planning
CustomizableRenderer/Visualizer
SAPIEN Engine
ROS Interface
PhysX Physical Simulator
World Interface
Articulation Interface
Sensor Interface
Controller Interface
CustomizableRenderer/Visualizer
SAPIEN Renderer
OptiX ShadersRay Tracing
GLSL Shaders
Renderer Interface
Normal
RGBD
Segmentation
SAPIEN Asset
PartNet-Mobility Dataset
Robot Model
Object Layout
Client API
Robot/Scene Builder
Task Demonstrations
Task DemonstrationsMovable PartSegmentation
Task DemonstrationsMovable PartSegmentation
Motion ParameterEstimation
Task DemonstrationsMovable PartSegmentation
Motion ParameterEstimation
Part Manipulation
Task DemonstrationsMovable PartSegmentation
Motion ParameterEstimation
Long-horizonPlanning Part Manipulation
Task Demonstrations
Movable PartSegmentation
● Standard vision problem: detection and segmentation.
Task Demonstrations
Motion ParameterEstimation
● New vision task
● Important for control tasks○ Given the motion parameters, we can
use control methods to manipulate the parts
Task Demonstrations
Part Manipulation● Manipulation and control○ Reinforcement learning○ Imitation learning○ robotics control
Task Demonstrations
● Manipulation and control○ Reinforcement learning○ Imitation learning○ robotics control
Task Demonstrations
Long-horizonPlanning ● Planning
○ Achieve meaningful tasks○ “Home assistant”
Future Plans/Applications• Benchmark (SAPIEN Challenge)
• Vision tasks
• Manipulation tasks
• Education Platform
• Control Algorithms
• Robot Learning
Demos
Demos
Summary
• SAPIEN provides simulation and rendering for vision and manipulation.
• SAPIEN hosts a large-scale dataset for articulated objects.
• SAPIEN is open source.
• pip install sapienhttps://sapien.ucsd.edu
Requirements: Python 3, Linux / Latest MacOS
Thank you
• Questions?
https://sapien.ucsd.edu
Requirements: Python 3, Linux / Latest MacOS
