Simulation of Visual Servo Control for Robotic Harvesting of Fruits (In V-REP and ROS)

Redmond Ramin Shamshiri, PhD

Contact Info 1

https://florida.academia.edu/RaminShamshiri

Outline

Visual Servo Control

Simulation approach to robotic harvesting

Research & Developments in Robotic Harvesting

Major Challenges, Variability and occluded fruits

Thinking outside the box

Manipulator mechanism?

Manipulator of choice?

Soft Robotic Grippers/Manipulators,

Lower-cost cameras, robust image processing

Efficient harvesting time

Optimization

Hybrid approach

Conceptual design (Arrays of Actuators Mechanism)

Virtual harvesting approach

Simulation Software

V-REP, Quick Intro

First step: Inverse kinematics

Inverse Kinematics in V-REP,

Simulated Sensors (RGB, Kinect, Proximity, Laser)

Simulation environment and scene objects (V-REP/ROS setup)

Calculation modules

Scan experiments (Plant/fruit scan with manual joint control)

Scan experiments (Plants scan)

Scan Experiments (Automated plant scan in x-y plane)

Scan Experiments (Automated plant scan in x-y-z space)

Simulation of visual servo control (Localization and tracking)

Experiments with path planning

Experiments with depth sensors (Microsoft Kinect)

Experiments with range finder sensors (Hokuyo URG-04LX-UG01)

Experiments with 3D laser scanner sensors (3D Laser scanner fast rangefinder)

Conclusion (Market trend)

Acknowledgement

2 / 47.

© Redmond Ramin Shamshiri, Simulation of Robotics Fruit Harvesting

Visual Servo Control Objective

Objective: A robot end-mounted camera is used to position the robot arm in a plane orthogonal to the axis,

such that the fruit to be harvested is centered in the camera’s field of view.

Control

law

Inverse

Kinematics

Feature extraction Pose determination

Joint

Controllers

Power amplifiers

+

-

𝑪𝒙𝒅

𝑪𝒙

Position-based eye-in-hand visual servo structure

Image processing Feature extraction Pose determination

Inverse kinematics Hardware setup

Reference values CONTROLLER

The system has no trajectory generator, instead a feedback loop, closed visually, is

used to control the robots arm position.

(Two control laws were simulated: join angles and joint velocity)

Input Output

3 / 47.


Simulation approach to robotic harvesting Problem statement

Difficulties and hassles of experiments with actual hardware setup in robotic harvesting Time constraints

Unavailability of equipment (i.e., sensors, cameras, and the robot manipulator)

Experiment costs

Some hardware setups may result in actuator saturation

Experiments creating unsafe situation for the operators and for the plants system

Detection rates and visual servo control need to be improved

Sensing from the gripper needs to be performed from multiple viewpoints

Virtual environment

simulating various combinations of plant/fruit/robot/sensors scenarios

4 / 47.


Research & Developments in Robotic Harvesting Sweet pepper harvester, CROPS and SWEEPER (Wageningen UR)

Courtesy of CROPS http://www.crops-robots.eu/

Courtesy of Wageningen UR and http://sweeper-robot.eu/

Source: https://www.wageningenur.nl/en/newsarticle/New-Sweet-pepper-harvesting-robot-Sweeper-ready-for-testing-in-commercial-greenhouse-.htm

5 / 47.


http://www.crops-robots.eu/



http://sweeper-robot.eu/



https://www.wageningenur.nl/en/newsarticle/New-Sweet-pepper-harvesting-robot-Sweeper-ready-for-testing-in-commercial-greenhouse-.htm

























Research & Developments in Robotic Harvesting Citrus harvester

Courtesy of Univ of Florida. Image source: http://ncr.mae.ufl.edu/index.php?id=research/citrus_harvesting

Courtesy of Energid Citrus Picking System Source: https://sites.google.com/site/cvhanaian/research

6 / 47.


http://ncr.mae.ufl.edu/index.php?id=research/citrus_harvesting

http://ncr.mae.ufl.edu/index.php?id=research/citrus_harvesting

https://sites.google.com/site/cvhanaian/research


Research & Developments in Robotic Harvesting Strawberry harvester

http://www.techradar.com/news/world-of-tech/future-tech/colour-

smart-robot-has-eye-for-strawberries-323902

Strawberry harvester. Courtesy Agricultural machinery maker Shibuya Seiki and the National Agriculture and Food

Research Organization

The arm of Robotic Harvesting's machine picking strawberries. http://mathscinet.ru/robocenter/

7 / 47.


http://www.techradar.com/news/world-of-tech/future-tech/colour-smart-robot-has-eye-for-strawberries-323902






















http://mathscinet.ru/robocenter/

http://mathscinet.ru/robocenter/

Research & Developments in Robotic Harvesting Strawberry harvester (Wageningen UR)

Courtesy of Wageningen UR. Source:

https://www.wageningenur.nl/en/show/Cucumber-harvesting-robot.htm

8 / 47.








Research & Developments in Robotic Harvesting Other platforms

The French grape-picking robot

Source: 6-DOF Robotic Arm Páprika Chili-Pepper

Harvester. Courtesy of DIROSE SAC https://www.festo.com/cms/en_corp/14014.htm

http://www.producebusinessuk.com/services/stories/2016

/08/23/robot-harvesting-and-weeding-well-on-the-way-

into-mainstream

MIT Robot Gardener

Berry nice: harvest ripe berries

MIT-CSAIL-DRL-Distributed Robotics Garden.

Photo used with permission from Nikolaus Correll.

Agrobot SW6010

9 / 47.


http://www.dailymail.co.uk/sciencetech/article-2209975/Meet-Wall-Ye-The-French-grape-picking-robot-work-day-night--vineyard-workers-job.html

https://www.festo.com/cms/en_corp/14014.htm

https://www.festo.com/cms/en_corp/14014.htm

http://www.producebusinessuk.com/services/stories/2016/08/23/robot-harvesting-and-weeding-well-on-the-way-into-mainstream





















http://www.newscientist.com/gallery/farm-robots/3

http://www.shibuya-sss.co.jp/

http://www.agrobot.com/products.html

http://www.agrobot.com/products.html

Major Challenges, Variability and occluded fruits Variability in Environment, Fruit color and Orientation

10 / 47.


Thinking outside the box

Eliminating the redundant vision feedbacks and mission time

Linearizing the problem

Creating Sub-Problems (smaller task)

Manual fruit detection, Automated harvesting

Re-Thinking the Manipulator

Re-Thinking the Gripper

Re-Thinking the image-processing

Re-Thinking the entire strategy

11 / 47.


Manipulator mechanism? How many degrees of Freedom?

12 / 47.


Manipulator of choice? Purchase or custom-build?

UR 5 KUKA LBR iiwa 14 R 820

KUKA LBR iiwa 7 R 800

UR 10

PhantomXPincher

Baxter

Jaco Arm

Mico Arm

FANUC LR Mate 200iD

KUKA LBR 4

ABB IRB 360 Adept Quattro 650HS

13 / 47.


Soft Robotic Grippers/Manipulators Are they promising?

14 / 47.


Soft Robotic Grippers/Manipulators Are they promising?

15 / 47.


Lower-cost cameras, robust image processing Fruit detection and tracking

Red Threshold Low=75 Red Threshold Low=60 Red Threshold Low=50

16 / 47.



Original image (left) and edge detection with fuzzy-logic

image (right) for a typical red-pepper fruit Demonstration of red-pepper fruit detection in MATLAB, (i), Red, Green and Blue bands, (ii): Mask of only

red object, is-red mask, (iii): Border smoothed, regions filled, masked red image, (iv): Masked image

showing only red detected object

R G

B

(i)

(ii)

(iii)

(iv)

(i)

(ii)

(iii)

(iv)

Demonstration of red-pepper tracking, (i):original image, (ii):extracting red component from

grayscale image and median filter to filter out the noise, (iii): Convert the resulting grayscale

image into a binary image and removing pixels less than 3000 pixels, (iv): blob analysis,

bounding the red objects in rectangular box and showing centroid

17 / 47.



Video Demo Real-time, robust and rapid red-pepper fruit

detection with Matlab © Redmond Ramin Shamshiri:

https://youtu.be/rFV6Y5ivLF8

18 / 47.


https://youtu.be/rFV6Y5ivLF8

https://www.researchgate.net/deref/https://youtu.be/rFV6Y5ivLF8

Efficient harvesting time Path planning, Shortest time

X Y

Z

19

Desired Time: 4 to 6 sec!!

19 / 47.


X Y

Z

Efficient harvesting time Path planning, maximum visibility

20

65% Visible

85% Visible

98% Visible

What about hand-

in-eye approach?

20 / 47.


Optimization 21 / 47.


Hybrid approach Simplified VSC, lower cost manipulators/cameras, lesser control effort

Camera View Camera View

20,000$ each

500$, each

22 / 47.


23 Contact: [email protected]

Sweeper Project, Redmond Sh, Presentation, SEP 2015

23 / 47.


Hybrid approach, Linear manipulators, suction gripper

Po

ints

of

eff

ec

t

Suction Gripper

Courtesy of Jochen Hemming, CROPS

24 / 47.


Z –Displacement is Controlled

by Laser, or Simple

Impact/Force Sensor

Z

Y

X

Hybrid approach, Arrays of actuator mechanism

Mesa Swissranger SR4000

Time of fly camera AVT Prosilica GC2450

Colour camera

25 / 47.


Conceptual design Arrays of Actuators Mechanism

26 / 47.


More conceptual design

Image source: http://www.freshplaza.com/2012/0816/paprikarobot.jpg

Image: Courtesy of Vision Robotics. http://www.wired.com/2007/06/robo-picker/

Courtesy of Hanaian. https://sites.google.com/site/cvhanaian/research

Vision Robotics.

27 / 47.


http://www.freshplaza.com/2012/0816/paprikarobot.jpg



http://www.wired.com/2007/06/robo-picker/








Virtual harvesting approach

Virtual Greenhouse

Virtual Plant / Fruit

Manipulator model

simulation can provide a cheap and safe experiment platform with a faster approach

for development, testing and validating control strategies and algorithms.

28 / 47.


Simulation Software

1. Robot Virtual Worlds

2. RoboDK

3. Microsoft Robotics

Developer Studio

4. Webots

5. Workspace

6. V-REP

7. LabVIEW

8. OpenHRP3

9. Player

10. Simbad

11. Algodoo

12. RoboWorks

13. RobotStudio

14. Gazebo

15. Actin Simulation

16. Blender

17. WorkcellSimulator

18. robotSim:Edu

19. 3DSimulate

20. Roboguide

21. RoboLogix

22. Ezphysics

23. SimRobot

29 / 47.


V-REP, Quick Intro

First public release in March 2010, V-REP V2.4.2

Latest version: V-REP V3.3.2 (September 2016)

Has big functionality (the system is developed since March, 2010)

Completely open-source (it is uploaded publicly in 2013)

Cross-platform — windows, mac, linux (works at Qt)

Has API & libraries for work with robots through C/C ++, Python, Java, Lua, Matlab, Octave or Urbi

Free for non-commercial use!

30 / 47.


First step: Inverse kinematics IK recipe in V-REP

1) Importing CAD file (i.e., from SolidWork)

2) Defining Manipulator Links

3) Adding Joints

4) Associating Joints and Links

5) Creating Kinematic chains, described with a "tip" dummy and a "base" object

6) Adding A "target" dummy that the "tip" dummy will be constrained to follow

7) Registering the Inverse Kinematic task and do the adjustments, settings, etc…

Note 1: V-REP can open .STL CAD files, if the CAD file from SolidWork has other extension (i.e., .SLDASM), and you do not

have SolidWork installed, you can use the free software “eDrawings” to do the conversion.

Note 2: Defining manipulator links can be quiet tricky, depending on the imported CAD file

Note 3: Adding joints need a good understanding of the manipulator links and position

In V-REP, an IK task requires specification of at least following elements:

• a kinematic chain described with a "tip" dummy and a "base" object.

• A "target" dummy that the "tip" dummy will be constrained to follow.

Attach the Tip-Dummy to the Last Link, select “Tip", then “last Link", then click [Edit --> Make

last selected object parent

31 / 47.


Inverse Kinematics in V-REP Example: 7-DoF Redundant Manipulator

Exploring Sub-Shapes Group relevant Shapes to create Links Joints Added

Building the kinematic chain Define an inverse kinematics task

Video Demo Inverse Kinematics 7 DOF Manipulator, V-REP

©Redmond Ramin Shamshiri

https://youtu.be/mHE_PzD3Y2M

32 / 47.




Inverse Kinematics in V-REP Fanuc - LR Mate-200iD

Video Demo Inverse kinematics Fanuc LR Mate200iD for Robotic Pepper

Harvesting, V-REP, ©Redmond Ramin Shamshiri

https://youtu.be/s6UZWDS_Cn8

33 / 47.





Simulated Sensors RGB, Kinect, Proximity, Laser

Fish-eye RGB Axis 212 PTZ sensor (left, Source) and V-REP perspective vision-sensor (right)

Microsoft Kinect, Fast Hokuyo URG-04LX-UG01‖ scanning Laser Rangefinder, and ―3D Laser

Scanner Fast‖ rangefinder

An example of Infrared Proximity Sensor Long Range - Sharp GP2Y0A02YK0F,

and Bottom: different proximity sensors types in V-REP, respectively from left to right: Ray-type,

pyramid-type, cylinder-type, disk-type and cone-or randomized ray-type proximity sensors

34 / 47.


http://www.gosecurenow.com/upload_picture/picture_product/m2_product-1406051637470023.jpg

Simulation environment and scene objects V-REP/ROS setup

3D laser scanner

Kinect Hokuyo URG-04LX-UG01

Arm camera

RGB vision sensor

Proximity sensor

Fixed Camera

A 180° scanning path

Cameras/sensors floating views and graphs

Plant model

Video Demo V-REP, ROS Env Setup


https://youtu.be/tKagjNQ9FW4

35 / 47.





Calculation modules

Minimum distance calculation collision detection

path planning

Control mechanism

Remote

API

Serial

port

Plugins

ROS

nodes

Control

Mechanisms

Embedded

Child Script

inverse kinematics

36 / 47.


Scan experiments Plant/fruit scan with manual joint control

Fish-eye RGB Axis 212 PTZ sensor (Source)

and V-REP perspective vision-sensor

Video Demo Fruit/Plant Scan Demo


https://youtu.be/6EOy1NesvQg

37 / 47.


http://www.gosecurenow.com/upload_picture/picture_product/m2_product-1406051637470023.jpg







Video Demo Scanning Around/Inside/Outside Red Pepper

Plants with Fanuc LR Mate 200iD ©Redmond Ramin Shamshiri

https://youtu.be/ZstETgfUDLg

Scan experiments Plants scan

38 / 47.





Scan Experiments Automated plant scan in x-y plane

Simulated scan of a red sweet pepper fruit. Camera view at 30 degrees increments around

the fruit model and leaves. The robot moves in the x-y plane for finding best angle of attack

Fruit/plant 360-degree scan experiment in x-y plane, and the right image is a top view of the

scanning setup. The RGB sensor mounted on the robot tip moves on the horizontal plane x-

y, to find the best view of the fruit

39 / 47.


Scan Experiments Automated plant scan in x-y-z space

Scanning around fruit/plant in x-y-z plane, the manipulator is ―twisted‖ to provide different

viewpoints for the end-effector camera

Vertical scan of fruits/plants in x-y-z plane, the end-effector camera is moved and rotated in

both x-z and y-z directions

40 / 47.


Simulation of visual servo control Localization and tracking

Joints position control law (Localization)

1 2 3

4 5 6

Joints velocity control law (Tracking)

Robot and camera with eye-in-hand configuration setup. The RGB camera was mounted on the tip of

the FANUC LR Mate 200iD robot manipulator.

Video Demo #1

Video Demo #2 Visual Servo Control in V-REP for Robotic harvesting of sweet pepper


https://youtu.be/VupoirQOL0Y

https://youtu.be/eCj9qTTeqzs

41 / 47.










Experiments with path planning

Video Demo Path Planning in V-REP with Fanuc LR Mate200iD for robotic harvesting of sweet pepper


https://youtu.be/mjJhDNNaE-4

42 / 47.







Experiments with depth sensors Microsoft Kinect

Depth sensors

Microsoft Kinect

Depth image to work image



43 / 47.


Experiments with range finder sensors Hokuyo URG-04LX-UG01 Rangefinder


Extract coordinates from work image

Work image to output image

Filter used in the ―Fast Hokuyo

URG-04LX-UG01‖ sensor

―Fast Hokuyo URG-04LX-UG01‖ scanning

Laser Rangefinder

―Hokuyo URG-04LX-UG01

44 / 47.


Experiments with 3D laser scanner sensors 3D Laser scanner fast rangefinder

Original depth image to work image

Extract coordinates from work image

Intensity scale work image

Work image to output image

3D Laser Scanner Fast‖ rangefinder

45 / 47.


Conclusion Market trend

Blueberry picking robot could win $250,000 prize!

Main proposed operating specifications include:

size/color sensing for detecting fruit maturity,

careful handling to avoid any damage to the fruit, ability

to directly fill field trays/baskets, 24/7 harvesting

capability, average production of 500 Kg/day, mobility

capabilities on uneven ground or dusty conditions, and

fabrication costs per unit <$500,000 USD

Read more: http://www.naturipechallenge.com/

46 / 47.


http://www.naturipechallenge.com/








(WUR)

(WUR)

(WUR)

(Univ of Florida)

(Univ of Florida)

(Univ of Florida)

(Univ of Florida)

For their insightful suggestions and ideas

47 / 47.


Simulation of Visual Servo Control for Robotic Harvesting of Fruits (In V-REP and ROS)

Education

Transcript of Simulation of Visual Servo Control for Robotic Harvesting of Fruits (In V-REP and ROS)