Hierarchical Robotic Crane System

for Post Grid Array Environments

Ted Shaneyfelt

Computer Science and Engineering Department

University of Hawaii at Hilo

Hilo, HI, 96720 USA t.s@ieee.org

Mo Jamshidi and Sos Agaian

Electrical Engineering Department

The University of Texas at San Antonio

San Antonio, TX, 78249 U.S.A. moj@wacong.org

Abstract – A robotic system of crane systems is proposed and

simulated for a vanilla pollination application. The system

could be used for other applications where a grid of posts is

serviced in an area suitable for overhead cranes. Existing

crane systems could work together with new crane equipment

in a hierarchical manner. A docking crane system with two or

more six degrees of freedom cranes services the area around

posts upon which it docks. Vision is used for subject

recognition and control feedback.

Keywords: Robotics, vision, agriculture, crane, simulation.

1 Introduction

Vanilla is a climbing vine that may be trained to grow on

poles in a greenhouse environment. The costly manual tasks

such as hand pollination could be replaced by a robotic

system[1]–[3]. This paper deals with the design of a robotic

system of cranes capable of maneuvering about an array of

poles for that purpose. It could also be used for similar work

in other areas where a gantry crane could be suspended above

a similar post grid array. Possible applications range from

cultivating post-climbing-vines in general to industrial factory

environments. Underwater applications are also possible if

submersible components are used.

Pollination of vanilla requires visual feedback, as tactile

feedback would damage the flower, and as the flower must be

located to be pollinated. Because a vanilla flower opens only

on one day, testing of the system on real plants is deferred

until plants are grown and blooming in a laboratory

environment. Meanwhile, computer simulations have been

used for preliminary testing of autonomous vision feedback

and control.

JReality [4] is a scene graph API. It is developed and

maintained Technische Universität Berlin and the City

College of the City University of New York. Robotic

Toolbox is a free software package for MATLAB.[5]–[7]. It is

developed and maintained by P. Corke.

This paper first provides horticultural background in

Section II. Then in Section III, a robotic system of cranes for

pollination is proposed which is flexible enough to utilize

existing gantry cranes in its design. Section IV shows

simulation results, and Section V is the conclusion.

Figure 1 Automated pollination simulation in progress.

2 Horticultural Background

2.1 Anatomy of the Vanilla Planifolia Flower

A diagram in Figure 2 shows the anatomy of the vanilla

flower. The petals and sepals are useful for feature recognition

from a distance. The labellum is useful for guiding the end

effector as it transfers pollinia from the male anther to the

female stigma. In nature, it provides a landing platform for the

melapona bee, which is the only natural pollinator of vanilla

planifolia.

2.2 Growing Environment

Though vanilla has been long grown outside the United

States, Kadooka is largely responsible for the research leading

up to the commercialization of Hawaiian vanilla.

Reddekopp’s family-operated Hawaiian Vanilla Company

was the first to commercialize Hawaiian vanilla [8]. Vanilla

vines at the Hawaiian Vanilla Company are trained to climb This work was, in part, supported by Lutcher Brown Endowed Chair and ACE Laboratory.

978-1-4673-5597-1/13/$31.00 ©2013 IEEE

Proc. of the 2013 8th International Conference on System of Systems Engineering, Maui, Hawaii, USA - June 2-6, 2013

82

vertical pipe posts that are galvanized with a zinc coating. The

zinc acts as an important nutrient in addition to supporting the

vine.

Figure 2 Vanilla flower cross section (image courtesy Bruno

Navez)

Greenhouse gantry crane systems already in place may

be adapted for use in the system of crane systems, provided

that they meet the requirements of hoisting and positioning a

docking crane module that weighs approximately 10 pounds

with precision of about one half of an inch.

Figure 3 Overall system dagram

2.3 Special Considerations

It is important that vanilla vines not be over pollinated,

as over-pollination leads to weak vines that are susceptible to

serious fungal outbreaks and other diseases. To address this

problem, hand pollinators are careful not to exceed a limit set

by the horticulturalist of pollinations per raceme. In a

computerized system, autonomous pollination would include

recordkeeping to ensure that the number of pollinations per

raceme does not exceed this limit [9], [10].

3 Robotic System of Cranes

The system of cranes overall design is presented in

Figure 3. In this system, elements of the system of cranes fall

into the categories of mechanical elements, and non-

mechanical software related elements. The system is used for

interacting with a vanilla plant, or in general with some

subject being examined and manipulated.

3.1 Mechanical Elements

The mechanical / hardware elements are designed

around a hierarchy of three basic physical requirements:

Inspecting and manipulating objects along the side of a pipe post.

Covering all sides of the pipe post

Covering all pipe posts in the grid array

The robotic crane design is patterned after a pair of

adapted NIST RoboCranes®[11], [12] with their six degrees

of freedom design. The RoboCrane® concept is extended by

balancing a set of cranes on opposite sides of (or equally

spaced around) the pipe post support using a novel balanced

docking support system unit. These are scaled down and the

winches are relocated to above the load. This crane unit

design also adds a slewing capability for working around the

post on all sides. Finally, it is extended with the ability to be

un-docked from one post and docked atop another post by

interfacing with a separate gantry crane system.

Additional hardware is included for visual sensing, range

finding, manipulation, feedback, user interface, computation,

and data storage and retrieval.

3.2 Software Related Elements

Governing all other systems is the System Controller and

Command Center. This system is attached to a user interface

for operator interaction, and to a database for storage and

retrieval. This data is required, for example, to prevent the

over-pollination of racemes, and to keep a record of post pipe

locations that need to be inspected. The anti-sway gantry

controller uses input shaping to minimize sway when moving

from post to post. This is possible because a gantry crane is

capable of moving in straight lines with controlled

acceleration. Sway control is important in this element and not

others because the gantry crane is a single rope crane, which

is highly susceptible to uncontrolled sway due to the inherent

lack of rigidity in single rope crane systems. The slewing

controller is the simplest of all, only needing to make two

rotations of 60 degrees each per pipe post, each time waiting

for both robotic cranes on its support structure to complete

servicing their work area before advancing 60 degrees to the

next sextant-pair. It is also possible to modify the design to

support three robotic cranes on the structure. In that case, the

slewing unit makes only a single 60 degree rotation instead of

SupportStructure

PlatformPlatform

System Controller& Command Center

Controller 1Slewing

Controller

Anti-SwayGantry

Controller

plant

Database

User Interface

GantryCrane

SlewingUnit

Camera 1& Sonar 1

RobotCrane 1

Manipulator1

Controller 2

Camera 1& Sonar 2

RobotCrane 2

Manipulator2

83

two. Each robot crane has its own controller which uses

feedback from the camera and sonar to plan the waypoints and

trajectory for its platform.

The kinematics of the robotic crane is identical to that of

the Stewart Platform after which the RoboCrane® was

patterned. Forward kinematics involve eighth-degree

polynomials, but inverse kinematics require only calculation

of Euclidian distance between the support and connection

points of the platform [13]. For control of the robotic crane,

only inverse kinematics is required. Winch control is given by

(1),

√( ) ( ) ( )

where is the desired angle of rotation of a winch motor, is

the effective winch radius, , and are displacements

from a nominal pose of the rope connection point on the

platform.

Denavit-Hartenburg (DH) parameters of a

serially linked rigid robotic joint and link describe its position

relative to the previous link. Parallel linked robots such as

Stewart Platforms and the six degrees of freedom robotic

cranes used in this system are not traditionally controlled

through DH parameters. Nevertheless, DH parameters were

used by representing an equivalent virtual serially linked robot

that covers the same work area. Then path planning was done

through the serial link model with its inverse and forward

kinematics along with the inverse kinematics for winch

control to achieve the desired end effector position. This

allowed the use of DH-parameter tools designed for serially

linked robots. The link variable is for joint movement.

(

)

(

√

)

(

)

The link variables are azimuth, radial, and

vertical position; yaw, pitch, and roll, of the platform; and

forward movement of the manipulator.

Using this model, the end effector position in relation to

its zero-reference position is:

– ( )

√

( )–

√

4 Simulation

Simulation was done with JReality and Robotic Toolbox

for MATLAB. A custom hypertext transfer protocol server

was created for controlling the JReality simulation, which

responded to requests for images. Headers were used to

communicate additional data about the simulation status, such

as distance sensor readings. This approach both provided

realistic 3D rendering for MATLAB image processing. The

model was based on photographic imagery and texture

mapping. The hypercomplex numbers based image processing

techniques of [1], [2] were adapted to be used in the

simulation. Figure 1 shows automated vanilla pollination in

progress, and Figure 4 shows the same pollination from the

camera perspective of the crane robot. Images from these

points of view were captured and examined to verify that

pollination was done correctly. Data was also recorded for

waypoints along the path and examined to confirm the results

of the simulation.

Figure 4 Automated pollination from camera perspective

Figure 5 shows the path of the robot during pollination

in three dimensions. Vertical positioning is shown in green,

and positioning in cardinal directions in red and blue. Figure 6

shows the winch rope lengths for the six degrees of freedom

robots during the same test run. Each colored line represents

one winch rope. Winch ropes are color coded in Figure 6 to

match the corresponding rope colors shown in Figure 7. The

diagram in Figure 7 is a plot of a Robotic Toolbox for

MATLAB dual SerialLink model based on the DH parameters

of (2) superimposed over a graphic representation of dual

cranes.

84

Figure 5 Path of end effectors through inspection and processing of vanilla orchids around one post pipe

Figure 6 Rope lengths for robotic cranes during test run

85

Examination of the data and image snapshots such as

those in Figure 4 for simulated pollination confirm the

simulation results. In the simulation, all of the flowers on the

post were recognized and the manipulator was activated each

at the labellum of each flower at least once. There were no

occurrences of the robot attempting to manipulate anything

except actual flowers at the work area of the labellum during

the test run.

Figure 7 Dual Robotic Toolbox for MATLAB SerialLink

model plot superimposed over robotic crane model plot

5 Conclusion

A system of cranes was presented for servicing a grid

array of pipe posts, and simulations were performed. The

simulations with vision feedback show promising results for

further development of a robotic crane for the automated

pollination of vanilla. The system could also be used in

similar environments for operating around a grid array of pipe

posts. The future of this research will include vanilla

cultivation in a robotically serviced environment.

Acknowledgment

Special thanks to Jim Reddikopp for access to his vanilla

greenhouse and shadehouse at every request, and for sharing

his experiences, 谢谢張星明教授 (Prof. Zhang, Xing Ming)

and 余志文博士 (Dr. Yu, Zhi Wen) of South China

University of Technology (華南理工大學) at Guangzhou

Higher Education Mega Center (廣州大学城) for their

inspiration and encouragement, and to the developers and

maintainers of JReality software, especially to Charles Gunn

of Institut für Mathematik at Technische Universität Berlin for

his exceptionally prompt responses to all inquiries about

JReality. Support of first author from University of Texas

ACE Laboratory is appreciated.

References

[1] T. Shaneyfelt, S. Agaian, and M. Jamshidi, “Quaternion

based segmentation for vanilla recognition,” in Systems,

Man, and Cybernetics (SMC), 2011 IEEE International

Conference on, Anchorage, Alaska, 2011.

[2] T. Shaneyfelt, S. Agaian, M. Jamshidi, and S. Erdogan,

“Quaternion Number Based Vanilla Recognition System

for Automating Polination,” in Proceedings of the

International Conference on System Science and

Engineering, Macau, China, 2011, pp. 558–563.

[3] T. Shaneyfelt, “Hypercomplex Number Based

Automated Robotic Vanilla Pollination System with

Vision Sensing,” Ph.D. dissertation, The University of

Texas at San Antonio, San Antonio, TX, USA, 2012.

[4] P. Brinkmann, C. Gunn, T. Hoffmann, H. Pietsch, M.

Schmies, and S. Wei\ssmann, “jReality—a thread-safe

Java scene graph for mathematics,” preprint, pp. 1011–

1018, 2006.

[5] P. Corke, Robotics, vision and control : fundamental

algorithms in MATLAB. Berlin: Springer, 2011.

[6] P. I. Corke, “A robotics toolbox for MATLAB,”

Robotics & Automation Magazine, IEEE, vol. 3, no. 1,

pp. 24–32, 1996.

[7] P. Corke, “Robotics Toolbox for MATLAB, release 9

[Software],” May-2012.

[8] D. Laitinen, “Growing a Flavorful Agribusiness: Vanilla

- Hamakua Couple Takes on the Finicky Vanilla Orchid

and Makes a Market for Hawaiian Vanilla,” Ke Ola, 01-

Nov-2011.

[9] E. W. Davis, “Experiences with Growing Vanilla

(Vanilla Planifolia),” in Proceedings of Third

International Symposium on Spice and Medicinal

Plants; 21st International Horticultural Congress,

Hamburg, Fed. R, 1982, vol. 132, pp. 23–30.

[10] J. Reddekopp, “Personal interview and observation at

Hawaiian Vanilla Company, Paauilo, Hawaii,” 20-May-

2010.

[11] J. Albus, R. Bostelman, and N. Dagalakis, “The NIST

RoboCrane,” J. Res. Natl. Inst. Stand. Technol., vol. 97,

no. 3, pp. 373–385, Jun. 1992.

[12] E. Amatucci, R. Bostelman, N. Dagalakis, and T. Tsai,

“Summary of Modeling and Simulation for NIST

RoboCrane® Applications,” in Proc. 1997 Deneb

International Simulation Conference and Technology

Showcase, Detroit, MI, 1997.

[13] P. Nanua, K. J. Waldron, and V. Murthy, “Direct

kinematic solution of a Stewart platform,” Robotics and

Automation, IEEE Transactions on, vol. 6, no. 4, pp.

438–444, 1990.

86