Teletactile system based on mechanical properties estimation

Applied Bionics and Biomechanics 8 (2011) 237–252DOI 10.3233/ABB-2011-0023IOS Press

237

Teletactile system based on mechanicalproperties estimation

Mauro M. Sette∗, Hendrik Van Brussel and Jos Vander SlotenDepartment of Mechanical Engineering, Katholieke Universiteit Leuven, Leuven, Belgium

Abstract. Tactile feedback is a major missing feature in minimally invasive procedures; it is an essential means of diagnosisand orientation during surgical procedures. Previous works have presented a remote palpation feedback system based on thecoupling between a pressure sensor and a general haptic interface. Here a new approach is presented based on the directestimation of the tissue mechanical properties and finally their presentation to the operator by means of a haptic interface.The approach presents different technical difficulties and some solutions are proposed: the implementation of a fast Young’smodulus estimation algorithm, the implementation of a real time finite element model, and finally the implementation of astiffness estimation approach in order to guarantee the system’s stability. The work is concluded with an experimental evaluationof the whole system.

Keywords: Tactile feedback, haptic, virtual environment, Kalman filter

1. Tactile feedback in surgery

In common clinical and surgical practice, tactilefeedback is considered as a real diagnostic tool. Manydiseases cause hardening of organ tissues, allowingthe doctor to identify them using only the sense oftouch. Commonly the tactile sense is used for locatingtumours, identifying gallstones as a harder ball embed-ded in a soft background, or localizing a vessel’s path,which is identified as pulsing bodies. In the last yearsmany surgical procedures have been converted fromopen to minimally invasive. Open surgery is performedby accessing internal organs via a wide incision in thepatient’s skin. These wide openings allow the doctorto reach anatomical structures with his hands and thusperform accurate palpation inspections. During min-

∗Corresponding author. E-mail: [email protected].

imally invasive procedures the access to the body isachieved with small incisions in the patient’s skin. Thesurgeon inserts long-shaft surgical instruments throughthese openings, while the sight is guaranteed by alaparoscope. If on one side minimally invasive surgeryallows trauma reduction with subsequent reduction ofpain and recovery time after the surgery, on the otherthe sense of touch is strongly reduced when this lasttechnique is used. Tactile feedback is even completelyabsent when the direct manipulation of a minimallyinvasive instrument is substituted with telemanipula-tion. A telemanipulation system is a device that enablesa person to perform manual operations while separatedfrom the site of work (an example of a commercialsystem is the da Vinci Robotic Surgical System (Intu-itive Surgical, Mountain View, CA, USA)). It is thenextremely important to restore the tactile sense in thetelemanipulation procedures, developing a device that

1176-2322/11/$27.50 © 2011 – IOS Press and the authors. All rights reserved

mailto:[email protected].

238 M.M. Sette et al. / Teletactile system based on mechanical properties estimation

enables a person to touch an object or surface whileseparated from the site where the object/surface is,using the analogy with the telemanipulation systemit can be called teletactile system.

2. A teletactile feedback system for minimallyinvasive surgery

This work presents the study of a new system forthe restoration of tactile feedback in minimally inva-sive surgery. The tactile sense is a complex sense: itprovides information about shapes, stiffness, and tex-ture. This is the reason why the design of a completetactile feedback system is not an easy task [31]. For thisreason we decided to limit the field of application ofthe system to only a single tactile tasks: the localizationof tumours in an organ. As an example, we will refer tothe procedure of liver metastasectomy, that is, surgicalremoval of metastases (tumours) from the liver. Thesearch for metastases is seen as a procedure of lookingfor a harder ball embedded in a soft background, andthis is the task that will be analysed during the systemdevelopment.

Nicholls and Lee [15] define a tactile sensor as adevice or system that can measure a given propertyof an object or contact event through physical con-tact between the sensor and the object. Such a tactilesensor is based on the interaction force between anobject and the sensor. If we transfer this definition tothe case of searching for liver metastases, we can make

a mental scheme in which we have a certain volumewith non-homogeneous structure and we measure thedistribution of reaction forces observable on the sur-face when the volume is subjected to a compression.From a certain point of view, we measure not the realmaterial stiffness but only how this stiffness distribu-tion affects the reaction force. This measurement givesinformation on the surface stiffness resulting from theinteraction of different deep layers with different stiff-nesses. It could be useful to have the information onthe stiffness distribution in the whole volume becauseit can happen that important information has a tooweak effect on the surface stiffness, and identifying thehard inclusion becomes difficult. This is the idea driv-ing this work: imagine a system where, instead of thesurface reaction force, we could measure the stiffnessdistribution directly by means of a quantitative elas-tography approach [29]. What would be the advantageof this approach? How we could feedback the tactileinformation to the user? What would be the technicalproblems in terms of reliability? Would this system befast enough to create a realistic feeling?

This work is going to propose an implementation ofsuch a teletactile system, and is going to analyze itsproperties and performances.

3. System design and implementation

At a concept level a teletactile system could bedesigned as a tool for a robotic system, (e.g. the da

Fig. 1. The teletactile system is composed of four different modules. a) Module that controls the position of the “Ultrasound probe” connectedto the “Robotic arm”. b) Module for the estimation of mechanical properties, the “Young’s modulus estimator. c) Module to synthetize the tactileinput for the tactile display with the implementation of a semi-virtual environment, the “Real time finite element environment”. d) Module forthe control of the “Haptic interface”. The for modules are integrated on three parallel loops. Loop (1) is the image acquisition system (roboticarm and ultrasound system) and ultrasound image processing unit for the estimation of the Young modulus data; loop (2) the Young modulus isintegrated into a finite element (FE) model interfaced with a haptic device. Loop (3) represents the interaction between the human operator andthe haptic interface.

M.M. Sette et al. / Teletactile system based on mechanical properties estimation 239

Vinci System) composed of four different modulesrunning in three different loops, as shown in Fig. 1:

a – A module that controls the position of the“Ultrasound probe” connected to the “Roboticarm”;

b – a module, for the estimation of mechanical prop-erties, the “Young’s modulus estimator”;

c – a module, to synthetize the tactile input forthe tactile display with the implementation of asemi-virtual environment, the “Real time finiteelement environment”; and

d – a module implementing the “Haptic interface”.

The following sections are going to describe indetails the four blocks composing the system.

3.1. Module a – Control of the ultrasound probeposition through the robotic arm

A laparoscopic ultrasound probe that is manipulatedby a robotic arm, for example the da Vinci CANVASdescribed in [14], composes the hardware part of thisblock. Figure 2 describes the adaptation of a laparo-scopic ultrasound probe as the tool for the da VinciSurgical Robot System. The ultrasound laparoscopicprobe described in the system is the Aloka End-fireLaparoscopic probe (Aloka, Mitaka-shi, Japan), fre-quency range 3 - 8 MHz, diameter of the insertion part12 mm, and length of the insertion part 375 mm. Thismodule controls the position of the ultrasound probe inorder to acquire images of the area under investigation.The position of the ultrasound probe is controlled bythe position of the surgeon’s “finger” i.e. the locationin which the surgeon wants to touch. Anticipating thedescription of the other blocks can be observed that,

the ultrasound image is acquired an processed in theloop 1 (Fig. 1), the will keep running only in the loop2 until the surgeon finger’s goes out of the acquiredarea, in this case a new segment should be acquiredand processed. Here follows two possible solution onhow to optimize the coupling between the finger andthe ultrasound probe:

(1) While the system is engaged in the loop 2 at thetime T, calculate an adjacent portion of volumethat will be explored at the time period T+1(Fig. 3).

(2) Calculate a second slice along the volume’stransverse direction (Fig. 4). In this case, themodel for the virtual environment will beupdated slice by slice. This solution impliesthe use of a three-dimensional model. The mainadvantage is the possibility of creating the inputfor a bi-dimensional matrix tactile display. Forthis case, the requirement for the fu is difficultto define because it depends strictly on the hap-tic device used and on its resolution. However,we can only guess that it could be higher thatthe one described before.

Both solutions could be implemented efficiently.Each solution is strictly dependent on the haptic inter-face used. For a kinaesthetic feedback device, the firstsolution would be preferable because in this case it isnecessary to explore the surface along one direction.The second may be more suitable when the systemfeeds back the information through a tactile device andbi-dimensional information is needed. In this work, theapplication for a kinaesthetic device will be developed,and thus a bi-dimensional model will be implemented,allowing exploration along only one direction.

Fig. 2. The picture represents a normal Aloka (Aloka, Tokio, Japan) ultrasound probe for laparoscopic procedures (above), and the integrationof the probe in the da Vinci Surgical System (Intuitive Surgical, Mountain View, CA, USA). This integration was performed by Leven and it isdescribed in [14]. The Aloka probe has a diameter of 12 mm and a length of 375 mm.


Fig. 3. Two different strategies for the exploration of the surface bythe ultrasound probe have been presented. The first strategy proposesthat: At the time T the portion explored was calculated at the timeT−1, while the system calculates the portion that will be exploredin T+1.

Fig. 4. Two different strategies for the exploration of the surface bythe ultrasound probe have been presented. The second strategy pro-poses that: The different sections of a certain volume are calculatedsequentially. The volume will become available slice by slice in thevirtual environment.

3.2. Module b – Young’s modulus estimator

The time requirements for the Young modulus esti-mation depend on exploration speed at which normallythe fingers explore a surface. The block b calculatesthe Young modulus distribution of a finite volume por-tion, and as long as the finger stays in this portion,no new calculation is required. The best control algo-rithm that is needed to realize this synergy betweenloop 1 and loop 2 is still the object of investigation.If we assume that at each period T we are able to cal-culate a surface with width w (which is the width ofthe Laparoscopic-US-probe’s footprint under the con-dition of using a linear probe) and considering that themaximum exploration speed vex for the finger is 120mm/s [24] the update frequency (fu) of loop 3 is givenby:

fu = vex

w(1)

If we consider that footprint of the probe in Fig. 2is 60, mm, the update frequency for the loop has anupper limit of 2 Hz. This means that all the calculationsmust be solved in less than 500 ms. This represents theworst case scenario in which the finger is sweepingthe surface at maximum speed. This feature influencestrongly the choice of the algorithm to be used forthe Young modulus estimation. The tissue stiffness (orYoung modulus) estimations is performed by solvingan inverse elasticity problem, which needs as input thedisplacement field within the tissue, calculated from aset of ultrasound images, and the boundary conditions.The output is the estimation of the examined volume’smechanical properties. This technique, applying itera-tive or direct methods, has been extensively studied bythe authors and has already been described in previouspublications. For a complete explanation of the algo-rithm we refer to: [3, 8, 9, 19–23, 26–29, 34]. Theseprevious work showed how different set of boundaryconditions could be used, this system make use of a setof mixed boundary conditions, both traction (force)and displacement boundary conditions are applied tothe model. The traction boundary conditions are esti-mated with a pressure sensor located at the US probetip. This pressure sensor has two basic functions: (i) toacquire the information to set the boundary conditionsfor the solution of the inverse elasticity problem, and(ii) to integrate a force control loop for the explorationof the organ surface as described by Trejos in [35].


The estimation of mechanical properties can beimplemented using different strategies, the state ofthe art lists two different approaches, iterative anddirect. Both approaches have advantages and draw-backs that must be analysed in order to make thecorrect choice for an eventual development. Iterativemethods have been studied mainly; they have the bigadvantage of being less sensitive to noise as well asstable; in fact they are based on the solution of anon-linear least squares problem that can be solvedwith optimization algorithms like the Gauss-Newtonor Levenberg-Marquardt method [18]. Direct meth-ods are faster but they are more sensitive to noiseand boundary conditions approximations. However ifa good level of strain estimation could be achieved thislast class of method is surely promising and can beimplemented efficiently on a real time device.

There exists a third possibility for the stiffnessestimation, which is the use of the shear wave elastog-raphy; this technique is very promising, and eventuallyallows a fast estimation of the mechanical properties.However it has not been studied by the authors of thiswork and will not be considered in the discussion ofpossible systems. Nevertheless, it remains an interest-ing approach that needs to be further analysed. Formore detail on this approach refer to Bercoff et al. [5],Sandrin et al. [32].

3.3. Module c – Real time finite element model

Let’s suppose that the information on the stiffnessdistribution is efficiently estimated from the intra oper-ative ultrasound images, as example we report thestiffness distribution calculated with a direct methodfrom a set of ultrasound images: Fig. 5 (picture takenfrom [30]). The problem that is still open is how torender this information to the operator. The systempresented here is a semi-virtual system, that is, theoperator interacts with a virtual environment that is

Estimated YM [Pa]

mm

mm

5 10 15 20 25 30

5

10

15

20

25

0

0.5

1

1.5

2

2.5

3

x 104

Fig. 5. The stiffness distribution is calculated with an elastographybased approach. Here is reported an example of stiffness distributionestimated with a direct elastography method. From [30].

updated in real time. Thus, the architecture at its basiscan be treated, at least for the human interaction part,as the architecture of a virtual reality application withhaptic feedback. Salisbury provides an example of sucharchitecture in [25], and Fig. 6 illustrates the concept.The simulation engine is responsible for computingthe virtual environment’s behaviour over time, the ren-dering blocks compute the responses for the visual,auditory, and haptic feedback, and finally betweenthe human operator and the virtual environment thereare the physical transducers. Although they presentinteresting and complicated challenges, the authorswill not treat the visual and auditory simulations andtheir rendering to the surgeon. Instead, possibilities,problems, and eventual solutions for the haptic simu-lation and rendering will be discussed. Compared tothe Salisbury scheme (Fig. 6), the scheme presentedin Fig. 1 is its analogous as highlighted in Fig. 7.

Fig. 6. A basicarchitecture for a virtual reality application classically includes twochannels:Theaudiovisual channel andahaptic channel. Thosetwo channels with the relative renderings realize the connection between the simulation engine and the human operator. From Salisbury [25].


Fig. 7. Analogies between the scheme presented here and Salisbury’s scheme.

The function of this block is to calculate the interactionforces between the virtual surgeon’s finger exploringthe volume, and the model built with the on-line esti-mated mechanical properties. In order to calculate sucha reaction force we make use of the theory of elasticity;the equation describing the mechanics of continuumis integrated numerically using a finite element (FE)method, as explained in [1]. The FEM is built in twodimensions; the elements used are quadrilateral ele-ments and the number of nodes should guarantee aspatial resolution equal to the finger tactile resolu-tion, that is, 1 mm [15]. The contact control blockcalculates the deformation applied to a set of nodesincluded in the area of contact between the avatar andthe model. This set of deformations is the displacementboundary condition applied to the model. The globalequation, representing the relationship among all theFE’s nodal values, is obtained by assembling elementwise equations by imposing inter-element continuityof the solution and balancing for inter-element forces(detailed in [1]). This can be reduced to solve a systemof differential equations:

Mu + Du + Ku = F (2)

where u is a 2n-element, nodal displacement vector;u and u, the respective velocity and acceleration vec-tors; F , the external force vector; M, the mass matrixof size (2n, 2n); D; the damping matrix; and K thestiffness matrix, while n is the number of nodes in the

FEM. During surgical manipulation, dynamic effectsare negligible [4], and thus it is sufficient for the quan-titative elastography algorithm to calculate only theelastic properties of the examined model. The FEMwas built through three different steps:

(1) The problem to solve was identified. The numer-ical solution which approximates the solutionof the elasticity problem, or the Navier-Lameproblem [1], was found.

(2) The FEM was implemented and the boundaryconditions were integrated. The numerical inte-gration of the constitutive equation leads to thedefinition of a linear system. The coefficient’smatrix K is built considering the model com-posed of quadrilateral meshes and linear hatfunctions (2n, 2n). The vector of the unknownsx (2n) is the vector of the nodal displacements.Finally the vector of the external forces f (2n)is applied on each node (n is the number ofnodes). The multiplication factor 2 is the num-ber of degrees of freedom (d.o.f.). The formaldescription of how to build the FEM is given in[1]. For simplicity we will assume it is possibleto build the FEM in the form:

Kx = f (3)

The Dirichlet boundary conditions (DBC),which are the displacement imposed at the


nodes, will be applied by means of the Lagrangemultipliers. The displacement is fixed in onespecified direction and is possibly free in others.This can be translated in defining the system:

BU = w (4)

where the matrix B has dimensions (d, 2n),where d is the amount of d.o.f. that need to beconstrained with the DBC and xd is the valueassigned to the degrees of freedom on �D. Thematrix B is initialized to values of zero, andfor each row d the 2n-elements are set to one.Finally, w contains the array of xd values. Cou-pling this set of conditions (4) through Lagrangeparameters with (3) leads to the extended sys-tem:(

K BT

B 0

)(x

λ

)=(

f

w

)(5)

(3) The linear system was solved with a classicalgorithm like Gaussian elimination [2]. TheLagrange multipliers approach is more conve-nient when it is necessary to implement theFEMs in a real time environment. This allowsthe calculation of the reaction forces on theDirichlet nodes with a small computationalload, and provides a fast and reliable systemto be implemented in real time [4]. In prac-tice the displacements induced by the contactwith the avatar are assigned to the vector xd andapplied to the d.o.f. d. This technique makes thealgorithm faster because it avoids the stiffnessmatrix’s re-computation at each loop.

3.4. Module d – Haptic interface, hardwareand control

The haptic interface (Fig. 7) block includes thecontrol algorithm for the haptic interface, a featureof fundamental importance because it covers the gapbetween the simulation’s updating frequency (lower)and the haptic control’s required updating frequency(higher) and contact control (the block that identifiesthe interaction between the haptic interface and themodel).

3.4.1. Control of the haptic feedbackIn past years researchers have been working on find-

ing a solution able to give a realistic feeling and at thesame maintaining the stability of the controlled system.The aim of this section is not to give an exhaustiveexplanation of the complex theory behind the hap-tic feedback and telemanipulation, but instead to treatone of the main problems that could be present wheninteracting with a virtual (or semi-virtual) haptic envi-ronment: the instability generated by the low updaterate in the simulation engine. The virtual environmentupdating frequency is dependent on the implemen-tation of the FEM and the type and size of FEMused. Colgate et al. [6, 7] have demonstrated how theupdate frequency can affect the system stability. Withsmall changes, Colgate’s work can be adapted for thepresented application as illustrated in Fig. 8. The sim-ulation engine works in a discrete time domain; itcalculates the reaction force FEM in response to adisplacement x. The Zero Order Holder block repre-sents the link between the discrete time domain and the

Fig. 8. System’s scheme. The simulation engine integrates the solver for the finite element model (FEM). The interaction between the operatorand the haptic device is modeled. Finally the link between the simulation engine discrete time domain and the real continuous time domain ismodeled with a zero order hold.


continuous time domain. Colgate’s major finding is aformulation of a general rule, based on the passivitytheory [6], in which the relation between the environ-ment H(e−jωT ) (in this case the FEM), the inherentdisplay’s damping b, and the sampling period T is:

b >T

2

1

1 − cosωTRe{(1 − e−jωT )H(e−jωT )}

0 ≤ ω ≤ ωN (6)

where ωN = π/T is the Nyquist frequency.From Colgate’s rule it can be concluded that, in

order to maintain stability, the parameter T should bereduced. A common approach is to linearly interpo-late between two values instead of holding the value offFEM between two periods. A better approach couldbe to use a stiffness estimation scheme. This schemewas implemented using the formalism of the extendedKalman filter as previously presented by De Gersem in[10]. The Kalman estimator is based on recursive leastsquares and uses measurements and process noise toweigh the updates, which takes the measurement noiseinto account.

3.4.2. A new concept for contact controlThe avatar, a software representation of the device’s

tip, represents the haptic device in the virtual environ-ment; usually the model to be touched and the avatarare incompenetrable; that is, they can only deform, andthis property reflects the real object’s behaviour. In theexperimental phase, this constraint will be removed,

giving the user the possibility of touching under thesurface; that is, touching a series of virtual surfacesgenerated below the surface representing the natu-ral organ’s surface. The expected advantage is theaugmentation of the tactile sense, leading to a morepowerful tool for the identification of metastases. Inorder to implement the capability of touching underthe surface it is necessary to formulate a novel schemefor the interaction between the avatar and the virtualsurfaces inside the volume (Fig. 9); the contact con-trol module works for this scope. A series of surfacesinternal to the model are set and the operator is free totouch along those surfaces.

4. Materials and methods

The first section of this paragraph explains whichplatform has been used for the physical implementa-tion of the system. Then the system has been tested withrespect to several aspects. The first experiment sets areaimed at characterizing the system under the perfor-mance aspects. The speed of calculation of the virtualenvironment has been tested, varying the model’s size;the stiffness estimation has been tested against the nor-mal stiffness; and finally the contact control strategyhas been tested evidencing how different the exploredstiffness appears when the boundary conditions arechanged. The experimental session concludes with aseries of experiments evaluating the system’s perfor-mances using psychophysical tests.

Fig. 9. Estimating the stiffness distribution within a volume, the surgeon has the possibility of touching under the natural surface. In the system’simplementation the surgeon’s finger is represented by an avatar. The touching surface is set by a secondary control at different levels of thevirtual model.


4.1. Hardware and software platform

The haptic rendering block has been implemented inthe PXI Real Time System (CPU-1) (National Instru-ments, Austin, TX, USA) with a dedicated programwritten in Labview (National Instruments, Austin, TX,USA). The simulation engine block implements theFEM for the elastic interaction between bodies and wasimplemented using a pseudo-code software: Octave[12]. The FEM was calculated by a second processingunit (CPU-2) in order to increase the computationalpower available and was run on a Linux operating sys-tem (Ubuntu) (http://wiki.ubuntu.com/). The data areexchanged between the two operating systems usingthe TCP/IP protocol [33] and between the CPU-1 andthe haptic device using dedicated digital ports. A gen-eral scheme is reported in Fig. 10. The system testingused the Phantom Premium (www.sensable.com) asthe haptic interface; this device is commonly used forhaptic systems and represents one of the most usedhaptic masters [25] (Fig. 11).

4.2. System’s performances

4.2.1. Effect of the Kalman filterAs explained in Sec. 3.4.1. the stability of the system

is improved by the introduction of a stiffness estimationscheme. This block uses a Kalman filter to decou-ple the virtual environment from the haptic interfacecontroller. It is necessary to measure which is the differ-ence between the stiffness estimated with the Kalmanfilter and the stiffness estimated in a ideal case. We

expect that the Kalman filter introduces some phaselag in the estimation due to the filtering process. Inorder to estimate the phase lag we compared a “realstiffness” that is the stiffness used as input to the filterwith the stiffness estimated with the Kalman filter. Theexperiment is conducted, recording the stiffness esti-mated during the surface exploration, the explorationwas simulated shifting the node on which the forcewas applied from one side of the model and back at thespeed of 120 mm/s, and at the same time the degreesof freedom on which the indentation displacement isapplied are recorded as a function of the time. Thoselast data are then used to calculate the real stiffnessoff-line.

4.2.2. Calculation speed over the distributedsystem

A major limiting factor for the system’s performanceis the FE calculation time and the speed of communi-cation between different systems. A test of the timerequired for the FE calculation and the communica-tion time required to transmit the data is performed.Different FE models where tested with degrees of free-dom ranging from 408 to 1071 (the FEM number ofcolumns was kept fixed to 50 while the amount of lineswas variated from 8 to 20).

4.2.3. New concepts for contact controland stiffness augmentation

The user perceived stiffness is measured as the ratiobetween the indentation force and the nodal displace-ment on which the force is applied. This quantity is

Fig. 10. The virtual environment is divided in two section each working on an independent processing unit (CPU), in the first block in calculatedthe interaction point of the haptic avatar with the virtual environment and there is block for the control of the haptic device; while the onthe second processing unit are solved the calculation of the virtual environment. The two blocks are physically on two different units and areconnected with a TCP/IP protocol.

http://wiki.ubuntu.com/

www.sensable.com


Fig. 11. Phantom haptic master (Sensable, Wilmington, MA). Thismater is able of providing up to six d.o.f. force feedback.

considered as index of the stiffness perceived by thesubject. A set of experiments was conducted with theaim of measuring this quantity over different touchingsurfaces of a FE model. Aim of this experiments is to

highlight the advantages brought by the semi-virtualenvironment. A linear elastic FE model was created,the plain strain approximation was implemented. Themesh was built with 50 by 10 quadratic elements. In theFE model was simulated a hard inclusion embedded ina softer background. The YM of the inclusion was fivetimes harder than the background. The model is pre-sented in Fig. 12. Nine different touching levels wereset. The first was at the “natural surface” (Y = 0 m) andthe last was at the line immediately above the hardinclusion (Y = −0.08 m). For numerical reasons wasnot possible to test the surface at Y = −0.09 m. Thesubject was asked to explore the surface with the hap-tic interface and the stiffness calculated as the rationbetween the applied force and the nodal displacementwas recorded.

4.3. Psychophysics experiments, subjects andprotocol

The aim of the psychophysical experiments was thetesting of the system in terms of the subject’s ability todiscriminate a hard inclusion embedded in a soft envi-

−0.05 0 0.05−0.1

−0.09

−0.08

−0.07

−0.06

−0.05

−0.04

−0.03

−0.02

−0.01

0

Mesh with normalized stiffness

X [m]

Y [m

]

0

0.1

0. 2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Fig. 12. The FE model used for the experiments was a linear elastic FE model, the plain strain approximation was implemented. The mesh wasbuilt with 50 by 10 quadratic elements, with dimension 10 cm by 10 cm. The harder inclusion, the black spot on the bottom, was five timesharder than the background.


Fig. 13. The protocol of the psychophysical test performed in thiswork foresees that the user is asked to identify two region ofthe virtual model, one stiffer (the inclusion) and one softer (thebackground). The model was implemented using the plain strainapproximation, and the model dimension was 20 cm by 20 cm.

ronment. The subject was asked to interact with thevirtual environment by means of the haptic master. Theprotocol method used for the psychophysical exper-iments was the method of constant stimuli (MCS),a commonly accepted psychophysical test [13]. Ran-dom combinations of stiffness for the inclusion and thebackground were presented to the subject (Fig. 13),who was asked whether or not he or she could identifythat the inclusion was harder than the background. Twosets of experiments were conducted. In the first ses-sion, “perfect stiffness” was presented to the subject.“perfect stiffness” means that stiffness is only depen-dent on the location touched on the surface. The forcereturned to the operator was the product of the modelindentation and the stiffness assigned to that coordi-nate. The values of stiffness used in this experimentare reported in Table 1, they represent some values inthe range of the Young modulus for sound and diseasedliver as reported in [36]. The second set of experimentsconsidered the influence of modelling the virtual envi-ronment with an FE method, and the “Kalman filter”stiffness estimation. The conditions in this last test areexpected to be more noisy with respect to the “perfectstiffness” case, firstly because the FEM has differentstiffnesses depending on whether it is tested at the bor-

Table 1Different values of stiffness used in the exper-iments, they represent some values in therange of the Young modulus for sound and

diseased liver as reported in [36]

Background [N/m] Inclusion [N/m]

25 2525 27.525 3025 32.525 3525 37.5

der or in the centre (even when the Young modulusis the same for the background and the inclusion),and secondly because the Kalman filter introduces adelay in the stiffness perception. For each session agroup of five healthy subjects was considered: onewoman and four men with an average age of 30 (±5)years.

4.3.1. Data processingAim of the data processing was to extract the thresh-

old for the recognition of two different stiffnesses,this threshold can then be used as evaluation of thesystem’s performances. For each pair of values of back-ground stiffness and inclusion stiffness, was extractedthe probability of identification of the harder inclu-sion (pca). It was calculated as the number of correctanswers (ca) over the total amount of trials and sub-jects (sum of ca and non correct answers (nca)) asexplained by the Eq. 7.

pca = ca

nca + ca(7)

The psychometric function was extracted, fitting theresults with a Weibull distribution (8).

p(s) = 1 − 2−( sα

)β (8)

where α and β are the parameters characterizing theWeibull distribution, α represents the coherence sup-porting threshold performance, and β is the slope ofthe curve [16]. The range was fixed between (0,1),the possible answers were fixed at 2, s is the stim-ulus amplitude (measured as absolute stiffness), andp is the probability of identifying the difference. Theexperimental probability distribution was used to fitthe a Weibull distribution. A nonlinear optimizationmethod (the Nelder-Mead method) was used for fit-ting [17]. The Weibull distribution of p(0.75) was used


to define difference threshold. The Difference Thresh-old is the minimum amount of stimulus intensity thatproduces a noticeable variation in sensory experience.The following relation, known since as Weber’s Law,defines the Weber’s fraction (Wf ) (Eq. 9):

Wf = �I

I(9)

where �I (in this case = p(0.75) – I) is the variationthat should imposed to the stimulus I to be perceivedas different, in this case the variation of the stimulus isa difference in stiffness [11].

5. Results

Figure 14 presents the real stiffness compared withthe stiffness estimated with the Kalman filter, in func-tion of the time. Aim of the measurements is to put inevidence the delay in time introduced by the Kalmanestimation. In Fig. 15 the time required for the FEMcalculation and the data communication through theTCP/IP protocol is reported as a function of the FEMsize measured in degrees of freedom (d.o.f.). In Fig. 16the relative variation of the perceived stiffness whenthe boundary conditions are varied is reported. Thedata obtained collected from all the experiments per-formed on the five subjects were analyzed together.The two Tables 2 and 3 show the collected data, forthe perfect stiffness and the Kalman filter respectively.

1.8 2 2.2 2.4 2.6 2.8

x 104

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Time [ms]

Nor

mal

ized

est

imat

ed s

tiffn

ess

Fig. 14. The model’s punctual stiffness has been estimated with aKalman filter. The figure presents the stiffness estimation vs. time.Real stiffness ©, Kalman filter estimated stiffness *.

300 400 500 600 700 800 900 1000 1100 120020

25

30

35

40

45

50

55

60

65

70

Model’s DOF

Cal

cula

tion

time

[ms]

Fig. 15. The amount of degrees of freedom composing the FEmodel affect strongly the time required for finding the solution. Timerequired for system solution and result’s communication (TCP/IP)in function of the d.o.f. is here presented.

For each experiment are reported the correct answers(ca), meaning that the subject recognized that the twostiffnesses were different, and the non-correct answers(non-ca), meaning that the subject did not recognizedthe two stiffnesses as different. For each set of datais reported the percentage of correct answers (pca).The pca was used to fit the psychometric function thatis reported in the two figures Figs 17 and 18 for theexperiment of the perfect stiffness and Kalman filter

Surface longitudinal position [elements]

Exp

erim

ent s

eque

nce

5 10 15 20 25 30 35 40 45 50

1

2

3

4

5

6

7

8

9

10

11

Fig. 16. The perceived stiffness was measured allowing the subjectto interact on different surfaces. The figure represents the stiffnessalong the surface in the nine experiments conducted. The perceivedstiffness was normalized over the biggest recorded stiffness.


Table 2Data perfect stiffness. The amount of correctanswers (ca) non correct answers (nca) andpercentage of correct answers are reported inthe table. The data were collected using themethod of constant stimuli testing the system

in the “perfect stiffness” modality

Stimulus ca nca pca

55 (25) 2 8 0.260 (30) 3 6 0.3365 (32.5) 2 3 0.470 (35) 5 2 0.71375 (37.5) 5 0 1

Table 3Data Kalman filter. The amount of correctanswers (ca) non correct answers (nca) andpercentage of correct answers are reported inthe table. The data were collected using themethod of constant stimuli testing the system

in the “Kalman filter” modality

Stimulus ca nca pca

55 (25) 3 2 0.660 (30) 3 2 0.665 (32.5) 4 1 0.870 (35) 5 0 175 (37.5) 5 0 1

respectively. by fitting the psychometric function arelisted in Table 4 for the “perfect stiffness” and for the“Kalman filter” experimental sessions. The parame-ters reported are:

I the two values for the Weibull distribution α

and β, andII the Weber fraction (Wf) calculated using the Ratio

value for p(0.75).

6. Discussion

This work presented a novel scheme for a teletactilefeedback system for minimally invasive surgery. Thetactile information is estimated in a quantitative man-ner using a method based on quantitative elastography.This system is able to implement one of the differentalgorithms that have been developed for the on-linequantitative estimation of the stiffness based on ultra-sound images. The system is designed to be used incombination with a robotic device for minimally inva-sive surgery. The ultrasound probe is manipulated bythe end effector of the robotic arm. The acquired ultra

1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 1.40.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Ratio − (Inclusion stiffness)/(Background stiffness)P

roba

bilit

y of

per

ceiv

ing

a di

ffere

nce

Fig. 17. Psychometric function estimated on the test with the perfectstiffness experiments. The ∗ are the experimental data, the line is thefitted Weibull distribution.

1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 1.40.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Ratio − (Inclusion stiffness)/(Background stiffness)

Pro

babi

lity

of p

erce

ivin

g a

diffe

renc

e

Fig. 18. Psychometric function estimated on the test with theKalman filter stiffness estimation. The ∗ are the experimental data,the line is the fitted Weibull distribution.

sound images are used for the estimation of the tis-sue stiffness distribution. An on-line updated virtualenvironment realizes the interface between the cal-culated stiffness distribution and the haptic interfaceused to render the tactile sensation to the user. Thisapproach provides several advantages. Among otherswas shown how the stiffness perception could be aug-mented variating the surface on which the model isexplored. Figure 16 explains this concept. The identi-fication of hard inclusion in a soft background consistsin the identification of the stiffness difference between


Table 4Results obtained in the “Perfect stiffness”

and “Kalman Filter” experiments

a b Wf

Perfect stiffness 32.57 8.91 0.40Kalman filter 27.03 5.08 0.23

one are respect to another. The closer the touching sur-face is respect to the harder inclusion the bigger isthe contrast between the background stiffness and theinclusion stiffness. This feature brings great advantagebecause represents a possible way to improve the tactilefeedback during the surgery.

Second aspect that was examined is the calcula-tion time required for the solution of the FE problem,this time represents a huge limiting factor for theupdating rate of the whole system. The time increaseswith the increasing of the model’s complexity, i.e.with the increasing of the degrees of freedom. Futureimplementation should evaluate parallel computingsolutions.

The update rate affects directly the system’s stabil-ity, as explained reporting Colgate’s work [6, 7]. Thestrategy proposed to overcome this problem is basedon the physical decoupling of the real time finite ele-ment model and the controller of the haptic device, thisdecoupling is realized introducing a stiffness estimatorbased on the Kalman filter.

As shown in Fig. 4, the Kalman filter estimationdeforms the shape of the stiffness used to controlthe haptic device, represented mainly by a delay inthe correct stiffness representation. The experimentalquestion is then whether or not this estimation couldaffect the stiffness perceived by the subject. In order togive an answer to this question a set of psychophysi-cal experiments have been performed with the aim ofdefining a psychometric curve able of quantifying theuser’s perception.

The psychophysical test has shown that the inclu-sion’s stiffness could be identified in both the “perfectstiffness” (PS) and “Kalman filter” (KF) experiments.The Weber fraction defines the difference in stimulusamplitude that lets the subject discriminate whether thetwo stimuli are different. The Wf value was found tobe bigger for the PS experiments; this value is unex-pected because the KF experiments are more noisy andthis should affect the discrimination ability. We canassume that such a difference could be due to the vari-ability of the selected population and the two values

are comparable. In general the system could providean understandable difference between the environmentand a simulated embedded tumour. Future experimentsshould extend the examined population in order toreduce the result’s variability.

As result of the stiffness estimation performance, itcan be observed that the Kalman filter is able to quicklyfollow the real stiffness; from the experiments only ashort delay was recorded. The delay has an effect onthe feedback provided to the operator and indeed wasreported as an annoying disturbance by many subjects.However the results have shown that a different stiff-ness due to an harder inclusion could be successfullyidentified.

7. Conclusion

In this work the importance of the tactile feedbackin minimally invasive surgery has been discussed. Thiskind of feedback is of extreme importance in surgi-cal procedures consisting in an essential tool for thediagnosis, as it gives intuitive and immediate infor-mation. The case study of procedures for resection ofliver metastases has been reported and has shown howis important to locate a hard spot in the liver by meansof palpatory procedures.

A teletactile feedback based on Young’s modulusestimation, on one hand, enables the use of interest-ing features like the possibility of touching under thenatural surface. On the other hand, it introduces somelimitation when integrated in a closed loop system.The limitation studied here is due to the time requiredto solve the system used to calculate the feedback.This system, implemented as finite element model,requires long time for the calculation, and this timeincrease with the increasing of the model’s degreeof freedom (see Fig. 15). The long sampling timecan introduce instability in the haptic interface controlloop, as explained in Sec. 3.4.1. This drawback couldbe compensated with the introduction of a stiffnessestimation block implemented as extended Kalman fil-ter. The psychophysical experiment conducted in thiswork demonstrated that although the filter introducesa phase lag in the estimation (see Fig. 14), this doesnot hinder the user from perceiving the correct stiff-ness, as results for the calculated Wf (see Table 4).Future works will study the implementation of a fastersolver for the FE model and the testing of the systemin a mock up of the surgical environment.


References

[1] J. Alberty, C. Carstensen, S.A. Fiunken and R. Klose, Mat-lab implementation of the finite element method in elasticity,Computing 69(3) (November 2002), 239–263.

[2] K.E. Atkinson, An Introduction to Numerical Analysis,John Wiley & Sons, New York, 1989.

[3] P.E. Barbone and N.H. Gokale, Elastic modulus imaging:on the uniqueness and nonuniqueness of the elastographyinverse problem in two dimensions, Inverse Problems 20(2004), 283–296.

[4] J. Berkley, G. Turkiyyah, D. Berg, M. Ganter and S. Weghorst,Real-time finite element modeling for surgery simulation: Anapplication to virtual suturing, IEEE Transactions on Visu-alization and Computer Graphics 10(3) (May 2004), 314–325.

[5] J. Bercoff, M. Tanter and M. Fink, Supersonic shear imag-ing: A new technique for soft tissue elasticity mapping, IEEETrans Ultrason Ferroelectr Freq Control 51(4) (April 2004),396–409.

[6] J.E. Colgate and J.M. Brown, Factors affecting the Z-widthof a haptic display, in: Proc IEEE International Conferenceon Robotics and Automation, May 1994, pp. 3205–3210.

[7] J.E. Colgate and G. Schenkel, Passivity of a class of sampled-data systems: Application to haptic interfaces, in: ProcAmerican Control Conference, Vol. 3, June 1994, pp. 3236–3240.

[8] J. Camino, M. Sette and M. Diehl, Young’s modulus recon-struction using a constrained Gauss-Newton method, in:Book of Abstracts of 26th Benelux Meeting on Systems andControl, Lommel, Mar. 2006.

[9] M.M. Doyley, J.C. Bamber, F. Fuechsel and N.L. Bush, Afreehand elastographic imaging approach for clinical breastimaging: System development and performance evaluation,Ultrasound Med Biol 27(10) (October 2001), 1347–1357.

[10] G. De Gersem, Kinaesthetic Feedback and Enanced Sensitiv-ity in Robotic Endoscopic Telesurgery, PhD thesis, KatholiekeUniversiteit Leuven, Faculty Mechanical Engineering, 2005.

[11] G. Deco, L. Scarano and S. Soto-Faraco, Weber’s law indecision making: integrating behavioral data in humans witha neurophysiological model, The Journal of Neuroscience :The Official Journal of the Society for Neuroscience 27(42)(October 2007), 11192–1200.

[12] J. W. Eaton, GNU Octave Manual, Network Theory Limited,2002.

[13] G. Gescheider, The Classical Psychophysical Methods, Psy-chophysics: The Fundamentals, chapter 3, Lawrence ErlbaumAssociates, 1997.

[14] J. Leven, D. Burschka, R. Kumar, G. Zhang, S. Blumenkranz,X. Donald Dai, M. Awad, G.D. Hager, M. Marohn, M. Choti,C. Hasser and R.H. Taylor, DaVinci canvas: A teleroboticsurgical system with integrated, robot-assisted, laparoscopicultrasound capability, Med Image Comput Comput AssistInterv Int Conf Med Image Comput Comput Assist Interv 8(Pt1) (2005), 811–818.

[15] M.H. Lee and H.R. Nicholls, Review article tactile sensingfor mechatronics – a state of the art survey, Mechatronics9(1) (1999), 1–31.

[16] J.R. Muller, M.G. Philiastides and W.T. Newsome, Micros-timulation of the superior colliculus focuses attention withoutmoving the eyes, Proceedings of the National Academy of Sci-ences of the United States of America 102(3) (2005), 524–549.

[17] J.A. Nelder and R. Meadf, A simplex method for functionminimization, The Computer Journal 7(4) (1964), 308–313.

[18] J. Nocedal and S.J. Wright, Numerical Optimization,Springer, August 1999.

[19] J. Ophir, K. Alam, B. Garra, F. Kallel, E. Konofagou andT. Varghese, Elastography: Imaging of tissue elastic proper-ties in vivo, in: Proc IEEE Ultrasonics Symposium, Vol. 2,October 1998, p. 1743.

[20] J. Ophir, S.K. Alam, B. Garra, F. Kallel, E. Konofagou,T. Krouskop and T. Varghese, Elastography: Ultrasonic esti-mation and imaging of the elastic properties of tissues, ProcInst Mech Eng H 213(3) (1999), 203–233.

[21] | J. Ophir, S.K. Alam, B.S. Garra, F. Kallel, E.E. Konofagou,T. Krouskop, C.R.B Merrit, R. Righetti, R. Souchon, S. Srini-vasan and T. Varghese, Elastography: Imaging the elasticproperties of soft tissues with ultrasound, J Med Ultrasonics29 (2002), 155–171.

[22] J. Ophir, I. Cespedes, H. Ponnekanti, Y. Yazdi and X. Li,Elastography: A quantitative method for imaging the elastic-ity of biological tissues, Ultrason Imaging 13(2) (April 1991),111–134.

[23] J. Ophir, B. Garra, F. Kallel, E. Konofagou, T. Krouskop,R. Righetti and T. Varghese, Elastographic imaging, Ultra-sound Med Biol 26(Suppl 1) (May 2000), S23–S29.

[24] W.J. Peine, P.S. Welleman and R.D. Howe, Temporal band-width requirements for tactile shape displays, in: Sith AnnualSymposium on Haptic Interfaces for Virtual Environment andTeleoperator Systems, ASME International Mechanical Engi-neering Congress and Expositions, 1997.

[25] K. Salisbury, F. Conti and F. Barbagli, Haptic rendering: Intro-ductory concepts, IEEE Computer Graphics and Applications24(2) (March 2004), 24–32.

[26] M. Sette, J. Camino, J. D’hooge, H. Van Brussel and J. Van-der Sloten, Comparing optimization algorithms for Young’smodulus reconstruction In. ultrasound elastography, In 2007IEEE Ultrasonics Symposium Proceedings, 2007.

[27] M.M. Sette, J.F. Camino, J. D’hooge, H. Van Brussel andJ. Vander Sloten, P4F-3 comparing optimization algorithmsfor the young’s modulus reconstruction in ultrasound elastog-raphy, in: Proc IEEE Ultrasonics Symposium, October 2007,pp. 2028–2031.

[28] M. Sette, F. Conversano, H. Van Brussel and J. VanderSloten, Minimally Invasive Technologies and Nanosystemsfor Diagnosis and Therapies, chapter Elastotouc, LupiensisBiomedical Publications, 2008, pp. 43–50.

[29] M. Sette, J. D’hooge, S. Langeland, P. Goethals, H. VanBrussel and J. Vander Sloten, Tactile feedback in minimallyinvasive procedures using an elastography-based method, IntJ CARS S504, 2007.

[30] M. Sette, Technological Contributions to Augmented Realityin Minimal Invasive Surgery, PhD thesis, Katholieke Uni-vesiteit Leuven, Leuven, 2010, p. 240. doi: D/2010/7515/81

[31] S. Schostek, M.O Schurr and G.F. Buess, Medical engineer-ing & physics review on aspects of artificial tactile feedbackin laparoscopic surgery, Medical Engineering & Physics 31(2009), 887–898.

[32] L. Sandrin, M. Tanter, S. Catheline and M. Fink, Shearmodulus imaging with 2-D transient elastography, IEEETransactions on Ultrasonics, Ferroelectrics, and FrequencyControl 49(4) (April 2002), 426–435.

[33] W. Richard Stevens, TCP/IP Illustrated, Vol. I: The Protocols,Addison-Wesley, Reading, MA, 1994.


[34] M. Sette, J. Vander Sloten and H. Van Brussel, New Tech-nology Frontiers in Minimally Invasive Therapies, chap-ter Estimation, Lupiensis Biomedical Publications, 2007,pp. 80–88.

[35] A.L. Trejos, J. Jayender, M.T. Perri, M.D. Naish, R.V. Pateland R.A. Malthaner, Robot-assisted tactile sensing for mini-

mally invasive tumor localization, Int J Rob Res 28(9) (2009),1118–1133.

[36] W.-C. Yeh, Y.-M. Hsu, P.-L. Kuo, M.-L. Li, P.-M. Yang, P.-H.Lee and P.-C. Li, IEEE ultrasonics symposium-1234, 2001IEEE Ultrasonics Symposium, 2001, pp. 1233–1236.

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttp://www.hindawi.com Volume 2010

RoboticsJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014


Active and Passive Electronic Components

Control Scienceand Engineering

Journal of



RotatingMachinery


Hindawi Publishing Corporation http://www.hindawi.com

Journal ofEngineeringVolume 2014

Submit your manuscripts athttp://www.hindawi.com

VLSI Design



Shock and Vibration


Civil EngineeringAdvances in

Acoustics and VibrationAdvances in



Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014

SensorsJournal of


Modelling & Simulation in EngineeringHindawi Publishing Corporation http://www.hindawi.com Volume 2014


Chemical EngineeringInternational Journal of Antennas and

Propagation




Navigation and Observation



DistributedSensor Networks


Teletactile system based on mechanical properties estimation

Documents

Transcript of Teletactile system based on mechanical properties estimation