Adaptation of a reaching model to handwriting: How ...

Psycol Res (1996) 59:64-74 © Springer-Verlag 1996

Ruud G. J. Meulenbroek. David A. Rosenbaum. Arnold J. W. M. Thomassen. Loukia D. Loukopoulos. Jonathan Vaughan

Adaptation of a reaching model to handwriting: How different effectors can produce the same written output, and other results

Received: 18 September 1995/Accepted: 15 November 1995

Abstract This report shows how a model initially developed for the control of reaching can be adapted for the control of handwriting. The main problem addressed by the model is how people can produce essentially the same written output with different effectors (e.g., the preferred or nonpreferred hand, the foot, or even the mouth). The model is based on the assumption that writers strive for invariant graphic outputs when they write with different effectors, when they write on surfaces with different orientations, or when they write large or small script; such output invariance is an essential requirement for later recognition of the written result. Given this assumption, the question is how the motor system enables the relevant effectors to generate the necessary pen strokes. The adapted model provides one possible answer to this question. It is the first fully working model of multijoint activity underlying writing and related graphic tasks. We describe how the model differs from other models developed in the past, and we review the model's strengths and weaknesses.

Introduction

A hallmark of skilled motor performance is the capacity to achieve the same aim through different means

R. G. J. Meulenbroek ( ~ ) • A. J. W. K. Thomassen Nijmegen Institute for Cognition and Information, P.O. Box 9104, 6500 HE Nijmegen, The Netherlands; e-mail: meulenbroek@nici. kun.nl

D. A. Rosebaum Department of Psychology, Pennsylvania State University, 621 Moore Building, University Park, PA 16802-3104, U.S.A.

L. D. Loukopoulos Department of Psychology, University of Massachusetts, Amherst, MA 01003, U.S.A.

J. Vaughan Department of Psychology, Hamilton College, Clinton, NY 13323, U.S.A.

(Bernstein, 1967). This capacity is most dramatically reflected in handwriting, where the written output of a normal adult remains invariant with changes in the size of the script, the orientation of the writing surface, and the effectors used (Lashley, 1942; Merton, 1972; Raibert, 1977; Wright, 1993; Swinnen, 1991; Castiello & Stelmach, 1992). How this equifinality is accomp- lished is a fundamental problem for theories of perceptual-motor control. A number of authors have suggested that the problem can be solved by assuming that the motor-control system relies on separate information-processing modules (e.g., Vredenbregt & Koster, 1971; Van Galen & Teulings, 1983; Keele et al., 1995). Here we show that the problem can be solved in this way. We rely on a two-tiered control system. The higher level has spatial maps of pen-tip target positions along with information about the order in which these positions should be adopted. The lower level allows the effectors to bring the pen tip to the target positions in the correct serial order. The key advance of the model is that it can produce specific handwritten output, including detailed time-varying kinematics of the pen tip and joints for the effector system in use. It can also account for several findings concerning handwriting and handwriting acquisition that previous models could not. The report is organized as follows. First, we provide additional background to our work. Then we describe our model in some detail. Next we present illustrative results concerning the model's performance. Finally we indicate additional reasons for advocating the model but we also indicate where the model's claims still need to be evaluated more fully.

Background and an overview of the current approach

Traditional research on handwriting has relied on psychophysical and neuropsychological data, and has led

65

to the view that handwrit ing is governed hierarchically: semantic, syntactic, lexical, and motor-control pro- cesses are successively applied to smaller writing units (Ellis, 1982; Van Galen, 1991). The final output is a series of pen-tip strokes whose control may be governed with respect to extrinsic spatial kinematics (Edelman & Flash, 1987), idealized time-varying force components (Denier van der Gon & Thuring, 1965; Vredenbregt & Koster, 1971; Hollerbach, 1981), or underlying muscle activity (Bullock, Grossberg, & Mannes, 1993). Because a complete model must explain how the muscles are controlled during writing, it is noteworthy that only one model, that of Bullock et al. 1993), attacks this problem. It assumes that desired graphic output is represented in an effector-dependent fashion. However, this leaves open the question of how the same graphic output is produced when different effectors are used.

Our analysis begins with the assumption that graphic targets are represented in an effector-independent, spatial form, and that the motor system serves as a slave system, physically realizing graphic target se- quences in whatever way is most etficient when writing is required. In our view, the problem of motor equivalence, as manifested in the capacity to produce the same, or nearly the same, written output in different ways, need not be addressed by asking what invariants of the motor system cause written outputs to remain the same, or essentially the same, when different motor activities are relied on. Instead, in our view, the goal of writing the same letters with different effectors (or on surfaces with different orientations or with different sizes) is to end up with the same output; otherwise the written product could not be recognized later. On the basis of this analysis, the term motor equivalence is actually a misnomer. Stimulus equivalence is the goal of writing the same thing in different ways, and motor flexibility is the means by which that goal can be achieved. Other authors have said as much (Berkenblit & Feldman, 1988; Keele et al., 1995). For example, as Berkenblit and Feldman (1988) wrote: "There is a ... level that creates an abstract image (verbal or graphic) of ... a circle, line, etc . . . . Then a combination of effectors and a coordinative structure is specified" (pp. 372 373). Given this perspective, our view is that the primary aim of motor-control research concerned with writing and drawing should be to understand how the appropriate combinat ion of effectors is recruited, given some higher-level task. Our aim is to suggest how this recruitment may be achieved.

Before we turn to the problem of effector recruitment, we need to observe that, in the past, only a hand- ful of authors have analyzed the involvement of the effectors in graphic behavior, and none of them, as far as we know, has come to terms with the entire set of issues that must be addressed to have a fully opera- tional multi-joint writing system. One classical view of handwrit ing product ion is that the pen tip may be

displaced through controlled movements in different, typically orthogonal, directions (Hollerbach, 1981; Vredenbregt & Koster, 1971). But because this kind of model is concerned only with the movement of the pen tip in extrinsic spatial coordinates, it sidesteps the question of how the limb segments are coordinated to generate the appropriate endpoint movements.

Bullock et al. (1993), as mentioned earlier, did develop a model for handwriting that included limb- segment contributions, but their central claim, that different synergies are enlisted when writing must be done in different regions of extrapersonal space or over different spatial extents, leaves several crucial questions unanswered. How are the synergies selectively enlisted? What mechanism fits existing synergies to task demands? What happens when no synergy exists for a novel task? Raising these questions does not imply that Bullock et al.'s claims are incorrect, only that they are incomplete. The goal of our work is to offer a more complete model.

A caveat about our work is that our intent is to indicate the potential power of our approach rather than indicate at this stage of the research that our model necessarily fits particular detailed features of recorded handwriting behavior. What we are presenti- ng amounts to a competence model for handwriting. We are mainly interested in solving a practical problem

how to get a multisegmented creature to write or draw desired shapes in the highly adaptive way that skilled writers can - using different limb segments, compensating for immobility of some joints, and so on. In a sense, our work is an exercise in cognitive engineer- ing. The fact that our method works as well as it does, especially since it was adapted from a model designed for another task (reaching), suggests that we may be on the right track for understanding how writing movements are controlled, or at the very least, that we have created a foil against which other, possibly better, models, can later be compared.

Method

Adapting the reaching model to writing

The method used for modeling handwriting production is adapted from the reaching model of Rosenbaum et al. (1993, 1995). An advantage of this model is that it permits joints to compensate for changes in the mobility of other joints. The central claim of the model is that a positioning task is achieved by first the selection of a goal posture and then the generation of a movement that reduces the distance in joint space between the starting posture and that goal posture. We have extended the model to handwriting by supposing that handwriting can be viewed as a series of reaches, The model is rendered as a pen-wielding stick figure with six degrees of freedom that permit displacement of a pen tip in the horizontal plane (Figure 1). The stick figure can move its shoulder, elbow, wrist, metacarpo-phalangeal joint between the hand and index finger, and two interphalangeal joints of the index finger. In the model, which is specified in only 2 spatial dimensions, these joints have one degree of freedom each. A writing stylus resting on the center of the

66

Fig. 1A-H Movement that brings a pen tip to a spatial target. A Starting posture and spatial- target location (center of the large cross). B Spatial-error cost associated with each of 200 random stored postures. Spatial- error cost is defined as the Euclidean distance between the location of the pen tip if the stored posture were adopted and the spatial-target location. In this and succeeding panels the size of the circle is proportional to the magnitude of the cost. C Travel costs associated with the stored postures. D Total costs of the stored postures. E Weights assigned to the stored postures. The small cross in panels C-E represents the pen-tip position of the actual starting posture. F Movement to the spatial target location (5 equal time steps shown). G Angular joint velocities. For reasons of clarity, only three of the six joint-velocity profiles are shown in panel G. H Horizontal (solid line) and vertical (dotted line) velocity profiles (top) and tangential-velocity profile (bottom) of the pen tip

A.Starting Posture 4-

D. Total Costs

B.Spatial Error Costs

E. Contribution Weights

. , o °o

C.Travel Costs

F. Execution

G. Joint Velocity Profiles

Shld Elbw Wrst o o A ~o , o

0.55

W(deg/s) j.....~ ""e..~. Time (s) 0.97

-0.56 ~ -

H. Pen-tip Velocity Profiles Velocity (cm/s) X 0.51

-0.51 Tangential velocity (cm/s) 0.71

Time (s) 0.9;

metacarpai bone is held between the thumb and tip of the index finger. Because of the latter task constraint, movements of the thumb are coupled to those of the index finger. The lengths of the modeled limb segment and the ranges of joint motion used in the model are meant to be physically realistic.

Because a point in a plane has just two degrees of freedom, infinitely many reaches permit the pen tip to be brought to the target (Figure 1A). The best reach is found, according to the model, by reliance on posture representation stored in long-term memory. Memory for positions is superior to memory for movements (Smyth, 1984), suggesting that postures rather than movements are stored. Posture representations provide a convenient means of planning movements and modeling motor learning. Movements generally bring joints to angles for which muscle lengths permit opposing muscle forces to balance out (Bizzi et. al., 1992), and muscle lengths are implied by the joint angles defining a posture (Shadmehr, 1993).

In the model, stored posture representations are evaiuated for the contributions they can make to the task, and they are assigned weights accordingly. To derive the weights, each stored posture is evaluated for the spatial error that would ensure if that posture were adopted (Figure 1B). The spatial error of a stored posture is the Euclidean distance, in extrinsic spatial coordinates, between the target spatial location and the point along the limb segment chain to be brought to the target spatial location; in the case of writing, this point is the pen tip. The spatial consequence of a posture, and thus its distance from the target position (the spatial error) can be computed through forward kinematics or arrived at through interpolation or extrapolation from a lookup table relating joint angles to spatial locations of limb-segment points.

Each stored posture is also evaluated for the movement-related effort that would ensue if a movement were made from the starting

67

posture to the corresponding posture (Figure 1C). Rosenbaum et al. (1993, 1995) called this movement-related effort the "travel cost" of the stored posture. A stored posture's travel cost is the weighted sum of the costs of moving all the joints from their starting angles to the angles implied by the stored posture. The weight assigned to each joint reflects the expense of displacing that joint through a unit angular displacement. These joint-specific weights can be selected arbitrarily for simulation purposes, or they can be estimated empirically, using a least-squares criterion from observed joint contributions in tasks in which all joints can contribute. In the present simulation, the more proximal joints are assigned larger weights (between 0 and 1) than the more distal joints (0.8, 0.8, 0.6, 0.4, 0.4, 0.4 for the proximal to distal joints, respectively). The joint-related weights are parameters in the travel cost evaluation. They are not the same as the contribution weights for each of the stored postures, which result from the combined evaluation of both spatial error and travel cost.

The spatial-error cost and the travel cost determine the stored posture's total cost (Figure 1D). This is the weighted sum of the stored posture's spatial-error cost and travel cost, and the weights assigned to the two costs are nonnegative reals that sum to 1. The weights are set according to speed-accuracy demands. When it is important to be spatially accurate, the spatial-error cost is high, but when it is not important to be spatially accurate, the spatial-error cost is low. The weight (Figure 1E) for the stored posture is based on its total cost in relation to the total costs of all the stored postures. The weighted sum of the stored postures defines the goal posture. The weight assigned to each stored posture equals the Gaussian transform of its total cost, divided by the sum of the Gaussian transforms of all the stored postures' total costs. The Gaussian transform yields the largest value when the total cost equals zero. The standard deviation of the Gaussian is proportional to the smallest total cost of any stored posture, ensuring an almost exclus- ive contribution of stored postures with very small total costs to the weighted posture sum. The goal posture then results from the weighted sum of the stored postures. Weighted summing is possible because the postures are treated as vectors in joint space.

Once a goal posture is found, a movement is generated by each joint being allowed to cover the distance between its starting angle and its angle in the goal posture (Figure 1F). A movement is modeled as a straight-line path through joint space: joints start and stop moving simultaneously and are assumed to have bell-shaped angular-velocity profiles. The angular velocity of a joint is typically a bell-shaped function of time (Soechting & Lacquaniti, 1981), as is the tangential velocity of the displacement of the hand measured in extrinsic spatial coordinates (Flash & Hogan, 1985). In this way, the pen-tip trajectory need not be modeled explicity, in contrast to other models (Flash & Hogan, 1985; Plamondon, 1993), but rather emerges from the joint displacements.

Additional constraints for handwriting

A few additional assumptions are needed to adapt the reaching model to writing. One set of assumptions pertains to pen-tip target locations. Under normal circumstances, a writer specifies pen-tip target locations for him or herself; no one other than the writer determines where the pen tip should go at each moment in time. The question, then, is how these target locations are specified. We hypothesize that they are represented in a visuospatial format, subject to three constraints: (1) the target locations are serially ordered; (2) the configuration of the target locations can be scaled; and (3) the target locations can be projected onto different writing surfaces (depending on where the written output is supposed to appear). The first assumption is supported by the fact that, from infancy through adulthood, people can learn the order in which spatial targets are presented (Kagan, 1971; Keele, Cohen, & Ivry, 1990). The second and third assumptions are supported by psychophysical experiments showing that people can project images mentally onto different distal surfaces and size-scale the image (Kosslyn, 1980).

The spatial locations to which the pen tip is directed are of two types - points on the writing surface at which stroke curvature is locally minimal, and points on the writing surface at which stroke curvature is locally maximal. In skilled handwriting, path curvature is known to be inversely related to tangential velocity (Lacquaniti, Terzuolo, & Viviani, 1983; Viviani & Stucchi, 1992). Local curvature minima and local curvature maxima correspond to points of maximum and minimum tangential velocity, respectively. In our model, maximum-curvature points are treated as target locations (locations at which the pen tip must, in principle, stop), whereas minimum- curvature points are treated as via locations (locations through which the pen tip must pass without stopping). Via points are points of locally minimum (nonzero) curvature. A point with zero curvature (i.e., a point on a straight-line segment) is not explicitly represented as a via point.

Next consider how movement is controlled from a starting location S, to a target location T, through a via location V. Such a movement is called a stroke. Stroke production is achieved in our model by capitalization on the fact that smooth, curved trajectories can be generated by the combination of movements aimed at different locations (Morasso, Mussa-Ivaldi, & Ruggiero, 1983; Flash & Henis, 1991). In our model, during the first half of the movement from S to T, a movement is added from S to V and a movement is added from T to V (Figure 2A). During the second half of the movement from S to T, a movement is added from V to S and a movement is added from V to T (Figure 2B). This method ensures that within a stroke a curved movement passes through V.

Another assumption for handwriting ensures realistic kinematics for stroke production. In the earlier reaching model of Rosenbaum et al. (1995), execution of an individual reaching movement was modeled by use of a time- and amplitude-normalized raised cosine running from - r r to +~z. This part of the cosine wave has a bell shape. This mechanism also underlies the postural transition that corresponds to the movement from S to T in our stroke-production mechanism. However, the movements added to the movement from S to T are modeled on the basis of half a sine wave-cycle running from 0 to ~z. The reason for this added feature is that whereas velocity profiles of individual reaching movements are known to be bell shaped, velocity profiles of repeti- tive movements are known to be either sinusoidal when individual horizontal and vertical velocity profiles are considered, or parabolic when tangential velocity profiles are considered (Guiard, 1993). As a result of the combined use of bell-shaped velocity profiles for the movement from S to T, and sinusoidal velocity profiles for the movements superimposed on the movement from S to T in the case of via-point movements, the ratio of peak velocity to mean velocity of the tangential-velocity profiles of the end effector in extrinsic space varies between 1.65 and 2.0. Such ratios correspond to those reported by Hogan and Flash (1987) in the context of discrete and cyclical movements, respectively. The ex- pected ratio according to the minimum jerk model of Hogan and Flash is 1.875.

A final way in which the reaching model has been adapted to handwriting concerns the connection of successive curved movements. In the past, stroke fluency has been modeled by the over- lapping of successive stokes (Flash & Henis, 1991; Morasso et al., 1983). We rely on the same mechanism here: just before the completion of a stroke (i.e., when the pen tip's tangential velocity ap- proaches zero), the next stroke is begun. As a result, the pen tip continues to move at stroke borders.

Results

W e n o w d e s c r i b e t he s i m u l a t i o n s a c h i e v e d w i t h t h e m o d e l . I n g e n e r a l , t h e m o d e l g e n e r a t e d s a t i s fy ing s i m u - l a t i o n s o f o b s e r v e d l e t t e r s h a p e s a n d p e n - t i p k i n e m a t i c s

68

Fig. 2A, B Modeling a curved trajectory from S through V to T. A During the first half of the movement from S to T, a movement is added from S to V, and a movement is added from T to V. B During the second half of the movement from S to T, a movement is added from V to S and a movement is added from V to T. The movements are depicted as dashed vectors. Dotted vectors are translated to their dashed counterparts. The velocity of the movement from S to T is modeled as a raised cosine between

- ~ and + ~z; the velocities of the other movements are modeled as half cycles of a sine wave. Units are arbitrary

A 60

40

20

Y 0

-20

-40

-60 -60

B

60

4 0 -

20 -

Y 0-

- 2 0 -

-40-~

-60.- -60

I I

-40 -20

T,

S e " %

l t 0 ~ 2'0 X

V

I 4O

Velocity (cmts)

s,/ I I I

-40 -20 0

X

V

1.2

1.0

0.8

0,6

0.4

0,2

0 . 0

-0 .2

-0 .4

60

- X -= y

.5ST / k"

/ E °,

.5SV+.5YV

Time (s)

Velocity (cmls) 1 2

1 0 . , . r"

0 8 / ,,"

II. 6 ? o,

0 . 4 {

O.O

- 0 . 2

- e . 4

- X

- - y

"\, .5ST

%.

.5VS+.5VT I I 1

20 40 60 Time (s)

(Figure 3). When points of minimum and maximum tangential velocity (i.e., maximum and minimum curvature) from a human writing sample (Figure 3A) were used to specify via locations and target locations (Fig- ure 3C), the model's written output (Figure 3E) and pen-tip tangential velocity profile (Figure 3F) came close to the person's (Figures 3A and 3B), and the joint kinematics (Figure 3D) were also realistic (For a detailed analysis of joint kinematics in graphic production, see Meulenbroek et al., 1993.) When observed and modeled tangential-velocity profiles were compared, using normalized time for the completion of each of the 26 lower-case cursive letters of the Roman alphabet, the

variance in the observed tangential-velocity profiles accounted for by the model ranged from 64% to 93% over the 26 letters (mean = 76%, standard deviation = 8%). Given the number of data points in each tangential-velocity profile (37 to 71), the level of fit is highly significant (~ = 0.01) for every letter. Note that this result was obtained without any exhaustive attempt to improve the goodness of fit of the model. One aspect of the procedure that improved the goodness of fit concerned the shifting in time of the predicted velocity profiles (a few samples either forward or backward) to compensate for small artefacts due to time normaliz- ation. The seven free parameters of the model, which

Fig. 3A-F Human and model- based writing. A Lower-case letter l recorded with a Calcomp 9400 digitizer at a sampling rate of 100 Hz and a spatial accuracy of 0.2 ram, with pen-tip displacements lowpass filtered using a cut-off frequency of 12 Hz. B Corres- ponding horizontal (solid) and vertical (dotted) velocity profiles (top) and tangential velocity profile (bottom) of the pen tip, with successively numbered local minima and maxima. C Spatial locations of the successive local velocity minima and maxima, treated as via locations and target locations, respectively. D Angular velocities of shoulder, elbow~ and wrist. E Written output of the simulation. F Model kinematics

A. Human Writing

C. Model Input

1

3

4 O 2 o •

5 • ~ 7

E. Model Output - Writing

\

B. Pen-tip Velocities Velocity (cm/s) X

6.60 / ~ . . . . . y

~ . . . .

-5.60 Tangential velocity (era/s) 8.00 / / ~

Time (s] 0.38

D. Model Output - Joint Kinematics Shld Elbw Wrst 0 0 & .~ o 0

6.00

W(degls] ~ ~ ~ Time(s) 0.42

i

-6.00

F. Model Output- Pen-tip Kinematics Velocity (cm/s) ~ X

0

'% .° °o.°°

-4.30 Tangential velocity {•m/s) 6.001 / \

Time (s) 0.42

69

could in principle be adjusted to improve the goodness of fit, were the six expense factors for the joints (whose values were reported above) and the relative importance of minimizing spatial error compared with travel cost in specifying goal postures (set to 3/1).

Although an exhaustive search for the best-fitting parameters was not pursued, it is still possible to comment on the effects of changing the parameters. First, consider the effects of changing the expense factors of the joints. Changing these values would not affect the spatial accuracy with which individual targets are

reached, nor the shapes of predicted pen-tip paths, but it would alter the relative involvement of the joints and the movement times of individual movements (see Rosenbaum et al., 1995, for more details and simulation examples). Thus, the fit of the model might be improved by consideration of stroke durations in addition to tangential-velocity profiles of pen-tip movements. Of course, expense factors could also be fitted directly to observed joint contributions (see e.g. Cruse et al., 1993).

Next, consider the effect of changing the relative importance of minimizing spatial-error cost versus

70

travel cost. This change would result in interesting, and even unexpected changes in the performance of the model. Increasing the importance of minimizing spatial error cost would result in the pen tip coming closer to spatial targets, which, of course, can result in improve- ments in the match between predicted and observed pen-tip paths. Decreasing the importance of minimizing spatial-error cost would cause the pen tip to come less close, on average, to spatial targets, but in addition, and more interestingly, it would cause pen-tip paths to be smaller than when the spatial-error cost weight is high. The reason for the latter outcome is that when the spatial-error cost weight is small, the travel-cost weight is large, and so it becomes important to make smaller movements. Just such effects have been simulated with the model, as is shown in Figure 4. Given this outcome, one recognizes that our model allows writing size to be modulated in two ways - either by the size of the spatial target array being scaled, as was mentioned earlier in this report, or by a decrease in the relative importance of minimizing spatial-error cost compared with travel cost. Human behavioral evidence has suggested two means for the modulation of writing size. Before people start to write, they appear to set an overall size parameter for the scale of their forthcoming written output (Van Galen & Teulings, 1983). In addition, writers gradually decrease the size of their written output while actually writing words (Van Galen, 1990). We cannot prove that these two methods map directly to the two model-based methods we have just described, but it seems plausible that the scaling of the spatial-target array, as allowed by the model, corresponds to the pre-movement size-scaling operation of Van Galen and Teulings (1983), and that the reduction of spatial-error cost weight, also allowed by the model, corresponds to the intra-word size decrease of Van Galen (1990). For the second correspondence to be sanctioned, one would need to assume that spatial accuracy becomes less important as one progresses through a word. This hypothesis seems reasonable in view of the fact that letter identity is easier to predict as one goes from the first letter of a word toward the end (Calls, Teulings, & Keuss, 1983).

Because a central aim of our modeling effort is to show that our model can yield essentially invariant written outputs under varying conditions, we need to show that the model can achieve this objective. That it can is shown in several ways. First, if a letter to be drawn is scaled in size, the quality of performance remains essentially unchanged, provided that it is geo- metrically possible to produce the required letter at the required size with the modeled effectors. Thus, the model can produce large and small versions of the same letter. Second, if the expense of moving a joint changes, as might occur if a joint were injured, the form of the written letter and the pen-tip velocity profiles remain essentially invariant, though the angular velocities of the joints change markedly. This outcome reflects the

A. Model Input

B. Model Output

10:1 5:1 1:1 1:5 1:10

Fig. 4A, B Sensitivity of the model to parameter change. A Lower- case letter m recorded by means of procedure described in Figure 2A and from which the Model input, i.e., a target series was extracted; see text. B Model output under different values of the parameter representing the relative importance of minimizing spatial error versus travel costs (indicated by the ratios below the simulated letters)

capacity of our proposed system to compensate for changes in the mobility of selected joints. Third, if a letter needs to be drawn with different effectors on different writing surfaces with the fingers in the horizontal plane (Figure 5A) or with the arm in the sagittal plane (Figure 5B) - the written forms and the pen-tip kinematics (Figures 5E and 5F) are similar, but the angular velocities of the joints are extremely different (Figures 5B and 5C). This outcome demonstrates that the system we have proposed is capable of producing essentially the same written product through very different means.

Discussion

It has often been said that one of the basic capacities of human performance to be explained by a motor-control theory is how people manage to produce the same, or nearly the same, written output with different part of the body (e.g., Rosenbaum, 1991). To our knowledge, this is the first time that a model has been proposed that solves this problem in sufficient detail that actual pen strokes and effector kinematics emerge during simulated writing. Said differently, this is the first computational model that enables an actor to produce the same (or essentially the same) written output with different effectors, in different planes, and in different amplitude scales. Other authors have sketched in glo- bal terms how such a model might work (Berkenblit & Feldman, 1988; Keele et al., 1995). Our elaborated model suggests that their broad ideas were correct.

Further reasons for advocating the model

It is possible, of course, that other motor-control schemes might do as well as ours if they were developed in sufficient detail. Indeed, one of our aims in

Fig. 5A-F Production of the same letter with different effectors. A Written output produced in the horizontal plane with the shoulder and five other, more distal joints. B Written output produced in the sagittal plane with the hip and two other, more distal joints. C Angular velocities of the shoulder, elbow, and wrist in the horizontal plane. D Angular velocities of hip, shoulder, and elbow in the sagittal plane. E Horizontal (solid)- and vertical (dotted)-velocity profiles (top) and tangential-velocity profile (bottom) of the pen tip in the horizontal-plane condition. F Corresponding velocity profiles in the sagittaI-plane condition

71

A

C Shld Elbw Wrst o o A ,5 O O

6.00

Time ( 0.36

E Velocity (cm/s) 5.49

5.49

~ X - - - - - y

/%...

B ) []

D Hip Shld Elbw

o o & A O O

6.00 t

-6.00

Tangential velocity (crn/s)

7.80

I I

Time (s) 0.36

F Velocity (cm/s) 4.00

m x . . . . 'V

,°.%

-4.00

Time (s) 0.7Z

Time (s) 0.72

publishing this model is to spur others to develop competing models to a similar level of detail.

There are several other reasons why our model may be a useful focus of future work. First, the method we rely on for compensating for joint immobility is simpler than methods used elsewhere (Mussa-Ivaldi, Morasso, & Zaccaria, 1988). Second, although the written forms produced by our model are essentially invariant with their means of production, they are not perfectly invariant, nor are the forms produced by human writers under different conditions (e.g., Merton, 1972; Wann

& Nimmo- Smith, 1990). Third, the region of the work- space in which, according to our model, graphic movements are least costly is also the region in which people usually choose to write (Meulenbroek et al., 1994). Fourth, as was shown by Pantina (1957), the best way to teach children to write is to let them visually analyze a model of a new allograph to be learned, and then let them extract a spatial-target sequence from the trace to be written (where the targets correspond to points of maximum curvature), and then let them encircle these self-projected targets before attempting to connect the

72

points through series of strokes. Pantina's finding sup- ports our assumption that writing consists of series of strokes aimed at spatial locations.

While describing the model's successes, it is useful to show how it can account for the results of a recent study which, on first glance, seems problematic for the model (Wright & Lindemann, 1995). These investigators found that when people practiced writing particular letters with the non-dominant hand and then tried to produce other letters with the same hand, the subjects showed almost perfect transfer if the letters produced in the test phase had the same strokes as the practiced letters (as in the letters a and d). By contrast, there was little or no transfer if the letters written in the test phase did not share strokes with the practiced letters (as in the letters a and f), This outcome suggests that subjects learned effector- dependent strokes. That they learned strokes is suggested by the fact that only letters that shared strokes with the letters previously practiced enjoyed positive transfer. That the learning of strokes was effector dependent is suggested by the fact that the letters on which these subjects did poorly were of course letters they had previously performed many times with the dominant hand.

Wright and Lindemann's (1995) findings seems problematic for our model, because our model says that letters are memorially represented as serially ordered spatial targets and via points, rather than as strokes. So the model seems to predict that subjects should not benefit specifically from repetitions of the same strokes with the same hand. However, it also says that a key component of motor learning is the acquisition and storage of appropriate stored postures. (For details of the learning process, see Rosenbaum et al., 1995.) Since the movements made by the left and right hands in normal handwriting entail largely different postures, our model in fact predicts the Wright-Lin- demann result: subjects should do better when they can make use of postures already stored than when they cannot. According to this analysis, when people write the same letter with either hand, they attempt to direct the pen tip to the same set of spatial-target locations, but they do better when they have many stored postures available for performing the task. This is the same only when they have practiced the necessary strokes with the same hand.

Further challenges for the model

Despite the positive features of our model, many challenges lie before us. One challenge is to determine whether the departures from perfect invariance predicted by the model when writing is produced with different effectors correspond to those exhibited by human writers. Such deviations have been studied most closely by Wright (1990), focussing on writing with the

preferred and the nonpreferred hands. One could also study departures from invariance associated with writing in large and small sizes and with writing on surfaces having different orientations. To our knowledge, such data have not been reported. Collecting these data and evaluating them against predictions of our model would be a useful exercise.

A second challenge is to extend the model to kine- tics (the production of forces); so far, the model is purely kinematic (concerned only with positions). Ki- netics must be dealt with to create a working three- dimensional system. One type of force than must be considered is the force of the pen tip on the writing surface. A way to control this force is to aim at virtual locations beneath the surface, where the force to be produced is specified by the distance of the virtual location beneath the writing surface. This method of force control was suggested by Bizzi et at. (1992) and was confirmed by Vaughan, Rosenbaum, Moore, and Diedrich (1996) in the context of tapping. A second type of force that must be controlled is the set of mechanical interactions that occur among limb segments. Traditional physical accounts of such interactions are extremely complex (Hollerbach, 1990) - so much so that some authors have despaired of ever understanding how these interactions are governed (Hasan, 1991). However, the fact that they are governed (Bernstein, 1967; Schneider, Zernicke, Schmidt, & Hart, 1989) implies that the nervous system has tools for solving this problem. Understanding what those tools are and incorporating them into our model will be a challenge for us, just as much as for all investigators trying to develop realistic working models of motor control.

A third challenge for our model is to determine how letters are connected. The joining of letters may be modeled by the direct extension of the joining of strokes within letters. Joining the exit stroke of a letter with the entry stroke of a next letter by means of superpositioning will result in a local minimum in the tangential-velocity profile, which is realistic for joining letters such as g and e (Meulenbroek & Van Galen, 1989). Alternatively, to produce a fluent stoke without a local minimum in the tangential-velocity profile for an n-letter sequence (n > 2), one could assume that the pen tip is moved through an appropriate via point between the letters (Meulenbroek & Van Galen, 1989). Further simulations and comparisons with human writing will help to gauge the validity of these ideas. Insofar as movement through via points demands coarticulation (the superposition of a starting movement onto an ending movement) it is perhaps not surprising that children learn to connect letters smoothly only after they have learned to connect strokes smoothly within letters (Meulenbroek & Van Galen, 1989). In general, the development of coarticulation is an important index of motor-skill acquisition (Schneider et al., 1989).

73

A final challenge for our model is to let it match the findings on the neural implementation of human handwriting. Neuropschologists have shown that damage to different parts of the brain leads to disorders of handwriting production on the one hand and of spatial representation on the other (Baxter & Warrington, 1986). This dissociation points to the biological plausi- bility of the two-tiered system we have described. The more detailed aspects of neural implementation that will be needed to realize a computational system like the one we have proposed may take the form of a distributed architecture like that recently outlined for motor planning in the posterior parietal cortex (Morasso & Sanguineti, 1995).

A final comment on this whole enterprise is that our model of writing was based on an earlier model of reaching. Other authors have suggested that the neural control of reaching may underlie the neural control of locomotion (Georgopoulos & Grillner, 1989). In fact, the computational model of reaching, on which our writing model is based, has also been shown to be capable of walking (Rosenbaum et al., 1995). Because the basic model seems to be so widely applicable, its principles appear very promising. The fact that our writing model was based on a reaching model suggests that the understanding of the neural control of writing may in fact be found within the understanding of the neural control of reaching. Although much research has been done on the neural control of reaching, rela- tively little research has been done on the neural control of writing. A promising start in this direction has already been taken by Schwartz (1994). It seems that a better grasp of writing and drawing is within reach.

Acknowledgements Supported by the Royal Netherlands Academy of Arts and Science, the Dutch Organization for Scientific Research (NWO Stimulans Premie), NSF grants BNS-87-10933, BNS 90085, and SBR 9308671Z, an NIMH Research Scientist Development Award (KO2 MH00977-01A1), and a BRSG Faculty Research Grant. We thank Herbert Heuer, Steven Keele, and an anomymous reviewer for very useful comments.

References

Baxter, D. M., & Warrington, E. K. (1986). IdeationaI agraphia: A single case study. Journal of Neurology, Neurosurgery, and Psychiatry, 22, 369-375.

Berkenblit, M. B., & Feldman, A. G. (1988). Some problems of motor control. Journal of Motor Behavior, 20, 369-373.

Bernstein, N. (1967). The coordination and regulation of movements. London: Pergamon.

Bizzi, E., Hogan, N., Mussa-Ivaldi, F. A., & Giszter, S, (1992). Does the nervous system use equilibrium-point control to guide single and multiple joint movements? Behavioral and Brain Sciences, 15, 603-613.

Bullock, D., Grossberg, D., & Mannes, C. (1993). A neural network model for cursive script production. Biological Cybernetics, 70, 15-28.

Castiello, U., & Stelmach, G. E. (1993). Generalized representation of handwriting: Evidence of effector independence. Acta Psycho- logica, 82, 53-68.

Calis, G., Teulings, H. L., & Keuss, P. J. G. (1983). In search of writing and reading habits in the microgenetic phase of letter recognition. Acta Psychologica, 54, 313-326.

Cruse, H., Briiwer, M., & Dean, J. (1993). Control of three- and four-joint arm movement: Strategies for a manipulator with redundant degrees of freedom. Journal of Motor Behavior, 25, 131-139.

Denier van der Gon, J. J., & Thuring, J. Ph. (1965). The guiding of human writing movements. Kybernetik, 2, 145-148.

Edelman, S., & Flash, T. (1987). A model of handwriting. Biological Cybernetics, 57, 25-36.

Ellis, A.W. (1982). Spelling and writing (and reading and speaking). In A. W. Ellis (Ed.), Normality and pathology in cognitive func- tions (pp. 113-146). London: Academic Press.

Flash, T., & Henis, E. (1991). Arm trajectory modifications during reaching towards visual targets. Journal of Cognitive Neuro- science, 3, 220-230.

Flash, T., & Hogan, N. (1985). The coordination of arm movements: An experimentally confirmed mathematical model. Journal of Neuroscience, 7, 1688-1703.

Georgopoulos, A. P., & Grillner, S. (1989). Visuomotor coordination in reaching and locomotion. Science, 245, 1209-1210.

Guiard, Y. (1993). On Fitts' and Hooke's laws: Simple harmonic movement in upper limb cyclic aiming. Acta Psychologica, 82, 139-159.

Hasan, Z. (1991). Biomechanics and the study of multijoint movements. In D. R Humphrey & H. J. Freund (Eds.), Motor control: Concepts and issues (pp. 75-84). Chichester: John Wiley & Sons.

Hogan, N., & Flash, T. (1987). Moving gracefully: quantitative theories of motor coordination. Trends in Neurol Science, 10, 170-174.

Hollerbach, J. M. (1981). An oscillation theory of handwriting. Biological Cybernetics, 39, 139-156.

Hollerbach, J. M. (1990). Fundamentals of motor behavior. In. D. N. Osberson, S. M. Kosslyn, & J. M. Hollerbach (Eds.), An invitation to cognitive science. VoI. 2: Visual cognition and action (pp. 153-182). Cambridge, MA: MIT Press.

Kagan, J. (1971). Change and continuity in infancy. New York: Wiley. Keele, S. W., Cohen, A., & Ivry, R. (1990). Motor programs: Con-

cepts and issues. In M. Jeannerod (Ed.), Attention and performance XIII: Motor representation and control (pp. 77-110). Hill- sdale, NJ: Erlbaum.

Keele, S. W., Jennings, P., Jones, S., Caulton, D., & Cohen, A. (1995). On the modularity of sequence representation. Journal of Motor Behavior, 27, 17-30.

Kosslyn, S. M. (1980). Image and mind. Harvard: Cambridge Univer- sity Press.

Lacquaniti, F., Terzuolo, C., & Viviani, P. (1983). The law relating kinematic and figural aspects of drawing movements. Acta Psy- chologica, 54, 115-130.

Lashley, K. S. (1942). The problem of cerebra1 organization in vision. Biological Symposia, 7, 301-322.

Merton, P. A. (1972). How we control the contraction of our muscles. Scientific American, 226, (May), 30-37.

Meulenbroek, R. G. J., & Van Galen, G. P. (1989). The production of connecting strokes: Developing co-articulation in 8 to 12-year old children. In R. Plamondon, C. Y. Such, & M. L. Simner (Eds.), Computer recognition and human production of handwriting pp. 273-286). Singapore: World Scientific Press.

Meulenbroek, R. G. J., Rosenbaum, D. A., Thomassen, A. J. W. M., & Loukopoulos, L. D. (1994). A model of limb-segment coordination in drawing behaviour. In: C. Faure, P. Keuss, G. Lorette, & A. Vinter (Eds.), Advances in handwriting and drawing: A multi- disciplinary approach (pp. 348-362). Paris: Europia.

Meulenbroek, R. G. J., Rosenbaum, D. A., Thomassen, A. J. W. M., & Schomaker, L. R. B. (1993). Limb-segment selection in drawing movements. Quarterly Journal of Experimental Psychology, 46A, 273-299.

Morasso, P., & Sanguineti, V. (1995). Self-organizing body scheme for motor planning. Journal of Motor Behavior, 27, 52-66.

74

Morasso, P., Mussa-Ivaldi, F. A., & Ruggiero, C. (1983). How a discontinuous mechanism can produce continuous movement in trajectory formation and handwriting. Acta Psychologica, 54, 83-98.

Mussa-Ivaldi, F. A., Morasso, P., & Zaccaria, R. (1988). Kinematic networks: A distributed model for representing and regularizing motor redundancy. Biological Cybernetics, 60, 1-16.

Pantina, N. S. (1957). Formitovanie dvigatel'nogo navyka pi'ma. Voprosy Psikhologii, 4.

Plamondon, R. (1993). Looking at handwriting from a velocity control perspective. Acta PsychoIogica, 82, 89-101.

Raibert, M. H. (1977). Motor control and learning by the statespace model. Technical Report AI-M-351, MIT.

Rosenbaum, D. A. (1991). Human motor control. San Diego: Aca- demic Press.

Rosenbaum, D. A., Engelbrecht, S. E., Busche, M. M., & Loukopoulos, L. D. (1993). Knowledge model for selecting and producing reaching movements. Journal of Motor Behavior, 25, 217-227.

Rosenbaum, D. A., Loukopoulos, L. D., Meulenbroek, R. G. J., Vaughan, J., & Engelbrecht, S. E. (1995). Planning reaches by evaluating stored postures. Psychological Review, 102, 28-67.

Schneider, K., Zernicke, R. F., Schmidt, R. A., & Hart, T. J. (1989). Changes in limb dynamics during the practice of rapid arm movements. Journal of Biomechanics, 22, 805- 817.

Schomaker, L., Thomassen, A. J. W. M., & Van Galen, G. P. (1986). A computational mode1 of cursive handwriting. In. R. Plamon- don, C. Y. Suen, & M. L. Simner (Eds.), Computer recognition and human production of handwriting (pp. 153-177). Singapore: World Scientific.

Schwartz, A. B. (1994). Direct cortical representation of drawing. Science, 265, 540 542.

Shadmehr, R. (1993). Control of equilibrium position and stiffness through postural modules. Journal of Motor Behavior, 25, 228 241.

Smyth, M. M. (1984). Memory for movements. In M. M. Smyth & A. M. Wing (Eds.), The psychology of human movement (pp. 83-117). London: Academic Press.

Soechting, J. F., & Lacquaniti, F. (1981). Invariant characteristics of a pointing movement in man. Journal of Neuroscience, 1,710 720.

Swinnen, S. (1991). Motor control. In R. Dubelco (Ed.), Encyclopedia of Human Biology, Vol. V. San Diego: Academic Press.

Thomassen, A. J. W. M., & Teulings, H. L. (1983). The development of handwriting. In: M. Martlew (Ed.), The psychology of written language: Developmental and educational perspectives (pp. 179-213). New York: Wiley.

Van Galen, G. P. (1990). Phonological and motoric demands in handwriting: Evidence for discrete transmission of information. Acta Psychologica, 74, 259-275.

Van Galen, G. P., & Teulings, H. L. (1983). The independent monitoring of form and scale factors in handwriting. Acta Psy- choIogica, 54, 9-22.

Van Galen, G. P. (1991). Handwriting: Issues for a psychomotor theory. Human Movement Science, 10, 165-191.

Vaughan, J., Rosenbaum, D. A., Moore, C., & Diedrich, F. (1996). Cooperative selection of movements: The Optimal Selection model. Psychological Research Psychologische Forschung, 58, 254-273

Viviani, P., & Stucchi, N. (1992). Biological movements look uni- form: Evidence of motor-perceptual interactions. Journal of Ex- periment Psychology: Human Perception and Petformance, 18, 603-623.

Vredenbregt, J., & Koster, W. G. (t971). Analysis and synthesis of handwriting, Philips Technical Review, 32, 73-78.

Wann, J. P., & Nimmo-Smith, I. (1990). Evidence against the relative invariance of timing in handwriting. Quarterly Journal of Experi- mental Psychology, 42A, 105-119.

Wright, C. E. (1990). Generalized motor programs: Reexamining claims of effector independence in writing. In M. Jeannerod (Ed.), Attention and performance XIII (pp. 294-320). Hillsdale, NJ, Lawrence Erlbaum.

Wright, C. E. (1993). Evaluating the special role of timing in the control of handwriting. Acta Psychologica, 82, 5-52.

Wright, C. E., & Lindemann, P. G. (1995). Generalization of motor representations for handwriting: What is learned when learning to write with the non-dominant hand? In M. Simner (Ed.), Proceedings of the Seventh biannial conference of the International Graphonomics Society (pp. 170 171). London, ON: University Press.

Adaptation of a reaching model to handwriting: How ...

Documents

Transcript of Adaptation of a reaching model to handwriting: How ...