Sampling, WLS, and Mixed Models Festschrift to Honor Professor Gary Koch
Festschrift in Honor of Douglas L. Nelson - USFpsychology.usf.edu/dnelson/files/Robert Bjork...
Transcript of Festschrift in Honor of Douglas L. Nelson - USFpsychology.usf.edu/dnelson/files/Robert Bjork...
Interpreting one’s subjective experience: Heuristics and biases
Robert A. BjorkUniversity of California, Los Angeles
Festschrift in Honor of Douglas L. NelsonUniversity of South Florida
Tampa FloridaTampa, FloridaMarch 21-22, 2008
Interpreting subjective indices of learningInterpreting subjective indices of learning and performance
Perceptual fluency;Perceptual fluency;Retrieval fluency; Fluency of induction;y ;Fluency of association;
Foresight biasInflation of conditional predictionsInflation of conditional predictions
Experience-based vs. theory-based predictions of one’s own forgetting or learning;Egocentrism in social communication
Concluding comments on our subjective experience as teachers.
Important qualification
Subjective experience is not always misleading, it is sometimes even the best basis for judgmentsj g
Example: Jacoby and Kelley (1987)p y y ( )
J b Bj k & K ll (1994)Jacoby, Bjork, & Kelley (1994)
“Subjective experience like the publicSubjective experience, like the public media, is unavoidable, serves useful functions and is not to be fully trusted”functions, and is not to be fully trusted
Prelude: Interpreting objective indices ofPrelude: Interpreting objective indices of performance
L i fLearning versus performanceWhat we can observe is performance; What we must infer is learning;…and the former is an unreliable guide to the latter.
Conditions of instruction that make performance improve rapidly often fail to support long-term retention and transfer, whereas
Conditions of instruction that appear to create difficulties for the learner, slowing the rate of apparent learning, often optimize long-term retention and transfer
We are susceptible to being fooled by our own currentWe are susceptible to being fooled by our own current performance
Example: Interleaved versus blocked practice
Simon & Bjork (2001)
HighActual Predicted Retention
EErrorRandom
Random
Low Blocked
Practice 24hrs Practice 24hrs
Simon & Bjork (2001)
HighActual Predicted Retention
EErrorRandom
Random
Low Blocked Blocked
Practice 24hrs Practice 24hrs
Rohrer & Taylor (2007)
Wedge V= Spheroid V= 2
2 πhr3
4 2 πhr3
Spherical Cone V= Half Cone V= 32 2 πhr
37 2 πhr
3 3
Interpreting and misinterpreting perceptualInterpreting and misinterpreting perceptual fluency or familiarity
Heuristic value
Misattributions and illusionsMisinterpreting the cause of perceptual fluencyMisinterpreting the cause of perceptual fluencyMisinterpreting the meaning of perceptual fluencyIllusions of competence
Example: Reder (1987, 1988)
(Reder, 1987, 1988)
“What is the term in golf for scoring one underWhat is the term in golf for scoring one under par?”
Interpreting and misinterpreting retrievalInterpreting and misinterpreting retrieval fluency
Heuristic value
Misattributions and illusionsInterpreting performance as learning; illusions of p g p g;competenceEgocentrism in instruction and social communicationIncomplete/faulty models of ourselves asIncomplete/faulty models of ourselves as learners/remembers
Example: Benjamin, Bjork, & Schwartz (1998)
B j i Bj k & S h t (1998)Benjamin, Bjork, & Schwartz (1998)
Phase 1: 20 (easy) general-knowledge questionsE. g., “Who was the first president of the United States?”Participants asked to:
(a) hit ENTER as soon as the answer “came to mind” (latency recorded);(b) say the answer;(c) predict the likelihood they would be able--at the end of the(c) predict the likelihood they would be able--at the end of the experiment--to free-recall having given that answer.
Phase 2: Distracting activity (spatial/map task)Phase 3: Final testPhase 3: Final test
Free recall: Write down as many of the 20 answers you gave earlier as you can;(Original questions were not shown again)( g q g )
I t ti fl f i d ti /l iInterpreting fluency of induction/learning
Kornell & Bjork (in press). Abstracting concepts and patterns: Is “spacing the g p p p genemy of induction”?Lesson on the reliability, or lack thereof, of researchers’ intuitionsresearchers’ intuitions.
Interleaving/spacing versus blocking/massingg p g g g
Interleaved/spaced: items re-studied after other items
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this pictureare needed to see this picture. are needed to see this picture.
Blocked/massed: items studied in succession
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Inductive learningInductive learning
• The ability to generalize concepts and categoriesThe ability to generalize concepts and categories through exposure to multiple exemplars.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.QuickTime™ and a
TIFF (Uncompressed) decompressorare needed to see this picture.
Hypothesis
• Blocking/massing allows the learner to notice characteristics that unify a categorycharacteristics that unify a category
• Interleaving/spacing makes doing so difficultGentoo Gentoo Gentoo Gentoo
Gentoo Lachesis Reinhard Gentoo
Method: phases
1. Instructions2 Study2. Study3. Distractor4. Test5. Questionnaire5. Questionnaire
Kornell & Bjork (in press, Psychological Science)
InstructionsInstructionsIn this experiment you’re going to look at some beautiful paintings. To start, you’ll be shown 72 paintings for 3 seconds each. The paintings will be by twelve artists, with six pictures per artist. Try to learn to recognize which artistsix pictures per artist. Try to learn to recognize which artist painted which picture based on their style.
Later you’ll be shown 48 new paintings which you haven’tLater, you ll be shown 48 new paintings, which you haven t seen before. You’ll have to identify who painted each one.
For example, if there were only two artists, named Al and Barb, you’d be shown paintings by Al and Barb, and later, you’d be shown new paintings and asked who paintedyou d be shown new paintings and asked who painted them, Al or Barb.
M S S M M S S M M S S MM
Lewis LewisLewisLewis Lewis Lewis
Pessani StratulatSchlorffWexler Juras Mylrea
Pessani StratulatSchlorff WexlerJuras Mylrea
Hawkins HawkinsHawkinsHawkins Hawkins Hawkins
Results
Actual Responses
11
Actual Responses
0.6
0.8
uenc
y0.6
0.8
uenc
y
0.2
0.4Fr
eq
0.2
0.4
Freq
0Massed The same Spaced
0Massed The same Spaced
0.8A
0 5
0.6
0.7 SpacedA
Spacing vs. massing manipulation within
0.3
0.4
0.5
Massed
pparticipants
0.2
0.7
0.8
SpacedB
Spacing vs massing
0.5
0.6
0.7 Spaced Spacing vs. massing manipulated betweenparticipants
0 2
0.3
0.4Massed
participants
0.21 2 3 4
Test Block
Why a spacing effect?
• The test depended on recalling name-style associationsy– Spacing is the friend of recall
• Experiment 2Experiment 2– Final test: Yes/No recognition test for
paintings by studied artistspaintings by studied artists
MethodInstructions
Study
Distractor
Test Only one change: Recognition test
Questionnaire
Recognition test
Differentiation hypothesisDifferentiation hypothesis• Original hypothesis: Blocking/massing highlights similarities
LewisLewis Lewis Lewis
• New hypothesis: Interleaving/spacing highlights differences
LewisLewis Lewis Lewis
LewisLewis Schlorff Hawkins
Interpreting and misinterpreting fluency of te p et g a d s te p et g ue cy oassociation
Foresight biasKoriat and Bjork (2005, 2006)
Prediction-inflation biasesN l D d l d G d (2005)Nelson, Dyrdal, and Goodmon (2005)Koriat, Fiedler, and Bjork (2006)Maki (2007a, 2007b)( , )
Foresight bias (Koriat & Bjork 2005)Foresight bias (Koriat & Bjork, 2005)
Likelihood the second word will be given as a free associate to the first?
Lamp: LightLamp: LightFind: SeekSell: BuyCheese: CheddarCitizen: Tax
F i ht bi (K i t & Bj k 2005)Foresight bias (Koriat & Bjork, 2005)
Likelihood the second word will be given as a free associate to the first?
Lamp: Light ( 71)Lamp: Light (.71)Find: Seek (.03)Sell: Buy (.56)Cheese: Cheddar (.03)Citizen: Tax (.00)
F i ht bi (K i t & Bj k 2005)Foresight bias (Koriat & Bjork, 2005)
Likelihood the second word will be given as a free associate to the first?
Lamp: Light ( 71) Forward pairLamp: Light (.71) Forward pairFind: Seek (.03) Backward pairSell: Buy (.56) Forward pairCheese: Cheddar (.03) Backward pairCitizen: Tax (.00) “Purely a-posteriori” pair
F i h bi (K i t & Bj k 2005)Foresight bias (Koriat & Bjork, 2005)
Th t d f l t i ill i fThe tendency of learners to experience an illusion of competence during learning, resulting in inflated predictions of later recallp
Judgments of learning are made in the presence of information that is absent, but solicited, on a subsequent testSuch as the targets in cue-target paired associates (e gSuch as the targets in cue target paired associates (e.g., Cheese-cheddar)
We are unable to anticipate the test situation, when the answer will be absent, but required., qThus, “Cheese ___?____” will trigger other strong associates, such as “mouse,” “bread,” “wine,” etc., which will compete with “cheddar.”
Koriat & Bjork (2005)
Pair type Predicted recall Actual recallForward 78 79Forward .78 .79
Backward .76 .60
Purely a-posteriori
.65 .47p
Nelson, Dyrdal, & Goodmon (2005), y , ( )
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.a e eeded to see t s p ctu e
What is preexisting strength? Predicting free association probabilities, similarity ratings, and cued recall probabilitiesprobabilities, similarity ratings, and cued recall probabilities (Nelson, Dyrdal, & Goodmon, 2005)
INSTRUCTIONS TO PARTICIPANTS:INSTRUCTIONS TO PARTICIPANTS:“Some word pairs are more related than others. For example, mostpeople would say that CAT DOG is more strongly related than GUARD DOG. The purpose of this study is to collect ratings on a large numberDOG. The purpose of this study is to collect ratings on a large number of people in order to determine whether such ratings can predict recall. If DOG were recently studied, would CAT be a better memory cue than GUARD? Such results will be important for understanding how memory
kworks.“What we want you to do today is rate each pair of words that you aregiven on a scale of 1-7 in terms of their degree of mutual association or relatedness If the first word readily calls the second word to mind or ifrelatedness. If the first word readily calls the second word to mind or if you can find an easy way to relate the two words to each other, thengive the pair a higher rating. Conversely, give the pair a lower ratingwhen they seem weakly associated or related.
Nelson, Dyrdal, & Goodmon (2005): R lt d di iResults and discussion
“Forward and backward strength emerged as the onlyForward and backward strength emerged as the only significant predictor of the ratings.”
“… both backward (r = .30) and forward (r = .20) strength were correlated with the ratings;correlated with the ratings;“… the ratings explained 8.8% … of the variance in cued recall”
“ i ti i il it ti t i fl d b“… associative similarity ratings are not … influenced by several predictors that influence cued recall, including mediated strength, target activation strength, target
tit t th d t t f ”competitor strength, and target frequency.”
Inflation of conditional predictions f f p(Koriat, Fiedler, & Bjork, 2006)
Conditional prediction: “What is the probability that event Y will occur under condition X?
E.g., “What is the likelihood that if Iran stops supplying crude oil, g , p pp y g ,the price of a barrel will exceed $110”?
Prediction-inflation bias: Owing to a backward activation process, conditional predictions will tend to be inflated,
The stated outcome, whose likelihood is to be assessed, brings to the fore aspects of the condition that support the occurrence of thefore aspects of the condition that support the occurrence of the outcome.Example: Predicting free-association probabilities
Inflation of conditional predictions f f p(Koriat, Fiedler, & Bjork, 2006)
INSTRUCTIONS TO PARTICIPANTS:“In a word-association ask, a person is presented with a word and is asked to say as fast as he can the first word that comes to mind in response to the presented wordin response to the presented word.
“You will be presented with pairs of words. For each pair we would like you to estimate the percentage of people who would say thelike you to estimate the percentage of people who would say the second word (on the right) as the first response to the stimulus word (on the left)
; Hebrew is written from right to left).“You will find a blank line next to each word pair. Write on that line a number between 0 and 100 that reflects the percentage of people who, according to your opinion, would say the second word (the right one) in response to the first word (the left one).”g ) p ( )
Koriat, Fiedler, & Bjork (2006)Experiment 5
80
90ActualPredicted- Control
Experiment 5
60
70
Predicted- ControlPredicted- Experimenta
40
50
20
30
0
10
U B F
Associative Directio
Koriat, Fielder, & Bjork (2006)Experiment 6
90
100Actual
Experiment 6
70
80
90Predicted- ControlPredicted- Experimenta
50
60
20
30
40
0
10
20
U B F
Associative Directio
How are memory judgments made?
Theory-basedDeliberate
Experience-basedAutomatic
In general…“… forgetting happens.”
E.g., Encoding fluencyRetrieval fluency
“… studying causes learning.”
et e a ue cyCue familiarity
Objective and subjective learning curvesObjective and subjective learning curves
Koriat, Bjork, Sheffer, & Bar (2004): Predicting one’s own forgettingone s own forgetting
Kornell & Bjork (in progress): Predicting one’sKornell & Bjork (in progress): Predicting one s own learning
Koriat, Bjork, Sheffer, & Bar (2004)jPredicting One's Own Forgetting: The Role of Experience-Based and Theory-Based Processes.
“ j d t f l i (JOL) if d b i“… judgments of learning (JOL), if governed by processing fluency during encoding, should be insensitive to anticipated retention interval.”
Subjective experience: “Various mnemonic cues contribute directly to produce an immediate feeling of knowing that can serve as the basis of judgments. Thus, for example, encoding and retrieval fluency may foster a feeling of competence that can serve as [an experience-basedfoster a feeling of competence that can serve as [an experience based basis for judgments of learning].”Domain-specific knowledge retrieved from memory. “Theory-based judgments, in contrast, rely on the deliberate use of specific beliefs and j g y pinformation to form an educated guess about one's own knowledge. Thus, JOLs may utilize such rules as “memory performance should be better on recognition than on recall memory test.”
Predicting one’s own forgetting (K i t Bj k Sh ff & B 2004)(Koriat, Bjork, Sheffer, & Bar, 2004)
E i t 1Experiment 1:60 paired associates
30 related;30 unrelated30 unrelated
Participants judge, pair by pair, the likelihood they will remember that pair on a later cued-recall testRetention interval to the final test (between-subjects):
Immediately after the study phase;One day;One week
Experiment 2:Experiment 2: To-be-predicted retention interval manipulated within-subjects.
Koriat et al. (2004): Experiment 4c
“This experiment was an attempt to bring the matter to an absurdity: Would participants fail to take into account a year’s retention interval in comparison toaccount a year s retention interval in comparison to participants who are asked to predict performance under immediate testing?”g
Results: Experiment 4c
Retention Interval
MeasureTen minutes
One week
One year
P di t d ll 35 2 40 4 37 1Predicted recall:(Exp. 4c)
35.2 40.4 37.1
Koriat et al. (2004): Experiment 7
S E i 4 b i i k d i b f iSame as Experiment 4c, but participants asked to estimate number of items forgotten:
“In a previous experiment that we conducted, students were presented with a list of 60 word pairs such as “table-chair” and “girl-eagle” one after the other. Each word pair wasword pairs such as table chair and girl eagle one after the other. Each word pair was presented for 4 seconds. The students’ task was to study these pairs so that when presented later with the first word, they would be able to recall the second word. The memory test took place ten minutes/one week /one year later.”“In the test the first word was presented with a blank line next to it for example:In the test, the first word was presented with a blank line next to it, for example:
Table _________Girl _________
and the students were asked to recall the second word and to write it down on the line. ““We would like you to estimate how many of the word pairs the students forgot after ten minutes/one week/one year. Your estimate can range from 0 to 60 pairs. ... Write down your estimate at the appropriate space at the bottom of the next page.”
Results: Experiment 7
Retention Interval
MeasureTen minutes
One week
One year
P di t d ll 35 2 40 4 37 1Predicted recall:(Exp. 4c)
35.2 40.4 37.1
Predicted recall: 43 4 31 1 18 7Predicted recall:(Exp. 7)
43.4 31.1 18.7
Koriat et al. (2004) conclusions
“… participants can access their knowledge about forgetting…
but only when theory-based predictions are made, and then only when the notion of forgetting is accentuated …y g g
either by manipulating retention interval within individuals, orby framing recall predictions in terms of forgetting rather than remembering ”remembering.
Experiment 1:Experiment 1:Predicting one’s own learning
24 paired associates24 paired associates12 related (Hill-Valley)12 unrelated (Clemency-Idiom)
Number of study/test cycles (between-subject):Number of study/test cycles (between-subject):STSTSTSTSTSTSTSTSTST
Participants judge, pair by pair, the likelihood they will remember that pair on the first, second, third, or fourth cued-recall test cycle
B t bj tBetween-subjects“Complete this sentence: When I make my 0-100% rating, I will do it about test number ___.”
Methodological reasons why predicting one’sMethodological reasons why predicting one s own forgetting and learning might differ
Fi t th i t h b t l i dFirst, there is a match between learning and “remembering”
Analogous to Koriat et al.’s Experiment 7 in which forgetting and “f tti ” t h“forgetting” matchVersus Koriat et al.’s other experiments in which forgetting and “remembering” mismatch
S d h di ti d f th dSecond, when predictions were made for the second, third, and fourth test cycle, it was clear that there could be multiple study/test cycles, which should then trigger th t f l i d ti t th b dthe concept of learning and activate theory-based judgments
100
Actual Recall (easy) Actual Recall (hard)Predicted Recall (easy) Predicted Recall (hard)
70
80
90
100
40
50
60
70
0
10
20
30
Exp. 1 (between-subjects)
01 2 3 4Test
Exp. 1 (between subjects)
100
Actual Recall (easy) Actual Recall (hard)Predicted Recall (easy) Predicted Recall (hard)
100
Actual Recall (easy) Actual Recall (hard)Predicted Recall (easy) Predicted Recall (hard)
70
80
90
100
70
80
90
100
40
50
60
40
50
60
70
0
10
20
30
0
10
20
30
Exp. 1 (between-subjects) Exp. 2 (within-subjects)
01 2 3 4Test
01 2 3 4Test
Exp. 1 (between subjects) Exp. 2 (within subjects)
Concluding comments:Predicting one’s forgetting versus predictingPredicting one s forgetting versus predicting one’s learning
Like Koriat et al ’s resultsLike Koriat et al. s results …participants were sensitive to item differences
Unlike Koriat et al.’s results … within-subject manipulation of number of study-test cycles did not appear to j p y y ppactivate the concept of learning
Why?Perhaps the concept of learning was activated, but people have a faulty and/or incomplete mental model of learning:incomplete mental model of learning:
the benefits of study are underappreciated;and the benefits of tests as learning events are not understood (e.g., Kornell & Son, 2006)
Or perhaps people are captured by item differences, which blocks access toOr perhaps people are captured by item differences, which blocks access to what they know (or don’t know) about learning:
Tentative conjecture: Among the hierarchy of experience-based and theory-based processes, the concept of learning falls below the concept of forgetting.
Concluding comments on our subjective g jexperience as teachers
Egocentrism in social communication
Newton (1990) as a parable of teaching;
Piaget (1962) quotePiaget (1962) quote
Calvin & Hobbes
Piaget (1962)
“Every beginning instructor discovers sooner or later that his first lectures were incomprehensible because he was talking to himself, so to say, mindful only of his point of view. He realizes only gradually and with difficulty that it is not easy to place one’s self in thedifficulty that it is not easy to place one s self in the shoes of students who do not yet know about the subject matter of the course.”
References
Baddeley A D & Longman D J A (1978) The influence of length and frequency of training session on theBaddeley, A.D., & Longman, D.J.A. (1978). The influence of length and frequency of training session on the rate of learning to type. Ergonomics, 21, 627-635. Benjamin, A. S., Bjork, R. A., & Schwartz, B. L. (1998). The mismeasure of memory: When retrieval fluency is misleading as a metamnemonic index. Journal of Experimental Psychology: General, 127, 55-68.Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.), Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press. Bjork, R. A. (1999). Assessing our own competence: Heuristics and illusions. In D. Gopher and A. Koriat (Eds.), Attention and performance XVII. Cognitive regulation of performance: Interaction of theory and application (pp. 435-459). Cambridge, MA: MIT Press. pp (pp ) gBjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. In A. Healy, S. Kosslyn, & R. Shiffrin (Eds.), From learning processes to cognitive processes: Essays in honor of William K. Estes (Vol. 2, pp. 35-67). Hillsdale, NJ: Erlbaum. Jacoby, L. L., Bjork, R. A., & Kelley, C. M. (1994). Illusions of comprehension, competence, and remembering In D Druckman and R A Bjork (Eds ) Learning remembering believing: Enhancing humanremembering. In D. Druckman and R. A. Bjork (Eds.), Learning, remembering, believing: Enhancing human performance (pp.57-80). Washington, DC: National Academy Press.Jacoby, L.L., & Kelley, C.M. (1987). Unconscious influences of memory for a prior event. Personality and Social Psychology Bulletin, 13, 314-336.
References (continued)
Koriat, A., Bjork, R. A., Sheffer, L., & Bar, S. K. (2004). Predicting one’s own forgetting: The role of experience-based and theory-based processes. Journal of Experimental Psychology: General, 133, 643-656.Koriat, A., & Bjork, R. A. (2005). Illusions of competence in monitoring one’s knowledge during study. Journal of Experimental Psychology: Learning, Memory, Cognition, 31, 187-194.Koriat, A., Ma'ayan H., Sheffer, L., & Bjork, R. A. (2006). Exploring a mnemonic debiasing account of the underconfidence-with-practice effect. Journal of Experimental Psychology: Learning, Memory and Cognition, 32, 595-608. Koriat, A., & Bjork, R. A. (2006). Illusions of competence during study can be remedied by manipulations that enhance learners’ sensitivity to retrieval conditions at test. Memory & Cognition, 34, 959-972.that enhance learners sensitivity to retrieval conditions at test. Memory & Cognition, 34, 959 972.Koriat, A., & Bjork, R. A. (2006). Mending metacognitive illusions: A comparison of mnemonic-based and theory-based procedures. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32(5), 1133-1145. Koriat, A., Fiedler, K., & Bjork, R. A. (2006). Inflation of conditional prediction. Journal of Experimental Psychology: General 135(3) 429 447Psychology: General, 135(3), 429-447.Kornell, N., & Bjork, R. A. (2007). The promise and perils of self-regulated study. Psychonomic Bulletin & Review, 14, 219–224.Kornell, N., & Bjork, R. A. (in press). Optimizing self-regulated study: The benefits—and costs—of dropping flashcards. Memory.
References (continued)
Kornell N & Bjork R A (in press) Learning concepts and categories: Is spacing the “enemy ofKornell, N., & Bjork, R. A. (in press). Learning concepts and categories: Is spacing the enemy of induction”? Psychological Science.Maki, W. S. (2007a). Judgments of associative memory. Cognitive Psychology, 54, 319-353.Maki, W. S. (2007b). Separating bias and sensitivity in judgments of associative memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33, 231-237.Newton, L.(1990). Overconfidence in the Communication of Intent: Heard and Unheard Melodies. Unpublished doctoral dissertation, Department of Psychology, Stanford University.Piaget, J. (1962). Play, Dreams and Imitation in Childhood. New York: Norton.Reder, L. M. (1987). Selection strategies in question answering. Cognitive Psychology, 19, 90-138.R h D & T l K (2007) Th h ffli f th ti ti bl b t l iRohrer, D., & Taylor, K. (2007). The shuffling of mathematics practice problems boosts learning. Instructional Science, 35, 481-498.Simon, D. A., & Bjork, R. A. (2001). Metacognition in motor learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 907-912.Ste-Marie, D. M., Clark, S. E., Findlay, L. C. & Latimer A. E. (2004). High levels of contextual interferenceSte Marie, D. M., Clark, S. E., Findlay, L. C. & Latimer A. E. (2004). High levels of contextual interference enhance handwriting skill acquisition. Journal of Motor Behavior, 36, 115-126.