A New Aptitude Test - GRAL Seminar...A New Aptitude Test • CASL Research on Language Aptitude DLAB...
Transcript of A New Aptitude Test - GRAL Seminar...A New Aptitude Test • CASL Research on Language Aptitude DLAB...
A New Aptitude Test• CASL Research on Language Aptitude
DLAB 2 Increase prediction of basic proficiency (ILR 2) & predict attritionALAB Language neutral testHi-LAB Predict ultimate attainment (ILR3+/4)ATI Aptitude-by-treatment interaction: Match training to aptitude
Theoretical Framework Develop a research program for getting USG language professionals to very high levels of proficiency?
Aptitude‐by‐Treatment Interaction!
Little research directly investigating aptitude for high‐level language.
Uses of Aptitude Tests
3
• Selection
• Diagnosis
• Placement
• Counseling
• Tailored Training (ATI)
• Predicting performance outside classroom
• SLA Research
4
Predicting Ultimate AttainmentProblem
Need to identify individuals who can attain near‐native proficiency, given motivation and opportunity.
RelevanceAdult language learning is time consuming, effortful and expensive.
SolutionCognitive abilities that underlie advanced language learning processes are measurable. High levels of cognitive abilities may predict near‐native attainment.
What predicts LL success?
Talent: Cognitive
Ability
Experience
Motivation
Personality Facets
Many factors
Natural abilityDrivePerseveranceTrainingCoaching
“L2 aptitude is the individual’s initial state of readiness and capacity for learning a foreign language, and probable facility in doing so [given the presence of motivation and opportunity].”
J. B. Carroll, 1981
7
Second Language Aptitude
Talent: Cognitive
Ability
What language aptitude explains
• Speed of language acquisition
under intensive learning conditions (rate)
(Carroll and others following)
• Individual differences in
post‐CP ultimate attainment (ceiling)
(CASL)
8
Carroll’s 4 Aptitude Constructs
DLAB MLAT PLAB
Phonetic Coding Ability
Part 2, Recognition of Stress PatternsPart 3, Foreign Language Grammar
Part 2, Phonetic Script Part 3, Spelling Clues
Part 5, Sound DiscriminationPart 6, Sound-Symbol Association
Grammatical Sensitivity
Part 3, Foreign Language GrammarPart 4, Foreign Language Concept Formation
Part 4, Words in Sentences
Part 4, Language Analysis
Rote Learning Ability
Part 3, Foreign Language Grammar
Part 1, Number Learning Part 5, Paired Associates
Part 4, Language Analysis
Explicit Inductive Language Learning Ability
Part 4, Foreign Language Concept Formation
Part 1, Number Learning
Part 4, Language Analysis
9
Hi‐LAB’s unique prediction
• Hi‐LAB is designed to predict the ultimate level of attainment by adult starters.
• While recognizing the importance of other individual differences and experience
• to success in language learning, the purpose of Hi‐LAB is to measure cognitive aptitude.
12
Summary
• Critical period offsets disrupt language learning.
• Empirical studies show wide variation in ultimate attainment post‐critical period.
• Successful individuals may be those with language aptitude which helps to counteract critical period effects.
14
InstrumentationHi‐LAB is a more direct battery
•Innovative: computer‐delivered behavioral measures (E‐prime CRO)
•Data types: accuracy, reaction time
•Not transparent to learners
•Instructions provided & response type and individual responses PRACTICED
•Fun!?
Hi‐LAB Constructs and MeasuresHi-LAB Constructs and MeasuresWorking Memory
Executive Functioning
Updating Running Memory Span
Inhibitory Control Antisaccade
Stroop Simon
Task Switching Task Switching Numbers (mix cost and switch cost)
Phonological Short-term Memory Letter Span
Non-Word Span
Associative Memory Paired Associates
Long-term Memory Retrieval (Priming) ALTM Synonym
Implicit Induction (have added explicit) Serial Reaction Time (Letter Sets)
Processing Speed Serial Reaction Time (could be from other measures)
Auditory Perceptual Acuity Phonemic Discrimination: Hindi, English Pseudo-Contrastive
Phonemic Categorization: Russian
15
Our iterative research approach
• Propose new constructs (motivated by theory)
• Develop measures, assess usabilityusability revise
• Gather evidence for reliabilityreliability revise
• Gather evidence for validityvalidity revise
• Gather information about testing environment construct operational tests
Aims of Reliability Research
• Improve the reliability of Hi‐LAB tests
• Reduce the size of the test battery to fit within reasonable testing time limit
(7 2.5 1.5 hours)
• Regularize scoring procedures
• Examine theoretical construct validity
(factor analysis)
17Doughty et al (2007) (RS1) and (2009) (RS2)
Study MethodParticipants
• RS1: 195 NSs of English (UMD)• RS2: 142 NSs of English (DLIFLC)Materials
• 42 measures 22 measures (based on RS1)• 2 parallel versions of the battery: Hi‐LAB Version A and Hi‐LAB Version B
Design
18
TestAB 2 weeks + Re-test
BA
Hi‐LAB ConstructsCognitive Construct Factor Reliable Stable
Jointly Acceptable
Short Term Memory 2 of 2 1 of 2 1 of 2
Updating 2 of 2 1 of 2 1 of 2
Inhibition ? 2 of 2 1 of 2 1 of 2
Task‐switching 2 of 2 1 of 2 1 of 2
Rote Memory 2 of 2 1 of 2 1 of 2
Perceptual Acuity 3 of 4 2 of 4 2 of 4
Speed 2 of 2 1 of 2 1 of 2
Semantic Primability 2 of 2 1 of 2 1 of 2
Implicit Induction 1 of 2 1of 2 1 of 2
Explicit Induction Letter Sets
20
20
Simon
Aims of Hi‐LAB Validity Research
• Phase 1: Factor analyses to theoretical construct validity
• Phase 2: Group discrimination
• (Phase 3: Gather longitudinal validity evidence )
• Phase 4: Operational validation (October 2013)
Mislevy et al. 2010 - AUA
Study Method
Aim – Investigate whether Hi‐LAB discriminates groups with known levels of L2 ability
(Pairs – propensity score matching;
Group discrimination – logistic regression)
Moderately Successful Group
Highly Successful Group
Age Gender Level of Education
Linck, Hughes, Campbell, Silbert, Tare, Jackson, Smith, Bunting & Doughty (in press, Language Learning 2013).
Hi‐LAB correctly classifies high‐level and normal attainment
GROUPListening (n = 152)
Reading(n = 188 )
Either(n = 206)
High‐Level vs.Normal Attainers
**70% **59% **67%
the working memory system
** p < .01
Validity: Hi-LAB is accurate in group discrimination
Linck, Hughes, Campbell, Silbert, Tare, Jackson, Smith, Bunting & Doughty (in press, Language Learning 2013).
Best predictors in regression modelListening Reading Either
Task switching
Letter Span Letter span Letter span
Paired Associates Paired Associates Paired Associates
Serial Reaction Time Serial Reaction Time Serial Reaction Time
24Linck, Hughes, Campbell, Silbert, Tare, Jackson, Smith, Bunting & Doughty
(in press, Language Learning 2013).
Further Test Development
• Can the aptitude dimensions be further reduced?
• Is language aptitude distinct from IQ?
25
Study methodParticipants
• 428 NSs of English
Analyses• Principal components analysis
• Hierarchical and non‐hierarchical cluster analyses
26
Language Aptitude distinct from IQ?CQBN Measures (n = 132)
Hi-Lab Measures (n = 428) Variable NameDescription
Processing Speed CQBN1PMPattern Matching
Serial Reaction Time CQBN2MT Matrices
Phonemic Discrimination CQBN3EU English Usage
Phonemic Categorization CQBN4WM Wording Meaning
Phonemic Cat. Learning CQBN5SPSpelling
Mix-Cost RT CQBN6DRDirections
Switch-Cost RT CQBN7AR Arithmetic
Anti-Saccade Accuracy CQBN8NN Name & Number Check
Stroop CQBN9AL Artificial Language
Letter Span
Running Span
Non-Word Span
ALTM
28
Is language aptitude unique?
29
Hi‐LAB CQBN
Hi‐LAB + CQBN
Significant
Non‐significant
Hi-LAB accounts for substantial unique variance
Quartimax rotated loadings, 4 Factors, Hi‐LAB only
Measure Factor 1 Factor 2 Factor 3 Factor 4
Processing.Speed.TS0.761
Serial.Reaction.Time
Phonemic.Disc0.576
Phonemic.Cat0.630
Phonemic.Learn0.364
MixCost.RT‐0.564
SwitchCost.RT0.514 0.404
Anti.ACC‐0.522
Stroop.RT
Letter.Span0.959
Running.Span0.512
NonWord.Span
ALTM‐0.528
30
Conservative and less conservative analyses of # of components to retain
31
• CQBN 2 or 8 of 9 components
• Hi‐LAB 4 or 10 of the 13 components
• Hi‐LAB
and CQBN 7 or 21 of the 22 components
Theoretical Framework : ATI
Assess effectiveness
Apply treatments (training)
Identify groups based on aptitudes
Collect data on participants
All participants
Group A
Training A
Training B
Group B
Training A
Training B
36
37
Aptitude‐by‐Treatment InteractionProblem
Individuals who are already selected for intelligence and motivation and have attained functional ability need to acquire higher levels of language
RelevanceHigher‐level ability is essential to the job
SolutionMatch cognitive aptitude profile to language training procedures to optimize training and boost outcome levels
QuestionTo what extent do aptitude profiles exist?
Cluster Analyses
• Solutions from hierarchical and non‐hierarchical clustering algorithms indicate that some clusters may usefully be defined with respect to average aptitude profiles.
38
Aptitude Profiles: 3 Cluster Solution: (1) all high (2) all low (3) mixed (speed)
40
Mixed group strength = memory and perceptual acuity
4 Cluster Solution
41
1 Cluster good almost across the board; 4th poor3rd cluster: good at only STM, Updating, Perceptual Acuity4th cluster: good only at Speed, Induction, Inhibition, Priming, STM
Steps to develop ATI materialsTAILORING SECOND LANGUAGE TRAINING: STEPS
Step 1. Identify a learning problem relevant for the learners’ and instructors’ needs
Step 2. Consider the learners’ aptitude profile card
Step 3. Consider practical constraints
Step 4. Design the training
Step 5. Determine the language outcome measures (pre‐ and post‐training tests)
Step 6. Implement the training
42
Example of ATI: Variety in speech
• L2 language use need: Interact with people speaking many registers and dialects of the target language.
• SLA principle: Rich and varied input is necessary for second language acquisition to high levels.
• General learning principle: Complex information is best learned in a structured way at first, moving on to variety.
• Aptitude interaction: People with high aptitude can benefit from the variety sooner. People with low aptitude may need to learn only in the structured format.
43
How to choose which aptitudes and treatments to investigate?
• Information about the participants– Focus on abilities that will impact language use– Target abilities that differ most
• Information from science– Research‐based training techniques– SLA measures (in addition to proficiency)– Stronger predictions for real‐world performance, beyond the lab
47
Triangulation: Needs and constraints
48
Needs analysis• Use• Learning
Practical limits on interventions• time available• time for design &
implementation• instructor role• accessibility• expertise availability
Aptitude profiles• distribution• groupings?• aptitude predictions
testable by intervention
Needs Analysis
• Russian is a highly‐inflected language complex inflectional morphology
– Instruction bias: focus on vocabulary, don’t pay attention to the ’endings’
– Students develop a strategy to ignore inflectional morphology
49
Needs Analysis
• Grammatical case especially problematic: – Nominal inflectional morphology marks grammatical roles of nouns in
a sentence;
– Russian allows scrambling/limited constraints on word order inflections are the only reliable marking of grammatical function.
• What makes it worse:‐ existing L1 bias – English has strict SVO word‐order;‐ a universal processing strategy to interpret (or to assign) the role of
the agent to the first noun (for L2: VanPatten’s First Noun Processing Strategy, FNPS);
‐ Shallow Structure Processing hypothesis (Clahsen) – L2 semantic processing bias (e.g., animacy)
50
Pre‐testPerception TasksTask 1: Picture/audio match (CASES/WO) (auditory presentation)‐ to assess whether students can correctly interpret the meaning of SVO and OVS
sentences.
Мама читаетмальчику. Мальчик читает маме.
The Mom.NOM reads to the boy.ACC The boy.NOM reads to the Mom.ACC
51
Pre‐test
Task 2: Plausibility task (CASES/WO) (visual presentation)‐ similar to Task 1, but allows for inanimate objects to be subjects in grammatically
correct sentences. Мужчину держит ложка.
The man.ACC holds the spoon.NOM
Task 3: Auditory Grammaticality Judgment task (CASES)‐ assesses how sensitive learners are to case violations (e.g., two nouns both
marked for nominative case).
На завтрак бабушка варила кашу.
For breakfast the grandmother.NOM was cooking oatmeal.ACC.
52
Pre‐test
Production TasksTask 1: Picture description/translation (CASES/WO) (visual presentation)• uses English/Russian translation to force participants to produce both SVO and
OVS sentences with appropriate inflections.
Prompt: dog/girl/brush
Собаку причесывает девочка.The dog.ACC brushes the girl.NOM
Task 2: Elicited imitation (CASES/WO) (auditory presentation)‐ are able to correctly reproduce grammatical inflections.
53
Aptitudes• Implicit/Explicit learning designated as possibility
• Profiles (Hi‐LAB) confirmed split in class, though not fully crossed
• All students showed strength in Explicit Learning
• Half also showed strength in Implicit Learning
• Intervention design adjusted to take this into account
54
Intervention challenges
• “Case” too broad for narrow intervention• Instrumental case chosen• Within‐subjects design needed• Two target constructions for instrumental available for contrast: passive subjects and instruments
• Explicit intervention: what instructor already had planned
• Implicit intervention: “murder mystery” logic puzzles
55
Design details
• Study design:– All participants received both interventions– Implicit intervention only on instruments (not passive subjects)
– Matched training (implicit training for implicit aptitudes) should show greater benefit
– Training designed to be useful for all participants (just more useful for some)
– Existing instruction plans disturbed very little
56
Intervention details
• Critical design features:– Implicit: no overt instruction on or excessive attention towards instrumental case
– Meaning‐based: students needed to comprehend the statements for meaning to engage in the task
– Exposure: frequent use of weapons in sentence provided high‐density exposure to the instrumental case, using instruments
57
Murder mystery
• Like “passive Clue”
• Had to guess murderer, location, weapon, etc.
• Students listened to a number of statements (between 4 and 15)
• Used witness statements to eliminate possibilities
58
1. The mechanic.NOMki lled the cook.ACC Механик убил повара.2. The nurse.NOMki lled the cook.ACCwith a belt.INSTR Медсестра убила кого‐то ремнём.3. The nurse.NOM ki lled the victim.ACC in the school.PREPwith a spade.INSTR Медсестра убила кого‐то в школе лопатой.4. Someone.NOMki lled the professor.ACC in the school.PREP Кто‐то убил профессора в школе.5. The mechanic.NOMki lled the victim.ACCwith a spade.INSTR Механик убил кого‐то лопатой.6. The nurse.NOMki lled the professor.ACC Медсестра убила профессора.7. The mechanic .NOM ki lled the professor.ACCwith a belt.INSTR Механикубил профессораремнём.
Outcome measure details
• Auditory grammaticality judgment task (GJT)
• Taps into knowledge of students without asking specific questions on case
• Auditory chosen to match auditory intervention
• Quickly designed and implementable
• Provides specific measure for outcomes
59
Procedure
• Monday: Pre‐test GJT
• Tuesday, Wednesday, Thursday: Implicit instruction,3 x 20 minutes
• Friday: Mid‐test GJT (after the implicit instruction)
• Later on Friday: Explicit instruction, 1 hour• Tuesday (following week): Post‐test GJT (after the explicit instruction)
• 3 weeks later: Delayed post‐test GJT
60
Results
• Learners with explicit and implicit learning profiles take advantage of both types of instruction, and (tentatively) benefit more from the combination of the two.
• Learners with an explicit aptitude profile also show a gain in accuracy as a result of the explicit instruction.
• The information on the individual differences among learners predicts different learning outcomes as a result of methodological intervention
feasibility of effective task tailoring to individual strength for accelerated learning outcomes
61
RESEARCH PROBLEM
• Learners have incomplete knowledge of lexical items, and when they sound similar to other items, they confuse the two.
• ATI study will leverage participants’ aptitude profiles to determine whether instruction designed to match their aptitudes will promote increased mastery of confusable words as measured by accuracy & automaticity of perception and retrieval of words.
62
• What constitutes exceptional ability?
• How many different aptitude profiles are there?
• At what level should the intervention be – approach, technique?
• Are there genetic correlates of language aptitude?
64 64
Solutions?
• Select people with talent, that is cognitive aptitude for language
• Tailor training to aptitude profiles, language learning needs, and workplace language needs
• Train the cognitive aptitude
• Train the job performance
• Coach the job performance
65