ULA network experience @ JANOG34, by Shishio Tsuchiya [APNIC 38 / APIPv6TF]
NCOLTCL 2011 Brigham Young University Shinsuke Tsuchiya
description
Transcript of NCOLTCL 2011 Brigham Young University Shinsuke Tsuchiya
Elicited Imitation and Automated Speech
Recognition as a Placement Test:
Analysis Of JSL Learners’ Language Development Process And Differences Between Instructed And Self-Instructed Learners
NCOLTCL 2011Brigham Young University
Shinsuke Tsuchiya
OverviewO IntroductionO PurposeO BackgroundO Research QuestionsO Test DesignO ResultsO AnalysisO LimitationsO ImplicationsO Future Research
IntroductionAssessment and Testing using technology can be:O QuickO ReliableO Objective
Brigham Young UniversityO Problem: Placement test for incoming learners is
all multiple-choice questions (near-native learners, instructed and uninstructed learners)
“Can we use Elicited Imitation and Automated Speech Recognition as a speaking portion of the
placement test?”
BackgroundO Elicited ImitationO Automated Speech Recognition
1. Ohayoo gozaimasu ‘Good morning.’
2. _______________“the more you know of a foreign language, the better you can imitate the sentences of the language.” (p. 247) Bley-Vroman & Chaudron (1994)
A viable oral testing method (Erlam, 2006; Graham et al., 2008)
BackgroundEnglish EI
BYU PSST Research Group http://psst.byu.edu (Pedagogical Software and Speech Technology)O 1659 tests were administered 1279 ESL
subjects (Lonsdale & Weitze, 2009)O “overall comparisons between EI scores of
ESL learners and their scores on other measures of oral language proficiency are promising” (Hansen, Graham, Brewer, Brewer, & Tieocharoen)
O “EI scores can predict Oral Proficiency Interview (OPI) scores within two levels of a scale of ten. (Graham, 2006; Lonsdale, Graham, & Madsen, 2005).
BackgroundJapanese EI test using ASR
Over 800 JSL learners have taken the EI.
O the correlations between human grading and ASR grading was r=0.84 (p < 0.001)
O the correlation between Japanese EI score and OPI scores was r=0.77 (p < 0.3)
(Matsushita & LeGare, 2010)
Sound recording ソノナカニナニハイッテンノ ‘sono naka ni nani haitten no’
Student’s utterance ソノナカナニハイッテタショ ‘sono naka ni nani haitteta syo’
Evaluation: C CCCD C CCCCCS SI
Correct = 76.92% 10 ( 10) substitutions = 15.38% 2 ( 2) Deletions = 7.69% 1 ( 1) Insertions = 7.69% 1 ( 1) Errors = 30.77% 4 ( 4)
Research QuestionsCan elicited imitation and automated speech recognition grading system be used as a placement test?
RQ 1. Can they differentiate instructed beginning, intermediate, and advanced learners of Japanese?• If so, what EI items?
RQ 2. Can they differentiate instructed and uninstructed learners?• If so, what EI items?
Test DesignO A total number of participants: 241 BYU JSL learnersO EI was taken in the computer lab in Fall 2010O 3-seconds pause before repeating back (Erlam, 2006)O 60 items (30 from textbooks, 30 from CSJ corpus) (15 min.)
O Gender-neutral contentsO no technical termsO Sentence length: 10-30 moraeO Randomized orderO Prompts (both male and female)
O ASR graded mean scores for each level as well as individual items were compared (t-test)
O Participants were divided into different groups for each research question
Test itemsO Beginning (20)Ex. 明日行きませんか。‘asita ikimasen ka’‘won’t you go tomorrow?’O Intermediate (20)Ex. 昨日は学生さんが手伝ってくれたんですよ。‘kinoo wa gakusee san ga tetudatte kureta n desu yo.’‘As for yesterday, a student helped (me) as a favor.’O Advanced (20)Ex. 日本がどんな国なのか知りたがっている人がたくさんいます。‘nihon ga donnna kuni na no ka siritagatte iru hito ga takusan imasu.‘There are many people who are eager to know what kind of country Japan is.’
Research Question 1Can EI and ASR differentiate instructed beginning, intermediate, and advanced learners of Japanese?
Instructed learners:O *1st year: beginning: 79O 2nd year: intermediate: 34 O **3rd year +: advanced: 21
*False beginners have been excluded**Native speakers and uninstructed learners who have been excluded.
Research Question 2Can EI and ASR differentiate instructed and uninstructed learners?
O *Advanced Instructed learners: 38O **1st semester Uninstructed learners in 3rd year
level: 40
*second half of 2nd year students were added to equalize number of participants.**uninstructed learners who had more than 2 semesters of instruction have been excluded.
Results (RQ 1)Beginning, intermediate, advanced learners’ mean scores were significantly different:Mean scores:O Beginning: 34.7O Intermediate: 45.3O Advanced: 58.8p-valuesO Beginning vs. Intermediate: p=0.000O Intermediate vs. Advanced: p=0.001O Beginning vs. Advanced: p=0.000
Results (RQ 1)Significant itemsAccuracy rate
All progressed:O Beginning-
intermediate40 itemsO Intermediate-
advanced25 itemsO Beginning-advanced58 items
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 590.00
20.00
40.00
60.00
80.00
100.00
120.00
100200300+ (Classroom)
Results (RQ 2)Advanced instructed learners and 1st semester uninstructed learners in 3rd year level were also significantly different. Mean scores:instructed learners: 56.6uninstructed learners: 63.4
p=0.034
Results (RQ 2)Significant itemsAccuracy Rate
17 significant itemsUninstructed learners > instructed learners (16 items)Instructed learners > uninstructed learners (1 item)
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 590.00
20.00
40.00
60.00
80.00
100.00
120.00
Classroom 202+RM 300
Analysis (Summary)1. EI and ASR used in this research can be used to differentiate:O beginning, intermediate, and advanced level
learners.O Instructed and uninstructed learners.2. JSL learners’ accuracy rate improved in each item in general:O beginning level items (beginning to intermediate)O advanced level items (intermediate to advanced)3. Uninstructed learners outperformed instructed learners.
LimitationsTestO Technological problem (background noise, mis-recording, data loss etc.) O Face validity O L2 interlanguage errors that are not
recognized by the present ASR systemO Discouragement SubjectsO Lack of advanced level instructed learners
ImplicationsEI and ASR could be used as a:O Placement test (listening/speaking portion)
only if it is accompanied by other tests. O Computer Adaptive Test (efficient time &
cost with reliable & consistent scores). O Program Review (inter/intraprogram
assessment and evaluation)O Second Language Acquisition Research
Future ResearchTest improvement:O Prevent technological problemsO Increase face validity (include comprehension tasks)O Improve ASR (dealing with L2 interlanguage)O Increase MotivationO More advanced level classroom students
Second Language Acquisition Research:O Item analysis
O Grammatical elements (i.e. case particles)O Sentence complexity and lengthO Vocabulary acquisition
AcknowledgementO Hitokazu Matsushita (Japanese EI/ASR
creator)O Ray Graham, Ray Clifford, Dan Dewey (BYU
Linguistics department)O Robert Russell, Paul Warnick and other BYU
Japanese professorsO BYU Japanese studentsO BYU JFSB Computer lab assistants