Understanding and Applying Valid Metrics Guidelines February, 2011
Applying Machine Translation Metrics to Student-Written Translations
description
Transcript of Applying Machine Translation Metrics to Student-Written Translations
![Page 1: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/1.jpg)
Applying Machine Translation Metrics to Student-Written
Translations
Lisa N. MichaudComputer Science Department
Merrimack CollegeNorth Andover, Massachusetts, USA
Patricia Ann McCoyLanguage Department
Universidad de las Americas PueblaPuebla, Mexico
![Page 2: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/2.jpg)
Michaud and McCoy3
Criteria for judging translations
fluency (is it well-formed?)
fidelity (does it convey original meaning?)
(Hovy et al., 2002)
![Page 3: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/3.jpg)
Michaud and McCoy4
Multiplicity of translations
In each one of these jobs the professor could have agreed to work 6 hours a day and therefore would not be surpassing the working day hour limit.
In each one of these jobs the teacher could have agreed to work 6 hours per day and therefore he wouldn't be bound by the limits of the working day.
In each of these examples the teaching could have been arranged so that he/she works six hours a day and would not be affected by any workday limitations.
In both of these jobs the professor could have agreed to work six hours daily and therefore he wouldn't be affecting his work shift limit.
![Page 4: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/4.jpg)
Michaud and McCoy5
Multiplicity of translations
In each one of these jobs the professor could have agreed to work 6 hours a day and therefore would not be surpassing the working day hour limit.
In each one of these jobs the teacher could have agreed to work 6 hours per day and therefore he wouldn't be bound by the limits of the working day.
In each of these examples the teaching could have been arranged so that he/she works six hours a day and would not be affected by any workday limitations.
In both of these jobs the professor could have agreed to work six hours daily and therefore he wouldn't be affecting his work shift limit.
![Page 5: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/5.jpg)
Michaud and McCoy6
BLEU
Hypothesis
Multiple References
![Page 6: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/6.jpg)
Michaud and McCoy7
TERp
Hypothesis
Single Reference
PHRASALEQUIVALENCE
orSYNONYM SHIFTSUBSTITUTION
INSERTION
SAMESTEM
![Page 7: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/7.jpg)
Michaud and McCoy8
TERp alignment and tags
![Page 8: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/8.jpg)
Michaud and McCoy9
Student translation corpus
Number of Subjects 13Native English Speakers 3Native Spanish Speakers 10Number of Articles Translated 11Avg Number of Sentences per Article
28
Total Translated Sentences 2,982
![Page 9: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/9.jpg)
Michaud and McCoy11
Does TERp agree with an expert?
Instructor Scores vs Inverted TERp650 sentences (22%)
Pearson Correlationr = 0.232236
![Page 10: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/10.jpg)
Michaud and McCoy12
Score distribution
0 10 20 30 40 50 60 70 80 90 1000
50
100
150
200
250
300
TERp-AInstructor
Assigned Grade Decile
Num
ber
of S
ente
nces
Rec
eivi
ng
Gra
de
![Page 11: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/11.jpg)
Michaud and McCoy13
Instructor rubric (original)
Conveys Original Meaning 55%Written in Natural Language 20%Uses Appropriate Vocabulary 10%Written in Accurate Language 15%
10 Excellent9 Good8Satisfactory0-7 Deficient
![Page 12: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/12.jpg)
Michaud and McCoy15
Evaluating TERp tags (pilot)
Precision
Recall
Phrase equivalence 83% 68%Stemming 100% 75%Synonymy 89% 65%Shifts 92% 89%
![Page 13: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/13.jpg)
Michaud and McCoy16
Future work
![Page 14: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/14.jpg)
Michaud and McCoy17
Instructor rubric (revised)
Uses Grammatical Language 50%Conveys Original Meaning 50%
100 Excellent90 Good80Satisfactory0-70 Deficient
![Page 15: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/15.jpg)
Michaud and McCoy18
Modifying the TERp Score
Hypothesis
Single Reference
PHRASALEQUIVALENCE
orSYNONYM SHIFTSUBSTITUTION
INSERTION
SAMESTEM
![Page 16: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/16.jpg)
Michaud and McCoy19
Recognizing false cognates
Hypothesis
Single Reference
SUBSTITUTION
cynical
brazen
Sourcecínico
![Page 17: Applying Machine Translation Metrics to Student-Written Translations](https://reader035.fdocuments.us/reader035/viewer/2022062520/568164d9550346895dd7231b/html5/thumbnails/17.jpg)
Michaud and McCoy20
Extracting mistranslation pairsSPANISH
DICTIONARYENGLISH
DICTIONARY
cynical
cínicocynicalbrazen
zona zone