A case study on using speech-to- translation alignments...
Transcript of A case study on using speech-to- translation alignments...
![Page 1: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/1.jpg)
A case study on using speech-to-translation alignments for language
documentation
Antonis Anastasopoulos, David Chiang
![Page 2: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/2.jpg)
http://www.worldmapper.org/images/largepng/583.png 2
![Page 3: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/3.jpg)
Goal
• Collect data now; analyze later
• The data must be:
• Sufficient
• Interpretable
3
![Page 4: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/4.jpg)
How much data?Qur'an
New Testament
Hebrew Bible
All classical Hebrew
All classical Latin
All classical Greek
Millions of words / Hundreds of hours0 6 12 18 24 30
4
![Page 5: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/5.jpg)
The linguistic graveyard
5
![Page 6: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/6.jpg)
Making an audio Rosetta Stone
• Aikuma: Android/web-based app
• Push-to-talk, push-to-translate
6
![Page 7: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/7.jpg)
Interpreting the audio Rosetta Stone
7
![Page 8: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/8.jpg)
Interpreting the audio Rosetta Stone
Now their farm will not stay behind forever.
8
![Page 9: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/9.jpg)
Interpreting the audio Rosetta Stone
Gila abur-u-n ferma hamišaluǧ güǧüna amuqʼ-da-č
Now their farm will not stay behind forever.
9
![Page 10: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/10.jpg)
Interpreting the audio Rosetta Stone
now their farm will not stay behind forever
10
![Page 11: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/11.jpg)
Background
11
![Page 12: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/12.jpg)
K-means Clustering
12
![Page 13: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/13.jpg)
K-means Clustering
12
![Page 14: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/14.jpg)
K-means Clustering
12
![Page 15: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/15.jpg)
K-means Clustering
12
![Page 16: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/16.jpg)
K-means Clustering
12
![Page 17: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/17.jpg)
K-means Clustering
12
![Page 18: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/18.jpg)
K-means Clustering
12
![Page 19: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/19.jpg)
K-means Clustering
12
![Page 20: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/20.jpg)
Aligning Speech to Translation
13
![Page 21: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/21.jpg)
Aligning Speech to Translation
[tanta plata]
13
![Page 22: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/22.jpg)
Aligning Speech to Translation
[tanta plata] [plata]
13
![Page 23: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/23.jpg)
Aligning Speech to Translation
[tanta plata] [plata]
[playa]
13
![Page 24: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/24.jpg)
Aligning Speech to Translation
[tanta plata] [plata]
[Mexico][playa]
13
![Page 25: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/25.jpg)
Aligning Speech to Translation
[tanta plata] [plata]
[Mexico][playa]
13
![Page 26: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/26.jpg)
Aligning Speech to Translation
[tanta plata] [plata]
[Mexico][playa]
Money
Beach
Mexico
13
![Page 27: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/27.jpg)
Example output é Valeria meletá o’ giornále
Valeria legge il giornale [Valeria reads the newspaper]
14
![Page 28: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/28.jpg)
Example output é Valeria meletá o’ giornále
Valeria legge il giornale [Valeria reads the newspaper]
14
![Page 29: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/29.jpg)
Example output é Valeria meletá o’ giornále
Valeria legge il giornale
Score: 0.82
[Valeria reads the newspaper]
14
![Page 30: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/30.jpg)
User Study
15
![Page 31: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/31.jpg)
Griko
16
![Page 33: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/33.jpg)
Resultsph
one
erro
r rat
e (lo
wer
is b
ette
r)
23
24.5
26
27.5
speech-to-translation alignments
none auto gold
24.5
25.7
27
error rates averaged across 6 Italian-speaking participants
18
![Page 34: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/34.jpg)
Resultsph
one
erro
r rat
e (lo
wer
is b
ette
r)
23
27
31
35
speech-to-translation alignments
Italian Spanish English
34.3
28.3
25.7
error rates averaged across all participants (6 Italian, 3 Spanish, 3 English)
19
![Page 35: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/35.jpg)
Consensus transcriptionuser transcription distance
it1 o ladro i so ndze mia buttu 5it2 o ladro isodZenti dabol tu 6it3 o ladro isodzeem biabiddu 5
correct o ladro isodZe embi apo ttu
Can we do better?
Combine them!
20
![Page 36: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/36.jpg)
String averaging
21
![Page 37: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/37.jpg)
String averagingo l a d r o i s o n d z e m i a b u t t u
o l a d r o i s o d Z e n t i d a b o l t u
o l a d r o i s o d z e e m b i a b i d d u
21
![Page 38: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/38.jpg)
String averagingo l a d r o i s o n d z e m i a b u t t u
o l a d r o i s o d Z e n t i d a b o l t u
o l a d r o i s o d z e e m b i a b i d d u
21
0 1 1 … 1
![Page 39: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/39.jpg)
String averagingo l a d r o i s o n d z e m i a b u t t u
o l a d r o i s o d Z e n t i d a b o l t u
o l a d r o i s o d z e e m b i a b i d d u
21
0 1 1 … 1
![Page 40: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/40.jpg)
String averagingo l a d r o i s o n d z e m i a b u t t u
o l a d r o i s o d Z e n t i d a b o l t u
o l a d r o i s o d z e e m b i a b i d d u
21
0 1 1 … 1
![Page 41: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/41.jpg)
String averagingo l a d r o i s o n d z e m i a b u t t u
o l a d r o i s o d Z e n t i d a b o l t u
o l a d r o i s o d z e e m b i a b i d d u
…
21
0 1 1 … 1
![Page 42: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/42.jpg)
String averagingo l a d r o i s o n d z e m i a b u t t u
o l a d r o i s o d Z e n t i d a b o l t u
o l a d r o i s o d z e e m b i a b i d d u
…
Average:
21
0 1 1 … 1
![Page 43: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/43.jpg)
String averagingo l a d r o i s o n d z e m i a b u t t u
o l a d r o i s o d Z e n t i d a b o l t u
o l a d r o i s o d z e e m b i a b i d d u
…
o l a d r o i s o d z e m b i a b u t t u Average:
21
0 1 1 … 1
![Page 44: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/44.jpg)
String averagingo l a d r o i s o n d z e m i a b u t t u
o l a d r o i s o d Z e n t i d a b o l t u
o l a d r o i s o d z e e m b i a b i d d u
…
o l a d r o i s o d z e m b i a b u t t u
5
6
5
3Average:
21
0 1 1 … 1
![Page 45: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/45.jpg)
Resultsph
one
erro
r rat
e (lo
wer
is b
ette
r)
20
25
30
35
subset for averaging
Eng Spa+Eng Spa Ita+Eng all Ita Ita+Spa
23.223.92424.7
26.927.7
32.1
25.6
consensus transcription
22
![Page 46: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/46.jpg)
Conclusions
23
![Page 47: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/47.jpg)
Conclusions• Alignment of speech with its translation is possible
even in small corpora
23
![Page 48: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/48.jpg)
Conclusions• Alignment of speech with its translation is possible
even in small corpora• And *possibly* helpful for manual transcription
(crowdsourced?)
23
![Page 49: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/49.jpg)
Conclusions• Alignment of speech with its translation is possible
even in small corpora• And *possibly* helpful for manual transcription
(crowdsourced?)• What are the needs for interface design?
23
![Page 50: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/50.jpg)
Conclusions• Alignment of speech with its translation is possible
even in small corpora• And *possibly* helpful for manual transcription
(crowdsourced?)• What are the needs for interface design?• Next: large scale case-study with collected data
23
![Page 51: A case study on using speech-to- translation alignments ...aanastas/research/ComputELpresentation.pdf · A case study on using speech-to-translation alignments for language documentation](https://reader033.fdocuments.us/reader033/viewer/2022041705/5e448412399dc96f0a695185/html5/thumbnails/51.jpg)
Conclusions• Alignment of speech with its translation is possible
even in small corpora• And *possibly* helpful for manual transcription
(crowdsourced?)• What are the needs for interface design?• Next: large scale case-study with collected data
23
Talk to me if you want to share comments/ideas/data!