ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar...
Transcript of ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar...
![Page 1: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/1.jpg)
ACCOLÉ: A Collaborative Platform of Error Annotation for Aligned Corpora
Emmanuelle Esperança-Rodier, Francis Brunet-Manquat, Sophia Eady
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 2: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/2.jpg)
Evaluation
✦ Golden Reference
2
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 3: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/3.jpg)
Evaluation
✦ Golden Reference
3
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 4: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/4.jpg)
Evaluation
✦ Difficulties
✦ Re-appropriation of evaluation data
✦ Evaluation of tools according to services
4
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 5: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/5.jpg)
Guiding Principle
✦ Competence-based and task-based evaluation of MT systems
✦ Offer users the tools adapted to the task to be performed according to the user skills,
eventually in a tool
5
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 6: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/6.jpg)
Evaluation
✦ « A Comparative Quality Evaluation of PBSMT and NMT using Professional
Translators » Sheila Castilho et al. MT-Summit 2017
✦ « A Challenge Set Approach to Evaluating Machine Translation » Pierre Isabelle et al.
EMNLP 2017
✦ …
6
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 7: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/7.jpg)
Error Analysis
✦ Translation Error Typology (Vilar, 2006)
7
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 8: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/8.jpg)
Error Analysis
✦ Multidimensional Quality Metrics MQM-DQF (Lommel, 2018)
8
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 9: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/9.jpg)
Error Analysis
✦ MeLLANGE Schema d’annotation d’erreurs (Castignoli, 2011)
9
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 10: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/10.jpg)
Annotation tools
✦ Coreference annotator (Tsoumari, 2011)
✦ YAWAT (Germann, 2008)
✦ BLAST (Stymne et al., 2011),
✦ TranslationQ (Steurs, 2015),
✦ QuEst++ (Specia et al., 2015),
✦ Kantan’s MT LQR (KantanMT.com)
✦ PET (Aziz et al. 2012)
10
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 11: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/11.jpg)
ACCOLÉ
✦ A Collaborative Platform of Error Annotation for Aligned Corpora
On line: http://lig-accole.imag.fr/
‣ Annotation according to several typologies
‣ Collaborative
๏ Supervision
๏ Discussion
‣ Search for error type, patterns, occurrence....
‣ Forward semi-automatic annotation
11
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 12: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/12.jpg)
ACCOLÉ: Projects
12
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 13: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/13.jpg)
ACCOLÉ : Projects
13
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
Error Typology
Corpora Vilar DQF-MQM
citi1 citi1 - Vilar/Demonstration
citi2 citi2 - Vilar
News 156 Moses News 156 LIG-Moses - Vilar
News 156 Google News 156 Google - Vilar
News 2000 News 2000 - Vilar
BTEC1-LIGMoses BTEC1 LIG-Moses - Vilar BTEC1 LIG-Moses - DQF-MQM
BTEC1-NMTLIG BTEC1 NMT LIG - Vilar BTEC1 NMT LIG - DQF-MQM
WIPOTranslateN Hawaii Brad Pitt
WIPOTranslateS Madagascar George Clooney
CANADA_eTranslation Nouvelle Calédonie Jude Law
CANADA_LegacyMT@EC New Zealand Robert
doc1_Fujitsu_180118 (monolingue) Altica test 1
doc2_Fujitsu_20180109 (monolingue) Altica test 2
Total 13 6
19
![Page 14: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/14.jpg)
ACCOLÉ : Projects
14
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
Projects CorporaErrorTypologies # phrases # words(source/target) # annotations Creation time
citi1 - Vilar citi1 Vilar 516 13218 / 11857 3265 23/03/17
citi2 - Vilar citi2 Vilar 1382 24402 / 21320 5569 23/03/17
News 156 LIG-Moses - Vilar News 156 Moses Vilar 156 4648 / 3831 982 08/06/17
News 156 Google - Vilar News 156 Google Vilar 156 4648 / 3715 616 08/06/17
News 2000 - Vilar News 2000 Vilar 2000 55368 / 45525 11224 15/06/17BTEC1 LIG-Moses - DQF-MQM BTEC1-LIGMoses DQF-MQM 469 3445 / 3048 0 22/06/17
BTEC1 NMT LIG - DQF-MQM BTEC1-NMTLIG DQF-MQM 469 3445 / 3004 0 22/06/17
BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18
BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar 469 3445 / 3004 801 02/02/18
Démonstration citi1 Vilar 516 13218 / 11857 0 16/05/18
Altica test 1 doc1_Fujitsu_180118 (monolingue) Vilar 41 1104 / 0 2 01/06/18
Altica test 2 doc2_Fujitsu_20180109 (monolingue) Vilar 26 723 / 0 0 01/06/18
Brad Pitt WIPOTranslateN DQF-MQM 9 272 / 305 11 06/06/19
George Clooney WIPOTranslateS DQF-MQM 9 272 / 302 25 06/06/19
Jude Law CANADA_eTranslation DQF-MQM 28 519 / 768 35 06/06/19
Robert CANADA_LegacyMT@EC DQF-MQM 28 519 / 776 51 06/06/19
Hawaii WIPOTranslateN Vilar 9 272 / 305 13 18/06/19
Madagascar WIPOTranslateS Vilar 9 272 / 302 29 18/06/19
Nouvelle Calédonie CANADA_eTranslation Vilar 28 519 / 768 36 18/06/19
New Zealand CANADA_LegacyMT@EC Vilar 28 519 / 776 59 18/06/19
Total 13 2 6817 134273 / 114511 23525
![Page 15: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/15.jpg)
ACCOLÉ: Error Typology
15
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 16: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/16.jpg)
ACCOLÉ: annotation platform
16
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 17: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/17.jpg)
ACCOLÉ: error representation thanks to SSTC
17
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
✦ Trinome :
{Snode Source ; Snode Target; Associated error type}
‣ Occurrences
‣ Morpho-syntactic analysis
![Page 18: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/18.jpg)
ACCOLÉ: search for error type
18
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 19: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/19.jpg)
19
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
ACCOLÉ: search for error type
![Page 20: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/20.jpg)
20
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
ACCOLÉ: search for error type
![Page 21: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/21.jpg)
21
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
ACCOLÉ: search for error type
![Page 22: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/22.jpg)
22
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
ACCOLÉ: search for error type
![Page 23: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/23.jpg)
ACCOLÉ: Example
23
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
![Page 24: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/24.jpg)
ACCOLÉ
24
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
✦ Online Platform
✦ Collaborative Annotation
✦ 2 included typologies (3rd one to come)
✦ 19 projects (13 corpora)
✦ Search tool
![Page 25: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/25.jpg)
ACCOLÉ: further work
25
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue
✦ MWE annotation (Agnès Tutin – UGA/LIDILEM)
✦ Semi-automatic error annotation
‣ Learning by neural network of error annotation with
propagation to other corpora
✦ Learner error Typology (Clive Hamilton – Paris
Diderot/CLILLAC-Arp)
✦ New measures/metrics
✦ Inter-rater agreement
![Page 26: ACCOLÉ: A Collaborative Platform of Error Annotation for ... · BTEC1 LIG-Moses - Vilar BTEC1-LIGMoses Vilar 469 3445 / 3048 807 02/02/18 BTEC1 NMT LIG - Vilar BTEC1-NMTLIG Vilar](https://reader034.fdocuments.us/reader034/viewer/2022050715/5f23b5f89e92246c2e2e1c0e/html5/thumbnails/26.jpg)
Thanks! Any Questions?
✦ Castagnoli, Sara, Ciobanu Dragos, Kübler Natalie, Kunz, Kerstin, Volanschi, Alexandra (2011) Designing a Learner
Translator Corpus for Training Purposes. Dans Corpora, Language, Teaching and Resources: From Theory to Practice.
Édité par Kübler N. Bern: Peter Lang.
✦ Lommel, Arle, and Alan K. Melby. 2018. Tutorial: MQM-DQF: A Good Marriage (Translation Quality for the 21st Century).
13th Conference of the Association for Machine Translation in the Americas (Volume 2: User Papers). Vol. 2.
✦ Steurs Frieda, Winibert Segers and Hendrik Kockaert. 2015. Translation Expert (TranslationQ & RevisionQ): Automated
translation process with real-time feedback & evaluation/ revision with PIE. (Abstract No. Keynote speech 4). Talking to
the World, University of Newcastle, UK. 09 Sep 2015-11 Sep 2015.
✦ Stymne, Sara. 2011. Blast: A tool for error analysis of machine translation output. The 49th Annual Meeting of the
Association for Computational Linguistics. System Demonstrations. 56–61. Portland, OR, USA, Jun.
✦ Tsoumari, Mara and Georgios Petasis. 2011. Coreference Annotator A new annotation tool for aligned bilingual corpora.
Recent Advances in Natural Language Processing. 43–52. Hissar, Bulgaria. 12-14 September 2011.
✦ Vilar, David, Jia Xu, Luis Fernando D’Haro and al. 2006. Error analysis of statistical machine translation output. 5th
International Conference on Language Resources and Evaluation. 97-702.
26
GETALPlangue
linguaj zyk
言語
bahasa
speech
parolepalabra spraak
口 arole
angue