MiTiN 2013 Keynote in Detroit Michigan

Copyright © 2009, Asia Online Pte Ltd Copyright © 2009, Asia Online Pte Ltd

The Impact of MT on the Translator

Kirti Vashee – [email protected]

http://www.twitter.com/kvashee http://kv-emptypages.blogspot.com

mailto:[email protected]

http://www.twitter.com/kvashee

http://kv-emptypages.blogspot.com/



Copyright © 2009, Asia Online Pte Ltd

The Increasing Importance of Technology & Automation

A Content Explosion Across The Globe

The Emergence of Social Media and Social Networking as Business Drivers and Influencers

New Open Innovation & Collaboration Business Models

A Rising Asian Market Changing Global Enterprise Priorities


0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

20

09

20

10

20

11

20

12

20

13

20

14

20

15

20

16

20

17

20

18

20

19

20

20

Total Exabytes of Information

Source: IDC Digital Universe Study, May 2010

2009 = 800 Million Petabytes 1 PB = 1,000,000 GB

More information was created in 2005 than in the previous 40,000 years !


As these conversations become increasingly

independent of these sites, falling traffic will

render them ineffective in their current form. Instead, the online

presence of each brand will necessarily expand

out into the social space to stay in touch with their

audience.

Simon Mainwaring


Assisted Self-Service Communities

Source: Consortium for Service Innovation

Customer Exceptions

Community Conversations Knowledge Base

Web Portal User Initiated Groups Support Center

Product Management

Development/ Engineering

X

10X

30X

Copyright © 2009, Asia Online Pte Ltd Source: Consortium for Service Innovation

Self-Service Support 100,000 @ $10/exception

Community Support 300,000 @ $1/exception? 90-95% of

Activity

5-9% Activity

1-3% Activity

Indirect Support

Direct Support

Assisted Support 10,000 @ $250/case

Corporate Investment and focus

Customer Interactions

95% Activity


• Real Time Search & Find Mode

• Information acquired as needed

• Comprehensive & dynamic knowledge base

• Continuously Updated

• Static Reference Material

• Long Shelf Life

• Just In Case

• Mandatory and necessary

• Information flow from company to consumer

• Human Filtered Information

• Expert Identification

• Trust agent based information gathering

• Continuously flowing and changing and often community based


Global enterprises face a content deluge with dynamic content coming from both internal and external sources

High volumes of content expected to be translated increasingly faster and faster

Customers increasingly in control of marketing and brand messages

A shift from corporate messaging to customer conversations and authentic communications

More Content, Faster Turnaround Times, Lower Cost


What We Translate – More Dynamic Real-Time Content

Why We translate – From Mandatory to Increase and Expand Communication with Customers

How We Translate – More Automation, MT and Open Collaboration Models

Highly Personalized Content to Customers when they need it in a variety of digital forms

More Content, Continuous, Faster Turnaround, Cheaper

Project Based TEP Continuous Streams


User Generated Content

Support / Knowledge Base

Communications

Enterprise Information

User Documentation

User Interface

Products

Corporate Corporate Brochures

Product Brochures

Software Products

Manuals / Online Help

HR / Training / Reports

2,000

10,000

50,000

200,000

500,000

10,000,000

20,000,000+

50,000,000+

Email / IM

Call Center / Help Desk

Blogs / Reviews

Example Words Human

Machine

Existing Markets $31.4B

New Markets


• Growth in Word Volume for Traditional Localization Projects

• Faster Turnaround Time Requirements

• Changing Translation Price-Value Expectations

• Increasing Acceptance of MT by Enterprise Buyers

• New Rapidly Growing Types of Content – Patents & Scientific Content – Customer Support & Care Content – Customer Conversations – User Generated Content


• Traditional Localization Projects – Documentation and Localization

• Focused on improving translation productivity • Same quality deliverable but faster and cheaper

• New MT Enabled Projects – Patents & Scientific Content

• Huge Volume – Hundreds of millions of words

– Customer Support & Care Content • Very high value but short-lived • Technical Support & Knowledge base

– Customer Conversations • Editing work only, focused on corrections.


Linguistic Translation Quality

Target Quality (TEP Level)

Raw MT Output Quality

The effort and linguistic work done to raise RAW MT to target quality levels is PEMT

Common Misperceptions • The target quality level is always the same as TEP or other HT standards

• The raw MT output quality is consistent from system to system

• The corrective effort is always the same from language to language

• There is little or no “a priori” control on the MT output quality

• MT error patterns are consistent from segment to segment


Target Quality

MT Output Quality Varies

Pre-Analysis of Source Material Linguistic Profiling and Identification of Key Patterns Terminology Standards Development Translation Quality Source Cleanup

Error Pattern Identification Error Pattern Correction Unknown Word Handling Development of Linguistic Rules Expansion of vocabulary Development of TL Style & Expression Data Corrective Feedback Process Development Raw Corrections Amplification

• MT engines (especially SMT-based ones) get better with feedback

• MT is not exactly the T of the TEP process

• MT engines require upfront investments and analysis for best results

• MT engines differ from language to language (FIGS easier than CJK)

• MT error patterns can vary from segment to segment


How do you pay post-editors fairly if each engine is different?

Tools Needed:

• Effective Quality metrics – Automated – Human

• Confidence scores – Scores on a 0-100 scale – Can be mapped to fuzzy TM match

equivalents

• Post Edit Quality Analysis – After editing is complete or even

while editing is in progress, effort can be easily measured


MT System Quality

Characteristics – Productivity Implications

Free Online Engines Can be useful in some languages but often lower productivity than using TM alone and impossible to adapt to specific needs 1,000 to 3,000 Words/ Day per human editor Average segment quality = ~25 to 40% TM Fuzzy Match

Human TEP Process Typically produce 2,500 Words / Day per translator

Low Quality - Moses Less than 5% of these systems can outperform free online MT and best case productivity may be in the 3,000 Words/Day range Average segment quality = 50% - 60% TM Fuzzy Match

Average Expert System

These systems can provide 5,000 to 7,000 Words/Day per editor Average segment quality = 60% - 75% TM Fuzzy Match

Superior Expert These systems can provide 9,000 to 12,000 Words/Day per editor

Average segment quality = 70% - 85% TM Fuzzy Match

Exceptional MT These systems can provide 12,000+ Words/Day per editor Average segment quality = 80% - 90% TM Fuzzy Match


Data Cleaning

Data Preparation

Data Collections

Training

Diagnostics and Fine Tuning

Customer Translation Data and Linguistic Assets

Translate

Quality Assurance

Language Pair Foundation Data

Domain Foundation Data


Identify Language Pair

Identify Top Level Domain

Upload Your Data

Receive Tuning and Test Set

File

Process Data

Select Best 3000 Segments Train Engine

1

2

3 4

5

7 6

Ready to Translate

Client Asia Online Your Data Bilingual Translation Memories In domain historical translations in source and target language.

Bilingual Dictionaries and Glossaries In domain and client specific glossaries and dictionaries.

Source Language Non-Translatable Terms Source language terms such as product names and place names that should not be translated.

Target Language Monolingual Data Monolingual target language text and URLs of in-domain websites.

Quality Improvement

Plan

8

Extra Data (If Available)

Source Material To Be Translated Source material can be analyzed and processed to further improve quality.

Style Guides Rules can be added to match client style guide requirements.


LP Source Human Reference Customized Foundation

JA-EN なお, 以下の座標系の定義は以下の通り。

Definitions pertaining to the coordinate systems are given below.

Furthermore, the definition of the coordinate systems are as follows.

Furthermore, the following coordinate system as defined.

JA-EN

せん断試験の管理特性を規定し判断基準は明確か

Are the control characteristics of shearing test defined to specify criteria for judgement clearly?

Are the control characteristics of shear test defined to specify criteria for judgement clearly?

Shear test criterion for defining characteristics of the clear?

JA-EN

ベントチューブスポット溶接の強度は確認しているか

Is the strength of spot-welds on vent tubes checked?

Is the strength of spot-welds on vent tubes checked?

It is the intensity of the welding spot vent tubes?

EN-DE An alternate host can start the meeting and act as the host.

Alternative Gastgeber können das Meeting starten und als Gastgeber handeln.

Alternative Gastgeber können das Meeting starten und als Gastgeber handeln.

Stellvertretendes Gastgeber beginnen können und so zu tun, als die Tagung des Aufnahmelandes.

EN-DE You can publish a recorded training session that was created with WebEx Recorder.

Sie können eine aufgezeichnete Schulungssitzung veröffentlichen, die mit dem WebEx-Rekorder aufgezeichnet wurde.

Sie können eine aufgezeichnete schulungssitzung veröffentlichen, die mit dem WebEx-Rekorder erstellt wurde.

Sie können eine namentliche Fortbildungsveranstaltung veröffentlichen, mit WebEx Fahrtenschreiber.

EN-DE Once customer approves your request, the customer can select an application to share.

Wenn der Kunde Ihre Anforderung genehmigt, kann er eine Applikation zum Teilen auswählen.

Wenn der Kunde Ihre Anforderung genehmigt, kann der Kunde eine Applikation zum Teilen auswählen.

Wenn Verbraucher stimmt ihrem Antrag, der Kunde auswählen können, einen Antrag zu teilen.

EN-ES Remove the steel ball from the main oil gallery before cleaning.

Retire la bola de acero de la canalización de aceite principal antes de limpiar.

Retire la bola de acero de la canalización de aceite principal antes de la limpieza.

Eliminar la bola de acero de la limpieza galería antes de petróleo.

EN-ES Continuously with the ignition on and the propulsion system active.

Continuamente con el encendido conectado y el sistema de propulsión activo.

Continuamente con el encendido en posición on y el sistema de propulsión activo.

Continuamente con la ignición en activo y el sistema de propulsión.

EN-ES The average response time goal is assigned a specific time goal.

El objetivo del tiempo de respuesta medio se asigna a un objetivo de tiempo específico.

El objetivo del tiempo de respuesta medio se asigna a un objetivo de tiempo específico.

La meta media del tiempo de respuesta se asigna una meta del momento específico.

Customization teaches an engine how to translate using YOUR style and vocabulary


Foundation Engine Output Custom Engine Output Frame at cherry tree massive (20 millimeter thickness); anticata finishing. Colourless glass, satin, bisellato and decorated.

Solid cherrywood frame (thickness 20 millimeter); antique finishing. White glass, satin, beveled and decorated.

Bodywork in chipboard class E1 (18 millimeter thickness) nobilitato melaminico; finishes cherry tree. Borders on the ABS and edges in such a laminatino; finishes cherry tree.

Carcass in Class E1 chipboard (thickness 18 millimeter) melamine-coated panel; cherrywood finishing. Edges in ABS and basic laminate edges; cherrywood finishing.

Back pain fibre-( 3 millimeter thick); primer; white. For pivot elements with glass or at the request, MoF (2.5 millimeter thickness) nobilitato melaminico; finishes cherry tree.

Back Panel of fibre (thickness 3 millimeter); lacquering; white colour. For elements with glass door or upon request, melamine MDF (thickness 2.5 millimeter) coated; cherrywood finishing.

At the request, flanks on sight in chipboard class E1; veneering with sliced cherry; painting acrylic polyurethane and.

Upon request, visible sides in class E1; chipboard veneering with cherrywood sliced veneer; acrylic and Polyurethane painting.

Anodized aluminium for sottolavelli and columns refrigerator. Protection from condensation and leakage of water.

Anodised aluminium for under sinks and tall units for built-in fridge. Protection against condensation and water losses.

Release the furniture into the environment. Contact holdings disposing of solid urban waste, in accordance with the municipal regulations waste. For household appliances, consult the SPC with the guarantee.

Not dispose of furniture in the environment. Contact the company responsible for disposal of urban solid waste, in compliance with municipal waste management regulations. For household appliances, consult the producer's technical specifications and relevant guarantee.

Comparison of Foundation and Custom Engine Output – IT to EN


A method of distilling a polymerizable vinyl compound selected from the group consisting of acrolein, methacrolein, acrylic acid, methacrylec acid,

hydroxyethyl acrylate, hydroxyethyl methacrylate, hydroxypropyl acrylate, hydroxypropyl methacrylate, glycidyl acrylate and glycidyl methacrylate, the

method comprising distilling the polymerizable vinyl compound in the presence of a polymerization inhibitor using a distillation tower having

perforated trays without downcomers and wherein the temperature of the inner wall of the tower is maintained at a temperature sufficient to prevent the condensation of the vapor being distilled, whereby the polymerizable

vinyl compound is distilled without the formation of polymer.

Actual sample of Japanese to English MT output • Requires a significant terminology database effort • Special handling for long sentences • Monolingual target language analysis • Linguistic parsing


Initial System put into production

All users allowed to suggest changes which goes through vetting process

Changes are collected and added to initial corpus to drive continuous retraining

Trained Internal Experts begin initial clean up and correction process

Expert Users also allowed to make changes

Engine Learning Iteration 1 2 5 4 3 6

Publication Quality Target

Post Editing Effort

Qu

alit

y

Raw MT Quality

Post-editing effort and cost can be managed by improving the quality and performance of the MT engine via corrective linguistic feedback


Original Source File

Machine Translate

Raw Machine Translations

Human Post Editing

Post Edited Translations

Send Raw MT and Post Edited Translations

back to Asia Online

2

1

2

1

Data Analysis and Manufacturing

Incremental Quality

Improvement Supported File Types


Translator 1

Translator 2

Translator 3

Translator 4 MT + Post Editing

Human Only

12,000 10,000 8,000 6,000 4,000 2,000 0 Words Per Day

• Productivity improvement results differ by translator. The above data is derived by studying 4 different translators productivity used only human and then with the addition of MT + human post editing by professionals

• Weaker translators often tend to benefit more from technology

• Customization is key to minimizing translator frustration

• Rapid measurement and assessment of quality is key to profitability


MT System Quality

Characteristics – Productivity Implications

Free Online Engines Can be useful in some languages but often lower productivity than using TM alone and impossible to adapt to specific needs 1,000 to 3,000 Words/ Day per human editor Average segment quality = ~40% TM Fuzzy Match

Human TEP Process Typically produce ~2,500 Words / Day per translator

Low Quality - Moses Less than 5% of these systems can outperform free online MT and best case productivity may be in the 3,000 Words/Day range Average segment quality = 50% - 60% TM Fuzzy Match

Average Expert System

These systems can provide 5,000 to 7,000 Words/Day per editor Average segment quality = 60% - 75% TM Fuzzy Match

Superior Expert These systems can provide 9,000 to 12,000 Words/Day per editor

Average segment quality = 70% - 85% TM Fuzzy Match

Exceptional MT These systems can provide 12,000+ Words/Day per editor Average segment quality = 80% - 90% TM Fuzzy Match


Standard TEP Excellent Moses

Average Expert

Excellent Expert

Translated Words / Day

2,500 3,000 6,000 9,000

Hourly Rate $45 $45 $45 $45

Word Rate 15 cents 12 cents 10 cents 7.5 cents

Daily Cost at Hourly Rate $360 $360 $360 $360

Daily Cost at Word Rate $375 $360 $600 $675

500,000 Word Project

Hourly Cost $72,000.00 $ 60,000.00

$30,000.00

$20,000.00

Word Rate Cost $75,000.00 $ 60,000.00

$50,000.00

$37,500.00

Man Days 200.00 166.67 83.33 55.56


• The productivity gain or loss can be calculated by counting the average number of words translated per hour.

• This productivity information can be used to determine PEMT compensation rate.

• Test Example: 4 Hours with 1 Translator (EN > DE)

• Productivity Gain: 328% • For a mature custom engine 328% is not uncommon and represents

productivity gains similar to using TM fuzzy matches of 75%-80%.

• Understand the quality of the MT engine output BEFORE you accept or reject work

Total Words

Words Per Hour

Human Translation 1,253 313

MT + Human Post-Editing 4,116 1,029

Difference (+ Gain / - Loss) +2,863 +715


Linguistic Steering

Pattern Identification, Corpus Analysis, Linguistic Problem Solver, Quality

Assessment, Linguistic Asset Development and Test & Tuning Set Development

MT-Savvy Translators & Editors

Rapid Error Identification / Correction

Manufacture Corrective Data and Drive Early Development of MT Engines

Less Skilled Editors with SME and Target Language Skill

Can be Monolingual, Students, Housewives

Monolingual Data Cleanup

N-gram Resolution and Preparation


• Training of post-editors – New Skills – MT Post Editing Is Different to HT Proof Editing

• Different error patterns and different ways to resolve issues • Some LSPs are creating e-learning courses for post editors

• 3 Kinds of Post Editors – Professional Bilingual MT Post Editors:

• Often with domain expertise, these editors have been trained to understand issues with MT and not only correct the error in the sentence, but also create learning material

– Early Career Post Editors: • Editing work only, focused on corrections

– Monolingual Post Editors • Experts in the domain, but may not be fluently bilingual • With a mature engine, this approach will often deliver the best, most

natural sounding results


S

– Original Source: • The original sentences that are to be translated.

– Human Reference • The gold standard of what a high quality human translation would look like.

– Translation Candidate • This is the translated output from the machine translation system that you are comparing.

S

R

C

Machine Translate Compare and Score

Multiple machine translation candidates can be scored at one time to compare against each other. E.g. Asia Online, Google, Systran

Note: C

3 Measurement Tools • Human Quality Assessment • Automated Quality Metrics • Sentence Evaluation

Original Source

Translation Candidate

Human Reference

R C


• The test set being measured: Different test sets will give very different scores. Very small test sets can give misleading results.

• How many human reference translations were used: If there is more than one human reference translation, the resulting BLEU score will be higher.

• The complexity of the language pair: Spanish is a simpler language in terms of grammar and structure than Finnish or Chinese.

• The complexity of the domain: A patent has more complex text and structure than a children’s story book. It is not practical to use two different test sets and conclude that one translation engine is better than the other.

• The capitalization of the segments being measured: When comparing metrics, the most common form of measurement is Case Insensitive.

• The size of the test set: Use 1,000 or more BLIND segments to get good assessments

• The measurement software: There are many measurement tools for translation quality. Each may vary slightly with respect to how a score is calculated

It is clear from the above list of variations that a BLEU score number by itself has no real meaning.


Excellent (4)

Read the MT output first. Then read the source text (ST). Your understanding is not improved by the reading of the ST because the MT output is satisfactory and would not need to be modified (grammatically correct/proper terminology is used/maybe not stylistically perfect but fulfills the main objective, i.e. transferring accurately all information.)

Good (3)

Read the MT output first. Then read the source text. Your understanding is not improved by the reading of the ST even though the MT output contains minor grammatical mistakes .You would not need to refer to the ST to correct these mistakes.

Medium (2) Read the MT output first. Then read the source text. Your understanding is improved by the reading of the ST, due to significant errors in the MT output . You would have to re-read the ST a few times to correct these errors in the MT output.

Poor (1)

Read the MT output first. Then read the source text. Your understanding only derives from the reading of the ST, as you could not understand the MT output. It contained serious errors. You could only produce a translation by dismissing most of the MT output and/or re-translating from scratch.

Evaluation Criteria of MT output


Human evaluators can develop custom error taxonomy to help identify key error pattern problems .


Corpus Analysis & Preparation Pattern Identification Linguistic Structural Analysis Linguistic Problem Solving

Linguistic Production Process Management Translation & MT Engine Quality Assessment

Rapid Quality Assessment Effective Use and Development of Automated Measurements Steering Guidance to MT Developers

Rapid Error Detection & Correction Open minded translators Better translator workbenches and tools Skilled monolinguals with subject matter expertise (SME)

Community Management Recruiting Quality Management


1 Gram 2 Grams 3 Grams 4 Grams 5 Grams

Just

a

short

walk

from

times

square

just a

a short

short walk

walk from

from times

times square

just a short

a short walk

short walk from

walk from times

from times

square

just a short walk

a short walk from

short walk from times

walk from times

square

just a short walk from

a short walk from times

short walk from times

square

Just a short walk from Times Square

The following n-grams, or word strings, can be generated from this sentence.

Identify high frequency phrases (n-grams) to translate


Before Machine Translation

Pre-Translation Corrections (PTC) - A list of terms that adjust the source text fixing common issues and making it more suitable for translation.

Non-Translatable Terms (NTT) - A list of monolingual terms that are used to ensure key terms are not translated.

Runtime Glossary (GLO) - A list of bilingual terms that are used to ensure terminology is translated a specific way.

After Machine Translation Target text is processed and modified.

Post Translation Adjustment (PTA) - A list of terms in the target language that modify the translated output. This is very useful for normalization of target terms.

Each of the above runtime customizations can be applied in 2 forms: Default: Applied to all jobs. Job Specific: A different set of customizations can be applied for different clients.

Source text is processed and modified.


Original Source Corrected Source

PrecisionTMWorkstations Precision™ Workstations

ChinaSingaporeSydney China, Singapore, Sydney

Hyper-VTM Hyper-V™

6TBExternal 6TB External

w/ with

TO Q1 TO QUESTION 1

— <wall/>:<wall/>

(\d)'|"(?=[ ](HD|disp|SAS|SATA)) ${1}-inch

• Support for case sensitive and case insensitive matches • Support for regular expressions


Term

New York Times

PCs Limited

Asia Online Pte Ltd

Fortune 500

John Jacob

Microsoft Office

Cisco Local Director

Man Yee Wai


Original Source Specified Translation

Portugal-Portuguese Portugais (Portugal)

Independent Software Vendor (ISV) éditeurs de logiciels indépendants (ISV)

South Holland Province La Province Hollande-Méridionale

Proof of Concept (POC) engagement mission de validation technique

HBA adaptateur de bus hôte

Fine print Clauses complémentaires

Standup HBA adapter pour adaptateur de bus hôte

HBA standup adapter pour adaptateur de bus hôte


Original Target Adjusted Target

double port 2 port

double-port 2 port

deux port 2 port

deux-port 2 port

I5 i5

e/s E/S

cloud computing Cloud Computing

ompm OMPM


Content Type Target Quality Process Volumes

Legal, Marketing, Mandatory

High Human Translation, TEP

Low

Reference, KB Moderate Custom MT + Professional Post-

Editing

High

User Generated Content

Moderate to Low Custom MT + Community Post-

Editing

Very High

Random Corporate Content

Low - Gisting Custom Corporate MT

High

Random Web Content

Low - Gisting

Free MT 150 Billion Words in Google Translate

in 2010

Match the production process to the value, volume and quality configuration of the content


• Product Training Materials

• Manuals & Documentation

• Design & Research

• Sales & Marketing

• Emails & Website

Internal Corporate Content

• Training Materials

• Customer Feedback

• Customer Care & Support

• Customer Blogs & Forums

• Social Network Content

External Partner & Customer

Content

Customize MT Engine

Translate & Refine MT Engine

Post-Edit and Correct

Analyze Source

Simplify & Clean

Build and Leverage Linguistic Assets for Translation Production Lines

Different Target Quality: TEP, MT+ Post Editing, Custom MT, Raw Corporate Baseline MT

Prioritize for Translation Process

Develop Linguistic Profiles of Key Content

Communicate

Listen & Learn

Distribute & Share


• What content has the greatest value for our target audience?

• How do we get it translated quickly, at the highest quality possible at the lowest cost possible?

• How do we build infrastructure that enables emerging new content to be quickly translated as needed?

• From localization projects to flowing streams of high value customer related content


Any Agency not using MT competently in 5 years time will be marginalized or be a niche player.

In 5 years time, leading Agencies will be translating more content in 1 year than in the previous 5 years combined.

There will be more demand for translators than ever before, but roles will evolve and change.


Kirti Vashee – [email protected] Follow Me on Twitter: @kvashee Join the Automated Language Translation Group in LinkedIn

www.kv-emptypages.blogspot.com

Thank You

mailto:[email protected]




http://www.kv-emptypages.blogspot.com/



MiTiN 2013 Keynote in Detroit Michigan

Business

Transcript of MiTiN 2013 Keynote in Detroit Michigan