March 10, 2005 METEOR 1

Translation Error Rate Metric

• Reference: “the Iraqi weapons are to be handed over to the army within two weeks”

• MT output: “in two weeks Iraq’s weapons will give army”

March 10, 2005 METEOR 2

Step-1: Remove Stop Words

• the Iraqi weapons are to be handed over to the army within two weeks

• in two weeks Iraq’s weapons will give army

March 10, 2005 METEOR 3

Step-2: Word Align with Matcher

• the Iraqi weapons are to be handed over to the army within two weeks

• in two weeks Iraq’s weapons will give army

March 10, 2005 METEOR 4

Step-3: Count Flips (Reorder)

• the Iraqi weapons are to be handed over to the army within two weeks

• in two weeks Iraq’s weapons will give army

• Reordered:• in Iraq’s weapons will give army two weeks• Flip count = 4

March 10, 2005 METEOR 5

Step-4: Count Insertions/Deletions/Substitutions

• the Iraqi weapons are to be handed over to the army within two weeks

• in two weeks Iraq’s weapons will give army

• Reordered:• in Iraq’s weapons will give army two weeks• Flip count = 4• Substitutions: 2• Insertions (from Ref): 5• Deletions (from MT): 1

March 10, 2005 METEOR 6

Step-5: Score

• the Iraqi weapons are to be handed over to the army within two weeks

• in two weeks Iraq’s weapons will give army

• Reordered:• in Iraq’s weapons will give army two weeks• Flip count = 4• Substitutions: 2• Insertions (from Ref): 5• Deletions (from MT): 1• TER = flips + subst + ins + del / ref length =

4+2+5+1/12 = 12/12 = 1.0

March 10, 2005 METEOR 7

Another Example

• Ref: north korea is prepared to allow washington to verify that it does not manufacture nuclear weapons

• MT: North Korea prepared to allow Washington to the verification of that is to manufacture nuclear weapons

• Flips: 0• Substitutions: 3• Insertions: 1• Deletions: 0• TER Score = 0+3+1+0/16 = 4/16 = 0.25