QUALETRA “QUALITY IN LEGAL TRANSLATION” WS 4 Testing, Evaluation & Assessment KU Leuven...
-
Upload
whitney-thorton -
Category
Documents
-
view
214 -
download
0
Transcript of QUALETRA “QUALITY IN LEGAL TRANSLATION” WS 4 Testing, Evaluation & Assessment KU Leuven...
QUALETRA“QUALITY IN LEGAL TRANSLATION”
WS 4Testing, Evaluation & Assessment
KU LeuvenAntwerpen
16-17 October 2014
QUALETRAJUST/2011/JPEN/AG/2975
With financial support from the Criminal Justice Programme of the European Commission Directorate General Justice
© Hendrik J. Kockaert & Winibert Segers
Translation Evaluation
Translation product (target text, the result of the translation process)
Translation process (think aloud protocol, eye tracking, key logging)
Translation service (contact with the client, offering the quotation, invoicing, compliance agreements, complaints, etc.)
© Hendrik J. Kockaert & Winibert Segers
Translation EvaluationTranslation product: methods
Holistic method Analytical methodCDI method (Calibration of Dichotomous Items)PIE method (Preselected Items Evaluation)
Translation EvaluationTranslation product: methods
Research Reports:How do evaluators carry out the following evaluation methods?
(1) Holistic(2) Analytical (3) PIE
Research ReportsTranslation Evaluation
Holistic & Analytical MethodsResearch Method
A sample translation from English to Spanish of a UK robbery judgment was allocated to ten evaluators who received a translation brief (structured translation specifications)
Shortly after submitting their assessment, the evaluators were asked to complete a questionnaire
These case studies were carried out by UAH (Holistic method) and DCU (Analytical method)
© Hendrik J. Kockaert & Winibert Segers
Research ReportTranslation Evaluation
PIE Method
Preselected Items Evaluation
Translation briefLT specific criteriaEssential documentsEAWECQA criteria
Case study
KU Leuven carried out a case study where 46 level A professional translators from federal ministries in Belgium did a translation from French to Dutch
The text contained 56 elements, including punctuation. Seven preselected items were used
An intern corrected the translations using the analytical evaluation method, referring to the error categories of the ATA evaluation grid, without implementing the actual ATA format and criteria
The standard error of difference between the scores obtained on the basis of the analytical method and the PIE method was 0.177, which is not statistically significant
Research ReportTranslation Evaluation
PIE Method
Sample TestJUGEMENT CORRECTIONNELDEMANDE DE MISE EN LIBERTÉDEBATS Avant l’audition de [PI 1] XXX, le président a constaté que celui-ci ne parlait pas suffisamment la
langue française ;Il a désigné YYY, interprète inscrit sur la liste du tribunal ; l’interprète a ensuite prêté son
ministère [PI 2] chaque fois qu’il a été utile.A l’appel de la cause [PI 3], le président a donné connaissance de [PI 4] l’acte qui a saisi le tribunal
[PI 5] et constaté la présence et l’identité de XXX, dont [PI 6] il a reçu les déclarations [PI 7].
Maître ZZZ, conseil [PI 8] du prévenu, a été entendu [PI 9] en sa plaidoirie.Le ministère public a été entendu en ses [PI 10] réquisitions.Le prévenu a eu la parole en dernier.Le greffier a tenu note du déroulement des débats.
© Hendrik J. Kockaert & Winibert Segers
PIE: Case StudyStatistics: Analytical Score vs. PIE Score
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 400.0
2.0
4.0
6.0
8.0
10.0
12.0
PIE Score (../10)Analytical Score
© Hendrik J. Kockaert & Winibert Segers
PIE: Case StudyStatistics: T-Test
P value and statistical significance: The two-tailed P value equals 0.1376By conventional criteria, this difference is considered to be
Not statistically significant
Confidence interval:The mean of Group One minus Group Two equals 0.268 95% confidence interval of this difference: From -0.090 to 0.626
Intermediate values used in calculations:t = 1.5151df = 40standard error of difference = 0.177
© Hendrik J. Kockaert & Winibert Segers
Conclusion:
No statistically significant difference between analytical scores and PIE scores
PIE proves a justifiable evaluation method
objectivitytransparencyequality
Research ProposalsTranslation Evaluation
PIE Method
ActivitiesTranslation Evaluation
ECQA Test Design
ECQA Test on three successive stages:
(1) MC questions on legal knowledge (specifically EDs and EAWs)
(2) MC and open questions on legal language (phraseologies, terminology, style, register), possibly including a recognition test (to recognise whether a proposed translation is correct or not); after passing each of the two previous stages
(3) A translation test corrected with PIE
Survey disseminated, analysed and survey report
Legal Translation Product Quality Assurance
(1) Three research reports drafted (holistic, analytical and PIE method)
(2) Strategy and methods have been decided for reporting on the evaluation methods
(3) Submission of “Objective Translation Evaluation through PIE” (KU Leuven), 1st International Young Researchers’ Conference on Translation and Interpreting (UAH, 7-8/11/2013)
Deliverables
No. Deliverable name/type (a) Format (b) Target group (d)
1 Survey on the testing, assessment and evaluation on the current legal translation practices in criminal proceedings in the EU (product, translator and service/process quality assurance)
Websites and manuals (updatable and online information on product, translator and service/process quality assurance)
Translator trainers Legal practitioners Professional associations
Deliverables
When you evaluate legal translations, how important do you find the following?
Very Im
portant
Importa
nt
Neither
Importa
nt nor U
nimporta
nt
Unimporta
nt
Not at a
ll Importa
nt0
10
20
30
40
50
60
70
80
90
100
Being able to differentiate between good and bad translationsAn objective evaluationThe efficiency of the overall evaluation pro-cedure
1 (highest ranking)
2 3 4 5 6 (lowest ranking)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Evaluation by comparing the translation to the source text and highlighting any errorsEvaluation of the translation of preselected segments in the source textEvaluation by means of a checklist or an evaluation gridEvaluation based on the general impression of the quality of the translation
Which evaluation method do you find most suitable for legal translations?
How important do you find the following?
Equiva
lence
betwee
n source
text
and ta
rget te
xt
Faith
fulness to
the s
ource te
xt
Lingu
istic c
orrectn
ess
Meaning t
ransfe
r
Idiomatic c
orrectn
ess
Style
and re
gister
Readab
ility o
f the t
arget
text
Term
inology0
20
40
60
80
100
120
Not at all ImportantUnimportantNeither Important nor UnimportantImportantVery Important
Which are the most important issues in evaluating legal translations?
1 (highest ranking)
2 3 4 5 6 7 (lowest ranking)
0
10
20
30
40
50
60
70
80
90
100
Having a suitable source text for legal translation evaluationDetermining if the candidate translator passes or failsDetermining the severity of errorsApplying the same criteria to each translationAvoiding subjective evaluation
How can the evaluator justify the evaluation?
0102030405060708090
100
AgreeNeutralDisagree
Test Formats
Three different testing formats:
Translation testRevision test Recognition test
Certification Procedure
Prerequisite
Only European Master’s in Translation Network (EMT) graduates may apply to take the ECQA certification examination.
Examination
The examination consists of 3 stages
A candidate must pass each element of each stage before passing to the next stage
A candidate who fails one or more elements of a stage only has to repeat the failed element(s)
ExaminationThe 3 stages of the examination are:
Multiple choice questions: LTiCP.U1 Legal KnowledgeU1.E1 Criminal law and procedure U1.E2 Monolingual legal terminology (in both languages) Internet access is not permitted at this stage
Open questions: LTiCP.U2 Professional aspects & LTiCP.U3 Instrumental competenceU2.E1 Professional practice U2.E2 Professional conductU3.E1 Information acquisition U3.E2 Legal terminology managementInternet access is not permitted at this stage
Translation: 2x250-word texts, one for each element of LTiCP.U4 TranslationU4.E1 Translation of essential documents (Directive 2010/64/EU) U4.E2 Translation of European Arrest WarrantsInternet access is permitted at this stage
ECQA Skills Card
Skill Card Type of ExaminationLTiCP.U1 Legal Knowledge
U1.E1 Criminal law and procedure MCQ
U1.E2 Monolingual legal terminology (in both languages) MCQ
LTiCP.U2 Professional aspects
U2.E1 Professional practice Open Questions
U2.E2 Professional conduct Open Questions
LTiCP.U3 Instrumental competence
U3.E1 Information acquisition Open Questions
U3.E2 Legal terminology management Open Questions
LTiCP.U4 Translation
U4.E1 Translation of essential documents (Directive 2010/64/EU) Translation
U4.E2 Translation of European Arrest Warrants Translation
Translation Test
maximum 2x250-words PIE method
Test FormatPIE Items
Report a lost or stolen passport [1] You must report the loss or theft of your passport as soon as possible, even if you don’t
want to replace it immediately. You’ll need to complete a Lost or Stolen Notification form [2]. Find out how to get the form and what to do if your passport is lost abroad.
If your passport is lost or stolen in the UK [3] You must report the loss or theft of your passport to the Identity and Passport Service [4]
(IPS) [5]. This will reduce the risk of anyone else using your passport or your identity. To report your passport lost or stolen, fill in and sign a Lost or Stolen (LS01) [6] Notification form [7] and return it to IPS [8]. The address is on the form.
You should [9] report all passport thefts to the police. You’ll need the crime reference
details for the LS01 form [10].
Revision Test
maximum 10 items maximum score = 10
Test FormatCandidates have to revise the English translation; have to recommend corrective measures for elements in the English translation that are not correct. Vérifiez la date du document.Check the data of the document. A candidate can make three types of erroneous revision:
1) The candidate revises an element in the translation that was correct, and proposes, for instance, to replace ‘Check’ by ‘Control’ or by ‘Verify’;
2) The candidate does not revise an element in the translation that was not correct: in our example the word ‘data’, which has another meaning then the French word ‘date’ (‘date’);
3) The candidate revises an element in the translation that was not correct, but her/his revision is not correct, and proposes, for instance, to replace ‘data’ by ‘information’.
Recognition Test
maximum 10 itemsmaximum score = 10
Example
Candidates have to indicate if the proposed English translation is correct or incorrect. Le témoignage de l’épouse de l’accusé n’est pas crédible.The testimony of the wife of the accused is not credible. correct incorrect
Test FormatPIE Translation, Revision, Recognition
After passing the three tests (translation, revision and recognition), the candidate will obtain a score with a maximum of thirty
The proposed testing approach is a better guarantee for objectivity than the traditional translation test
The score of the candidate does not depend on the personal appreciation and interpretation of the evaluator
Another advantage of this testing approach is the time-saving aspect: The time you spend to develop the three tests is largely compensated by the short evaluation time
Validity of Tests
Candidate 1 Candidate 2 Candidate 3 Candidate 4 Candidate 5 Candidate 6 Candidate 7 Candidate 8 Candidate 9
0
5
10
15
20
25
Score
RecognitionRevisionTranslation
ECQA Overview Question TypesSample
Learning Elements to be covered consistent with skill card
Type of Examination Developer
CLTiCP.U1Professional competence
U1.E1Professional practice Open Questions
U1.E2Professional conduct Open Questions
ECQA QuestionsSamples
LTICP.U1.E1.PC2: The candidate has a sound understanding of criminal procedure in the legal systems involved (e.g. levels of jurisdiction, legal structures, institutions, settings, parties).
1. A person who __________________ a crime may be sentenced to imprisonment. Fill in the blank with one of the four choices:
A doesB is convictedC rendersD performs
ECQA QuestionsSamples
1.1. LTICP.U1: Legal knowledge1.1.1. LTICP.U1.E1: Criminal law and procedure
CLTICP.U5.E1.PC5: The candidate masters the main domains and sub-domains of criminal law, especially the most frequent offences in essential documents and European Arrest Warrants, e.g. drugs, fraud and theft.
1. From the following list, choose one example of an offence against the person:
A theft B smugglingC assaultD possession
ECQA QuestionsSamples
LTICP.U1.E1.PC3 The candidate has a general awareness of current legal issues and their development in the relevant countries.
1. From the following list, choose one example of a current legal issue of special concern in the EU cross-border criminal context:
A Illegal sale and distribution of food products B Trafficking in human beingsC Electronic surveillance by state security servicesD Non-payment of fines
ECQA QuestionsSamples
LTICP.U1.E1.PC4: The candidate is familiar with the EU directives on legal translation.
1. A criminal suspect has the right to translation of (choose one of the following):
A Questions from the judgeB Prosecution statementsC Transcripts of evidence given in court by witnesses D Other