Critical Review: Chun, C. W. (2006). An Analysis of a Language Test for Employment: The Authenticity...
-
Upload
jake-heyer -
Category
Documents
-
view
215 -
download
2
Transcript of Critical Review: Chun, C. W. (2006). An Analysis of a Language Test for Employment: The Authenticity...
Critical Review: Chun, C. W. (2006). An Analysis of a Language Test for Employment: The Authenticity of the PhonePass Test
Tianfei (Faye) Jiang & Peipei (Christina) DongCTL 3799 Communication and L2 Learning in WorkplaceProf. Julie KerekesFebruary 11, 2014
OUTLINE
• Background & Context• Characteristics of the PhonePass test– Procedure– Setting
• Frameworks• Analysis of the PhonePass test• Conclusion• Critique & Implications• Questions & Comments
BACKGROUND
• Global technology has quickly progressed to provide more accessible business modes of telecommunication, like telephone, video phone, emails and other internet/computer-mediated system in workplace.
• Tests over telecommunication networks are usually highly commercialized..
CONTEXT
PhonePass Spoken English Test-10 (SET-10)• WHAT IS IT? An expedient assessment tool for screening job candidates’ ability
in spoken English• WHAT IS IT FOR? Evaluating employees for promotions and measuring service
employees with an accurate and objective test of their spoken English spoken capabilities.
• HOW DOES IT WORK? This test is a 10 minutes spoken English test for non-native English speakers, which is administered over a language-line telephone and immediately scored by computer. It consists of 5 sections, based on four diagnostic subscores which includes sentence mastery, vocabulary, fluency, and pronunciation. The cost ranges from $25 to $40.
• WHEN & WHERE IS IT? Anytime and Anywhere with a land-line phone system available.
• HOW IS IT SCORED? The first four sections are recorded and scored by computers, while the last section is not scored, which is open for review.
CONTEXT
Purpose:to see if the characteristics of the
domain of the test tasks found in the PhonePass SET 10 test correspond to real life domains of nontest tasks.
• Key Words: telecommunication; spoken language
test; authenticity;
CHARATERISTICS IN PHONEPASS TEST
Access to an
Internet-connected Computer
Download and Print out a Test
Paper
Access to a Land-line
Telephone
Follow Further
Instruction in the Phone
Testing
• PROCEDURE (requires employee’s computer technology & telephone technology capabilities)
CHARATERISTICS OF PHONEPASS TEST• SETTING
• “Please read No. 3.”• “The winter in Toronto is freezing.”
A: Reading Aloud
• “War broke out.”• “There are three basic ways in which a story
might be told to someone.”
B: Repeating Sentence
• “What season comes before spring?”• “Does a blanket go on the bed or the wall?”
C: Giving Short Answers to Question
• “in/ bed/ stay”• “we wondered/ would fit in here/ whether the
new piano”
D: Building Sentences
• “Which would you prefer to live, in the big city or a small town? And why?”
E: Responding to Open Questions
CHARATERISTICS OF PHONEPASS TEST
• Any location & time • High degree of speed• 10 minutes
• visual and aural channels
• Grammatical knowledge
• Textual cohesion
• Sentence mastery• Vocabulary• Fluency• Pronunciation• No reciprocal tasks
Flexibility & Short Length Input
ExpectationsTasks
FRAMEWORKAuthenticity• “the degree of correspondence of the characteristics of a given
language test task to the features of a target language use task.” (Bachman & Palmer, 1996, p.23)
• “the developers need to show that the content of the test tasks is representative of the demands of the corresponding task outside the test situation, and that the scoring reflects this.” (Luoma, 2004)
• “authenticity of input, purpose, and outcome” (Lewkowicz, 2000)
Test Fairness Framework “access, administration, validity, absence of bias, and social
consequence.” (Kunnan, 2004)
ANALYSIS: Authenticity
Rationale PhonePass Test (PART A)
Created real-life discourse Structured speaking task
Grammatical Knowledge Without correct sentence production and comprehension
Textual Knowledge Read in random order of items
Functional and Sociolinguistic Knowledge
Standard pronunciation
ANALYSIS: Authenticity
Rationale PhonePass Test (PART B)
Proof of comprehension Sounds reorganization and repetition & Memory
Speech production Without discourse creation and interpretation
ANALYSIS: Authenticity
Rationale PhonePass Test (PART C)
Vocabulary of professional employment level
low level of lexical knowledge
Extended production responses Limited production responses
Sociolinguistic knowledge N/A
ANALYSIS: Authenticity
Rationale PhonePass Test (PART D)
Fluency (constructing, reading and repeating the sentences)
Inadequate constructing
Pragmatic knowledge N/A
Sentence mastery One in six chance of guessing word groups
ANALYSIS: Authenticity
Rationale PhonePass Test (PART E)
Authenticity Extended production responsesNot scored
ANALYSIS: Fairness
Rationale PhonePass Test (PART A)Minimal bias No biased content or language
Educational access (speaking & listening)
Advantages for test takers who have had more opportunities to learn and become familiar with the tasks on the test
Telecommunication access (land-line telephone, printer & internet connected computer)
The absence of supporting telecommunication in underdeveloped countries
Appropriate accommodations Uncertain physical (e.g. sight and hearing) impairment
Rescoring and reevaluation N/A
CONCLUSIONS
• The performance on these tasks does not necessarily predict or reflect the speaking ability of the test taker to function in real-life domain of workplace.
• PhonePass SET 10 is considered relatively less authentic.
• The test tasks are relatively less specific to a particular domain and are not interesting and engaging.
CRITIQUE
Strengths: The author used strong theoretical basis to support his
analysis. It provides a valuable evaluation as a reference, emphasizing
the importance of authenticity, to language assessment in workplace.
Weakness: This article does not offer further practical implication in
telecommunication in workplace. It would be better if the author provides more constructive
suggestions to the phone language test.
IMPLICATIONS
• Since telecommunication lacks non-verbal compensation, such as body languages and facial gestures, the requirement of language is much higher than daily face-to-face communication.
• This analysis brings readers to have more deep provoking thoughts about telecommunication networks, for example, how can we communicate effectively without these compensation, and how can we minimize the misunderstandings in telecommunication.
• For employers, phone interview or language assessment via telephone should focus more on the authentic communicative competence than on linguistic performance.
• Communication effectiveness is a critically important criteria in assessing individual's language ability in workplace. Standards for communicative language capability in workplace is needed to be further explored.
Questions AND/OR Comments?
----- Tianfei Jiang & Peipei DongFebruary 11, 2014