Good at listening or good at listening tests? Dr John Field CRELLA, University of Bedfordshire, UK...
-
Upload
ethen-chambley -
Category
Documents
-
view
220 -
download
0
Transcript of Good at listening or good at listening tests? Dr John Field CRELLA, University of Bedfordshire, UK...
Good at listening or good at listening tests?
Dr John FieldCRELLA, University of Bedfordshire, UK
Faculty of Education, Cambridge University
ANUPI 2013 Huatulco
With thanks to Oxford University Press for supporting
this talk as part of their Professional Development
programme
Professional Development3
Building bridges: research into practice
• Action research
Teacher investigates own practice
• Academic research into ELT / SLA
Methodology
Learner performance / development
• Applying research from other domains
General education – Discourse analysis – pragmatics Sociology – Psycholinguistics – Social psychology
Phonetics and phonology – vocabulary studies
Professional Development4
Building bridges: research into practice
• Action research
Teacher investigates own practice
• Academic research into ELT / SLA
Methodology
Learner performance / development
• Applying research from other domains
General education – Discourse analysis – pragmatics Sociology – Psycholinguistics –Social psychology
Phonetics and phonology – vocabulary studies
Professional Development5
Good at listening or …?
• A cognitive approach to testing listening• What learners tell us about listening tests• What is listening?• Recent developments in the testing of L2
listening
A cognitive approach
Professional Development7
Listening: where teaching and testing merge
Listening is a process taking place in the mind of the listener.
The only way we can find out if understanding has occurred
is indirectly, by asking questions.
This applies to checking understanding in the classroom as well as to designing tests.
So what follows has implications for teachers as well as testers.
The need for a cognitive approach
• There is a new interest among testers in what goes on in the mind of the test taker.
• We need to know whether high-stakes test actually test what they claim to test. Can a listening test, for example, accurately predict the ability of a test taker to study at an English medium university? Work in the health service? Survive as an immigrant?
• At a local level, we need to use tests to diagnose learner problems so that the tests can feed into learning. This is especially true of listening.
Cognitive validity asks…
• Does a test elicit from test takers the kind of process that they would use in a real-world context? In the case of listening, are we testing the kinds of process that listeners would actually use ?
• Or do the recordings and formats that we use lead test takers to behave differently from the way they would in real life?
Two possible research approaches
• A. Ask learners to report on the processes they adopted when taking a test (e.g. by explaining how they got their answers)
• B. Use a model of listening that is supported by evidence from psychology. Match the processes produced by a test against the model.
Listening to listeners
Professional Development12
Listening and reporting
• A listening passage is paused after about every third (every 70 secs out of 3.5 mins.).
• Learners report their answers for this section• Learner explain why they gave the answer.• After a short piece of listening, learners’ recall is fresh and
accurate.• Answers serve as triggers to memory.
Professional Development13
What learners tell us 1
1
You will hear a talk by a man called Tom about the advantages and disadvantages of golf courses.
• Question 13:Tom suggests that golf courses could also be used as ………..
(Source Cambridge CAE: past test)•
Professional Development14
What learners tell us 1 Question: Tom suggests that golf courses could also be used as NATURE RESERVES’
• (30) S: number 13 is I’m not sure but um + he said ‘crack’• R: you heard the word ‘crack’?• S: crack …but I don’t know the meaning of ‘crack’• R: er you know it seemed to be an important word• S: yes I think so• R: ok + how did you spell ‘crack’ if if you don’t know the • S: c-r-a-c-k• R: right so you guessed the spelling did you?• S: I guess yes
• Most importantly, courses should be designed to attract rather than drive away wildlife.
•
Professional Development15
Conclusion: learner behaviour
• Learners sometimes simply listen out for prominent words – even if they do not understand them.
• This is partly a reflection of their level. At level B1 and below, listeners are very dependent upon picking up salient words rather than chunks or whole utterances.
• But this tendency is increased by the use of gap filling tasks, which focus attention at word level.
Professional Development 16
What learners tell us 2
• R: is there anything that you heard that helped you?• S: I have the problem about that because I am concentrate
on the two of the questions so …I didn’t realise• R: so• S: his his + he’s already go to the 9• R: right ok so you were still listening out for number 8• S: yeah and number 7• R: so that’s why you missed number 9• S: yes actually it’s a lot of problem about how to know where
he’s talking
Drawbacks
• It is common practice to pre-set questions so as to ensure focused listening. Most conventional methods anticipate in writing information that is going to occur in the recording.
• Learners form predictions which may shape their listening• It is a convention that questions follow the same order as
the recording• Learners often listen out for an answer to Q1; miss it and
as a result miss the answer to Q2 as well
18
What a conventional set of test items provides for free
The items in (e.g.) a gap-filling task potentially provide a candidate with:
• A summary of what the recording covers• A set of gapped sentences that follow the sequence of the
recording• Key words with which to locate information• Sequences which may echo the wording of the recording
or the order of words
What is listening?
Professional Development20
Expert listening (Field 2008)
Meaning
Speech signal
Words
20
Decoding
Word search
Parsing
Meaning construction
Discourse construction
Professional Development for IATEFL 201321
How close to real–life input?
and the recording
Decoding
Word search
Parsing
The recording (vs real-world input)
• Scripted (even semi-authentic) recordings bear little resemblance to natural everyday English
• Actors mark commas and full stops• There are no hesitations and false starts • Utterances are quite long and regular in rhythm• Voices do not overlap• Test setters sometimes put in distractors – making the
recording much more informationally dense than a natural piece of speech would be.
Extending listening practice
It is critical to maximise the exposure of learners to spoken L2 sources – ideally authentic ones
Thanks to the internet, there is now a wide range of potential L2 listening materials which can be downloaded and disseminated in MP3 form for:
• Classroom practice• Listening homework• Self-study at home• Study centres• Listening for pleasure
Professional Development24
How close to a real–life task?
and the task
Decoding
Word search
Parsing
Evaluating a task: multiple choice
You hear an explorer talking about a journey he’s making. How will he travel once he is across the river?
A. by motor vehicle B. on horseback C. on foot
(FCE Handbook, 2008: 60)
Recording 1 (FCE Sample Test 1:1)
• The engine’s full of water at the moment, it’s very doubtful if any of the trucks can get across the river in this weather. The alternative is to carry all the stuff across using the old footbridge, which is perfectly possible …and then use horses rather than trucks for the rest of the trip all the way instead of just the last 10 or 15 kilometres as was our original intention. We can always pick up the vehicles again on the way down…
Recording 1 (FCE Sample Test 1:1)
• The engine’s full of water at the moment, it’s very doubtful if any of the trucks can get across the river in this weather. The alternative is to carry all the stuff across using the old footbridge, which is perfectly possible …and then use horses rather than trucks for the rest of the trip all the way instead of just the last 10 or 15 kilometres as was our original intention. We can always pick up the vehicles again on the way down…
Conclusion
Conventional formats require the listener to:• Map from written information to spoken• Eliminate negative possibilities as well as identify positive
ones (esp with MCQ and T/F)• Read and write as well as listen (esp gap filling)• Engage in complex logistical tasks which take us well
beyond listening (esp. multiple matching)
Some solutions
• Provide items after a first playing of the recording and before a second. This ensures more natural listening, without preconceptions or advance information other than general context.
• Keep items short. Loading difficulty on to items (especially MCQ ones) just biases the test in favour of reading rather than listening.
• Items should not echo words in the recording• Some tasks (e.g. multiple matching) allow items to ignore
the order of the recording and to focus on global meaning rather than local detail.
The listening / reading conflict
• It is important to ensure that a task really practises listening and is not too dependent upon reading.
• Visual tasks avoid the problem – but are limited in the complexity of what they can show
• Multiple-choice questions make very heavy reading demands. The learner needs to discriminate finely between options and to hold the options n the mind.
• Gap filling makes multiple demands – reading, writing and listening.
• Solution: Keep questions short or rely on oral questions.
Reducing the reading load (Oxford Test of English B level)
Prices and time
0 Tickets cost A £6.50 B £8.50 C £10.50
1 The trip is on A Friday B Saturday C Sunday
2 The coach leaves school at A 8.30 B 9.30 C 11.30
At the film studio
3 We’ll see actors performing in A a comedy B a drama C a thriller
4 We’ll have the chance to
A use the cameras B talk to actors C try on clothes
5 Visitors can’t take into the studios
A bags B phones C sandwiches
Professional Development32
Does a given test represent all levels of listening?
Meaning
Speech signal
Words
32
Decoding
Word search
Parsing
Meaning construction
Discourse construction
Professional Development for IATEFL 201333
Higher levels of listening
• Meaning construction (= context).• Take a bare fact and relate it to what was said before• Interpret the attitude or intentions of the speaker• Bring in world knowledge or knowledge of the speaker
• Discourse construction• Link pieces of information together• Identify main and minor points• Report the line of argument or overall main point
TargetsAn item in a test can target any of these levels:• Decoding:
She caught the (a) 9.15 (b) 9.50 (c) 5.15 (d) 5.50 train.• Lexical search:
She went to London by …….• Factual information:
Where did she go and how?• Meaning construction:
Was she keen on going by train?• Discourse construction.
What was the main point made by the speaker?
Targeting levels of listening
• In theory, a good test should target all levels of listening in order to provide a complete picture of the test taker’s command of all the relevant processes.
• In practice, higher levels of listening may be too demanding in the early stages of L2 study. Novice listeners focus quite heavily on word-level decoding, which does not leave them enough spare attention to give to wider meaning. It is therefore appropriate to focus mainly on factual information.
Question focus
Test formats often focus narrowly on one level of listening.– Gap-filling items tend to focus on word recognition– Items in other task types tend to focus on facts.
• At higher levels of L2 knowledge, questions rarely tap into more complex levels of listening:– What are the speaker’s intentions or attitude?– What is the overall main point or points?– What is the line of argument that links the facts?– What can we infer that the speaker did not say?– What will the speaker go on to say?
Spread of targets
The inflexibility of high stakes tests
Large scale high-stakes tests have major constraints which prevent them from testing listening in a way that fully represents the skill.– Reliability and ease of marking– Highly controlled test methods, using traditional
formats that the candidate knows– Little attention possible to individual variation or
alternative answers
Solutions for local testers
Ask questions at discourse level:• What is the main point of the recording? / Give three main
points.• What is the connection between Point A and Point B?• Complete a skeleton summary of the text with main points
and sub-points
Ask learners to compare two recordings for similarities and differences
Ask learners to summarise a recording orally or in the form of notes (in L1 or L2)
Recent developments in the testing of L2 listening
Middle and low-stakes tests
• Test providers have recently begin to offer middle- and low-stakes tests. These tests can be much more flexible because they are not bound by some of the limitations of high-stakes tests.
• They can be used as– Entry tests– Progress tests– End of course tests– Professional selection tests– Diagnostic tests (useful in listening)
OTE Level B listening specifications
1 5 short monologues 5 MCQ visual Specific information
2 Longer monologue Note completion 5 MCQ items 3 option
Specificinformation
3 Longer dialogue(speaker opinion)
5 MM items 3 option
Stated opinionimplied meaning
4 5 short monologues / dialogues
5 MCQ items 3 option
attitude/feeling/opiniongistfunction/reason/purposespeaker relationshiptopictype/genre
Oxford Test of English - multiple matching: opinion
You will hear two students talking about a local history project they are doing at college. Match the opinion to the person who says it. Choose the correct answer (the woman, the man, or both) for each statement (1–5). There is one example (0).
• 0 It would be a good idea to check the research topics with the teacher.
• 1 The project will be improved by researching what local people did in the past.
• 2 Preparing questions for the interviews will be useful.• 3 It is advisable to interview some extra people.• 4 The best people to interview are family members.• 5 People who are interviewed should get a copy of the project.
Advantages of this kind of task
• Items do not need to follow the order of the recording• Items can test opinion – main point – fact – inference etc• Items minimise the reading load.
Meaning level questions (OTE B Level)
• 1. A man and a woman are talking about a concert they have just seen. What did the woman think of the concert?
• A The music was too loud. B The tickets were too expensive. C The stage was too far away.
• 2. A woman and her son are in a newsagent’s at the airport. Which magazine do they choose?
• A a football magazine B a computer magazine C a politics magazine• 3. Two friends are talking on the phone. What is the man’s job?• A a receptionist B a chef C a waiter• 4 A man is leaving a voicemail message for his friend. Why?• A to complain B to apologize C to suggest something • 5. Two friends are talking about a new computer game. What is the boy’s
opinion of the game?• A. Not good value. B Not challenging. Not original.
Computer delivery
Many test boards offer computer delivered versions of their tests. Unfortunately, they do not always take advantage of the new possibilities that computer delivery offers. In the case of listening, these include:
• Improved audio perception through individual headphones• The possibility of delivering the questions after the first
listening. This means that the first listening is much more natural and not unduly influenced by seeking answers.
• Thanks to an adaptive algorithm, the level of each question presented to the test taker can be based on their success in answering the previous question, so most questions are pitched at the approximate ability of the test taker.
References
• Field, J. (2008) Listening in the Language Classroom. Cambridge: CUP
• Field, J. (2009). The cognitive validity of the lecture listening section of the IELTS listening paper. IELTS Research Reports 9, Cambridge
• Field, J. (2013) Cognitive validity. In Geranpayeh, A. & Taylor, L. (eds.) Examining Listening. Cambridge: Cambridge University Press
• Oxford Test of English: https://elt.oup.com/feature/global/ote
Thanks for listening!