Evaluating the effectiveness of a task-based syllabus for teaching

speaking to advanced learners of English

Melanie EllisForeign Language Teacher Training

College, Zabrze, Polandmelanie@ellis.pol.pl

Defining terms

Evaluation: the process of determining how a program worked in practice, by collecting varied informative data from a variety of sources

Why? When? Whom? What? How? For whom?(Alderson 1986)

Task-based syllabus

• Process merging ‘what’ is to be learnt and ‘how’ this is to take place (Nunan 1989)

Specification of tasks: type, topicSpecification of performance conditions: support,

planning time, rehearsal, interactionSequencing criteria: progession from easy to

more difficult (after Ellis, R. 2003:206)

Plus syllabus design principles based on Stenhouse (1975)

Stenhouse (1975:4)

An attempt to communicate the essential principles and features of an educational proposal in such a form that it is open to scutiny and capable of effective translation into practice

*Seen as key, as little research available on TB syllabuses in practice

Definition of task

• Activity where communication of meaningis of first importance

• Successful completion of the task is one of the main aims

• Usually involves some communicative challenge to solve

• Is open in terms of language (‘unfocused’)(based on Skehan 1998:95)

Why evaluate?

• Accountability: Is the program effective? Does the syllabus meet its objectives? Is it valid?

• Curriculum development and betterment: Is a task-based syllabus feasible? Does it give learners a sense of progress?

• For purposes of teacher self-developmentDoctoral research conducted by the teacher as researcher(Rea-Dickens & Germaine 1992:26)

When? Whom? What?

• 3 year foreign language teacher training college in Zabrze, Poland

• First year undergraduates, school leavers and mature students. 3 class groups, n=39

• Academic year (2002-3) one 2 hour class a week, over 30 weeks

• Learners at B2-C1 (CEF) Target on graduation from college is C2

• ‘Conversation’ course

Syllabus design principles:sequencing

1. Task category Description>description/instruction>narration>opinion-giving/analysis/synthesis (Brown & Yule 1983, Bloom 1956)

2. Performance conditions-organization & support, feedbackClosed pair………………………………………open forumPrompts + planning + rehearsal…………………………..spontaneous

speech (Bygate 2001, Nunan 1989)

3. Task difficulty code & cognitive complexityFamiliarity. No. of elements, Structure etc.(Skehan 1998, Robinson 2001)

Syllabus objectives

The course aims to help learners:become more confident in speaking Englishdevelop fluencyincrease vocabularyincrease awareness of their strengths and weaknesses in speaking and how to improve thesedevelop compensation strategies

How was the evaluation done ?

• Questionnaires after first and second semesters, closed and open questions

• 9 Case studies, 8 guided interviews conducted• Teacher notes• Quantitative evaluation of speech samples for

evidence of development of fluency (MLU, words /min., % pause ) and increase in lexical complexity (type-token ratio, no. of clauses)

• Intact class group results compared with sub grps divided according to ability

Expected outcomes

Learners • become more confident in speaking• develop greater fluency• produce more complex speech, particularly

richer in lexis• feel that there is a progression from easier to

more difficult in terms of task sequencing

More confident?

Questionnaire 1In what ways has your speaking improved in

this class in this semester?40% ‘more confident’, ‘more self-assured’ ‘more

brave’ in speaking35% more fluent55% vocabulary has increased20% better pronunciation

More confident?

Questionnaire 226% more confident, particularly at speaking in front of the

groupInterview data: In what way has your speaking improved?

6/8 mention confidenceI gained lots of confidenceNow I’m more confident in my use of the language. I’m less

stressedI’m not afraid now to talk in EnglishI break the barriers. The barriers were my shyness

Was there a gain in fluency?

CaveatsLearners are in an English-medium program 24

hours a weekComparability of first and final tasks- problematicPicture description is not representatative, ‘task’ ?

Sample picture from final assessment

First assessment: October

• Picture description: repeated task• Describe picture for partner to identify• Describe from a prompt: new task, no

preparation eg. Describe sb you know well; Describe a member of your family; Describe one of your relatives

October-June: Fluency

Group MLU MLU % pauses % pauses WPM WPM

A 11.9 20.96 19.08 11.83 70.07 91.6

B 8.01 20.34 20.69 12.87 63.27 86.92

C 10.64 15.48 18.17 14.2 65.12 66.36

October to June

/Well the lady is is about I think she’s in her fifties/she’s a bit plump/she’s wearing a black pleated skirt/under a um striped coat/um:um she has:blond hair tied er in a plait/

/OK/: these are two mothers with their children/one is with a boy with a boy and the other one is with a little girl/: um the woman the mother on the left is wearing a white sweater/she’s also a necklace/: and her hair is brown/

Sub-groups: October to June

• Significant increases in fluency in all measures in Strong group and Below Average group

• Average group: significant gains in MLU, significant reduction in amount of pause, no gain observed in words per minute (SD = 17.17 October; 21.97 June)

• WPM as useful measure of fluency across a group? Very high SDs in all cases

Is there a sense of progression? Sample task sequence

• Known tasks (matura); rehearsed: describe picture of house

• Parallel: you visited this place last weekend, tell a friend at work about your impressions

• Personalise: place you often visited as a childTake your partner on a guided tour. They will

later have to explain to someone else where they went and what they experienced.

Is there a sense of progression?

Data from 2 questionnairesLearners rated tasks 1 (very difficult for you) – 5

(very easy for you)Open questions: Explain which tasks were most difficult for you

and whyExplain which tasks were easiest for you and

why

Most difficult

Picture story (new task, no preparation) : 52%People and places tasks with no support: 10%People task no support: 10%All unfamiliar tasks: 10%

Why?Lack of vocabulary 40%

Not enough preparation time 22%Didn’t know what to say 17%

Easiest

All pictures seen before 35%Re-telling a story 35%Describing people 25%Places: pair task (describe and identify) 25%

Why?I had practised and learnt vocabulary 37%

Task was familiar 12%Task was interesting 10%

Sequencing criteria confirmed

• Familiarity • Repeated, rehearsed tasks • Describing people • Minus support • Minus familiarity • Minus preparation • Minus structure

Increase in vocabulary?

Contradictory evidence Questionnaire 1: In what ways has your speaking

improved in this class in this semester?55% report increase in vocabularyQuestionnaire 2: In what ways has your speaking improved

and how much (1-5 where 5 = a lot)Vocabulary : mean value 4.67But: No gains in Type Token Ratio although increase in

number of clauses > > > More research needed

Complexity: October- June

Group Clauses Clauses TTR TTR

A 1.05 2.09 0.31 0.24

B 0.77 1.89 0.28 0.22

C 0.81 1.63 0.33 0.28

Conclusions

• Syllabus achieves objectives in the eyes of the learners

• Fluency increases• Sense of progression perceived• Design principles confirmed• Vocabulary development? Other measures ?

Other tests? More research needed

Thank you for your attention

melanie@ellis.pol.pl

References

• Alderson, J.C. 1986. The nature of the beast. Trends in Language Programme evaluation. Bangkok: Chulalongkorn University Language Institute

• Bloom, B.S. (ed.) 1956 Taxonomy of Educational Objectives. New York: David McKay Company Inc.• Brown, G & Yule, G. 1984. Teaching the Spoken Language. Cambridge: Cambridge University Press• Ellis, M. 2004. Developing speaking skills in the teaching of English at the advanced level. Design,

implementation and evaluation of a task-based syllabus for trainees in pre-service education. Unpublished PhD thesis . University of Warsaw, Poland

• Ellis, M. 2008. Design & evaluation of a task-based syllabus for developing speaking skills. In Pawlak, M. (ed.) Investigating English Language Learning & Teaching. Poznan-Kalisz: Faculty of Pedagogy & Fine Arts in Kalisz, Adam Mickiewicz University , Poznan

• Ellis, R. 2003. Task-based Language Learning and Teaching. Oxford: Oxford University Press• Nunan, D. 1989. Designing Tasks for the Communicative Classroom. Cambridge: Cambridge University

Press• Rea-Dickens, P & Germaine, K. 1992. Evaluation. Oxford: Oxford University Press• Robinson, P. 2001. Task complexity, task difficulty and task production: exploring interactions in a

componential framework. Applied Linguistics 22/1: 27-57• Skehan, P. 1998. A Cognitive Approach to Language Learning. Oxford: Oxford University Press• Stenhouse, L. 1975. An Introduction to Curriculum Research and Development. London: Heinemann