The Place of Direct and Indirect Tests in the

Assessment of Speaking Skills:

A Case Study

Dr. Hale KızılcıkMay 30, 2012

OUTLINE•E

NG 211 Academic Speaking Skills •T

he development of the midterm exam•A

closer look at the exam: «Discussion skills»•E

xam results•I

nterpreting the exam results

ENG 211 Academic Speaking Skills•R

evised in 2009-2010 •N

ot only to focus on «speaking as performance» (presentation)

•But also to focus on «speaking as transaction»

(Richards, 2007)

AssessmentPRESENTATIONS 50%

Team Presentation Informative Presentation Final Presentation

10%10%30%

SPEAKING ACTIVITIES 20%

Current Events News StoryGoing One Step Further Tasks

5%15%

MIDTERM EXAM 20%WRITING ASSIGNMENTS 10%

Speech Analysis ReportSelf-evaluation Report 

5% 5%

Preparing a written test for a speaking course

•Valid

•Reliable

•Authentic

•Practical

•Backwash

(Hughes, 1998)

•Course objectives

•Cooperation with the course coordinators

•Face validity

Parts of the Midterm Exam•L

istening for specific information and main ideas (podcast) 6 pts.

•Synthesizing & expressing opinions

(6 pts.)

*Listening for note-taking•O

rganizing a speech (4 pts.)•D

iscussion skills (4 pts.)

Discussing testing «discussion skills»

«They already do it in the class (a lot of times!)» «Is it a good idea to test spoken language in written discourse?»«It may have benefical backwash effect. It can encourage students to pay attention to signposts» «They already do it... But let’s see if they can integrate sources and use appropriate strategies» «In this way, we can also test their critical thinking skills»

Part IV: Discussion Skills (4 pts.)

The following excerpts are taken from discussions. Participate in each discussion as instructed. Express your views in 3-4 sentences. (2 pts. each)

A: Read the transcript of a discussion on graffiti. Interrupt the last speaker appropriately, agree with Victoria, and support your stance by giving your own reasons.

Victoria: I think one should consider graffiti from a broader angle. Graffiti is not limited to painted large letters on walls. Think of a graffiti artist like Banksy. I would love to have one of his paintings on my wall.

Sadi: I do not think graffiti can be justified on any grounds. It is a form of vandalism and personally I would sue anybody, Banksy or Picasso, who would paint graffiti on the walls of my house.

You: ______________________________________________________________

Sample answer: I am sorry but I’d like to make a point here. I think Victoria is right and we have to distinguish between different forms of graffiti. Besides, I’d also really appreciate a Banksy on my wall. First, his graffiti is full of important political and social messages which are worth spreading. Second, they cost a fortune. One of his graffiti works can make me a rich man.

Criteria for marking:Signpost and stance: 0.5 pt.Support: 1.5 pts.

Testing Discussion Skills Directly

Sample speaking task&

Debate & discussion rubric

Criteria•Q

uality of arguments (5 pts.)•S

tyle (5 pts.)•D

elivery (5 pts)•L

anguage (3 pts.)•T

eam work (2 pts.)

Interpreting the Midterm Results

Success Rate in Different Sections of the ExamPart I (listening): 61%Part II-A (synthesizing): 72%Part II-B: (expressing opinions): 77%Part III: (organizing a speech): 78%Part IV-A: (discussion skills): 49.5%Part IV-B: (discussion skills): 49.5%

CHART DISPLAYING ST. GRADES IN PART IV AND IN-CLASS DISCUSSIONS

So? What does this mean?Is there a problem with the reliability of in-class speaking tests?Is there grade inflation on the part of the teacher?Is there a problem with the discussion grading rubric?Is there a problem with the validity of the written test?Are the two tests testing different skills?

A closer look at the extreme cases•G

rammar becomes more important in the indirect test.•R

epetitions & digressions may be overlooked in the direct test.

•Quality of ideas are scrutinized more in the indirect test.

•The criteria for the direct test is broader in scope.

•There are personality issues.

Conclusions•O

n testing speaking indirectly•O

n testing speaking as performance•O

n structures speaking tasks & automatic scoring of tests (Luoma, 2004)

•On teaching discussion skills

Thank you

REFERENCES

Hughes, A. (1989). Testing for language teachers.

Cambridge: Cambridge University Press.

Luoma, S. (2004). Asessing speaking. Cambridge: Cambridge

University Press.

Richards, J., C. (2007). Developing Classroom Speaking Activities: From Theory to Practice: Summary article. From http://www.professorjackrichards.com/pdfs/developing-

classroom-speaking-activities.pdf