Nükte DurhanMETU, Northern Cyprus Campus, School of Foreign Languages(Ankara, 30 May 2012)

General look at socio-cognitive framework Cognitive validity (theory-based) and

routine skills Needs analysis results of the Academic

Speaking Skills (211) Course Description of the speaking tasks in the

course and evaluation from cognitive validity aspects

Context validity: considering its relevant aspects and the speaking tasks

Areas that need work: scoring validity

Performance tests are characterised by the relationship of the test task to the world beyond the test. Ideally tasks are…

direct and authentic

- situationally authentic

- interactionally authentic

But a test can never fully represent reality.

We want candidates to… perform relevant language tasks adapt their speech to circumstances make decisions under time pressure implement them fluently make any necessary adjustments as

unexpected problems arise

Test Taker

Context Cognitive

Response

Scoring Validity

Score/Grade

ConsequentialValidity

Criterion-Related Validity

Test-taker characteristics Context focused Cognitive focused (theory-based) Scoring focused Criterion focused (external) Consequentially focused

The abilities to be tested are demonstrated by the mental processing of the candidate (the cognitive dimension)

The use of language in performing tasks is viewed as a social rather than a purely linguistic phenomenon (the socio- dimension)

Test construct resides in interactions of underlying cognitive ability and context of use, i.e. socio-cognitive model

Executive processes◦ Conceptualizer◦ Pre verbal message◦ Linguistic formulator◦ Phonetic plan◦ Articulator◦ Overt speech◦ Audition◦ Speech comprehension

(Weir,2005)

Executive resources◦ Language knowledge

Grammatical Textual Functional Social

◦ Content knowledge Internal External

Routine Skills (Bygate, 1987)

Informational Routines expository routines:

involve factual information evaluative routines:

involve drawing of conclusions, expression of reasoning; explanation, predictions, justifications, preferences and decisions

Routine Skills (Bygate, 1987)

Interactional Routines : telephone conversations interviews meetings discussions decision making tasks

Routine Skills (Bygate, 1987)

Interactional Routines : They are assisted by… interaction management skills involving

content-focused agenda management and interaction-focused turn-taking

Improvisational Skills brought into play when an interaction

falters

Informative Speech (individual) (10%) Persuasive Speech (group) (20%) Final Presentation (individual) (30 %)

No assessment of discourse and strategic competences

Participants want more of the following: More focus on spontaneous speech A variety of speaking activities More presentations with smaller

percentages Different types of presentations Focus on fluency Interactive activities Creativity and individuality Speech critiques

Debate Role plays Discussions Impromptu speech Pronunciation activities Critical thinking activities Asking & answering questions

Reporting the team’s observation of a store’s merchandising strategies and store design

Organizing and sharing points in the group Presenting findings Giving evidence from the observations Evaluating the effectiveness of the

strategies

To inform and enlighten the audience about a given topic on the general theme of technology

Choosing a news item related to art, analyzing it and presenting the analysis to class

Includes: justification of the selection of the item reaction to points and style evaluation of the ideas

Giving a balanced view of the selected topic (related to the five themes in the book)

Includes: extensive literature research reaction to the presented

material/discussion or evaluation of the topic

Debates Discussions Panel discussion Role plays Conflict resolution tasks

Three different activities should be carried out

in class throughout the semester.

Setting: task◦ Purpose◦ Response format◦ Known criteria◦ Weighting◦ Order of items◦ Time constraints

Setting: administration◦ Physical conditions◦ Uniformity of administration◦ Security

(Weir,2005)

Demands: task◦ Linguistic

Channel Discourse mode Text length Nature of information Topic familiarity Lexical range Structural range Functional range

1. Rubric

Is the rubric accurate and accessible?

2. Purpose

Is the purpose of the test made unequivocally clear for the candidate?

3. Weighting

Are any weightings for different test components adequately justified?

4. Known criteria

Are the criteria to be used in the marking of the test explicit for the candidates and the markers?

5. Time constraints: particularly important for speech as normal speech takes place under time pressure

Is the timing for each task (e.g. preparation and completion) appropriate?

6. Discourse mode:Reciprocity conditions are concerned with the

dimension of interpersonal interaction, the relation between speaker and listener.

Are there reciprocal exchanges where speakers have to adjust vocabulary and message and take each other’s contributions into account?

7. Type of information Is the type of information appropriate for

the target situation requirements of the students being tested?

8. Content knowledge Is the content sufficiently familiar to

candidates so that they have sufficient existing schemata to enable them to deploy appropriate skills and strategies? (Urquhart & Weir, 1998)

8. Content knowledge“every attempt should be made to allow

background knowledge to facilitate performance rather than allowing its absence to inhibit performance” (Alderson, 2000)

30

9. Linguistic knowledge Lexical range

Are the lexical items in the test both in input text and required as output appropriate for the level of the candidates?

Syntax Are the grammatical items in the test

both in input text and required as output appropriate for the level of the candidates?

9. Linguistic knowledge Function: is a term used to describe the

illocutionary force of what is said. Are the functions in the test both in

input text and required as output appropriate for the level of the candidates?

The variety of tasks with different functional purposes and increasing number of tasks improve the validity.

Rating◦Criteria / rating scale

◦Rating procedures Rater selection Rater training Standardisation Moderation Rating conditions Statistical analysis

dnukte@metu.edu.tr