Workshop on the Table of Specifications

7/27/2019 Workshop on the Table of Specifications

1/46

Workshop on the Table of

Specifications

Presented by

Dr. Chan Chang Tik


2/46

Nov. 2006 Semester Break Training 2

Contents

The Table of Specifications and the

Bloom Taxonomy

Construction of Examination Questions

in line with the Learning Outcomes


3/46


Group Activity 1

Discussion

How do you know whether theexamination questions you set are

valid?


4/46


Valid?

Moderators comments:

I think is a significant problem. The

students cant answer the question

properly on the basis of the information

given . I consider the question is

either unfair or unanswerable.

Why it is here? I accept auditing is partof the learning outcomes but there

seems to be too much auditing.


5/46


Test Validity

To be valid a test must measure what it

is intended to measure.

Three points to consider in test validity:

Validity is specific to some particular use

Validity is a matter of degree

Validity pertains to the results of the test


6/46


Types of Validity

Content validity

Criterion-related validity (concurrent and

predictive)

Construct validity

Which one to choose depends on the

aims of your testing.


7/46


Content Validity

The test should cover a cross section ofthe course.

Emphasize important topics but give

less attention to less important topics.The relative importance of topics in a

test is directly proportional to the

instructional emphasis given to thetopics.


8/46


Table of Specifications

Students Response Q

Step 1

Number of lectures/chapters/units

Number of questions in the finalexam

One table of specifications for each

question

Why?


9/46


Step 1 Cont.

17 lectures in a course and final exam

consists of 5 questions

Q1 3 lectures Equally distributed?

Q2 4 lectures

Q3 3 lectures Grouping

Q4 4 lectures

Q5 3 lectures Discussion


10/46


Step 2

Learning objectives

Instruction

Assessment

Example (Mat250, Q1)

List down the number of lectures

List down the instructional objectives


11/46


Step 3

Blooms taxonomy of cognitive skills:

Knowledge remembering facts, terms,

definitions, and concepts. What?, list, name,define, describe

Comprehension understanding the

meaning of material. Explain, interpret,

summarize, give examples, predict, translate


12/46


Blooms taxonomy

Application selecting a concept or skill and

using it to solve a problem. Compute, solve,

apply, modify, construct

Analysis breaking material down into its

parts and explaining the hierarchical relations.

How does apply?, Why does work?,

How does relate to ?


13/46


Blooms taxonomy (2)

Synthesis producing something original

after having broken the material down into its

component parts.

How do the data support ?

How would you design an experiment that

investigates

What predictions can you make based uponthe data?


14/46

Nov. 2006Semester Break Training 14

Blooms taxonomy (3)

Evaluation making a judgment based upon

a pre-established set of criteria.

What judgments can you make about ?Compare and contrast criteria for ?

Refer to the table of specifications


15/46

Nov. 2006Semester Break Training 15

Table of Specifications Selected-

Response Q

Work as a team, forexample each lecturer is

assigned 5 lectures to come up with 50

multiple choice questions. Store any extra questions in the question

bank.


16/46


Why Table of Specifications

External Examiners Comments

Several topics covered in the lecture

programme are not assessed by this exam.

My analysis would indicate no assessment of:The Netherlands, France, Poland, Singapore,

South Korea, Hong Kong (10 of the 28 weeks

of teaching, 35.7%).

The module descriptor, the lecture programmeand the exam do not tie up very well.


17/46


Why Table of Specifications (2)

INTI-UC lecturers comment

I just want to say a million thanks for teaching us how to do the Table ofSpecifications in setting exam questions. It is only now that I am able toappreciate how useful that table is, now that I have managed to create myfirst set of final exam questions (deadline today lah!). I use the table andI am happy with the questions I set because I could justify them if I wereasked.I think this part of the skill training (the Table of Specs part ) shouldbe made compulsory for all new lecturers.


18/46


Table of Specifications Students Response Questions

Consider Question 1 of MAT250. There are 4 parts in this question

taken from the following lectures:

Learning OutcomesLecture 1 To define real numbers.

To describe exponents and to produce the laws of exponents.To explain and simplify radicals.

Lecture 2 To define the various types of equations.

To relate linear equation to its applications.

Lecture 3 To solve quadratic and radical equations.

To discover some practical problems of equations.

To explain some coordinates geometry topics.

Lecture 4 To define inequalities.

To solve linear, quadratic and rational inequalities.To solve equations and inequalities involving absolute values.

Lecture 9 To produce partial fractions.


19/46


Table of Specifications (2)

Knowledge Comprehension Application Mark

(a) To define and

simplify exponents

and radical

expressions.1 2 3

(b) To solve radical

equations

5 5

(c) To solve rational

and quadratic

inequalities 7 7

(d) To produce

partial fractions5 5

Total

1 2 17 20

Bloom 3


20/46


Table of Specifications Selected Response Questions

Knowledge Comprehension Application No. of

Questions

1 5 (50%) 2 (20%) 3 (30%) 10 (20%)

2 2 (10%) 6 (40%) 7 (50%) 15 (30%)

3 7 (50%) 8 (50%) 15 (30%)

4 1 (10%) 1 (20%) 3 (70%) 5 (10%)

5 1 (20%) 1 (20%) 3 (60%) 5 (10%)

9 (18%) 17 (34%) 24 (48%) 50 ( 100%)


21/46


Group Activity 2

Set three questions each at different level of

the Blooms Taxonomy using a table of

specifications. Indicate clearly your learningoutcomes.

Present your questions to the participants.

Time : 45 minutes


22/46


Types of Assessment

Essay

Mathematical and

technical problem

items


23/46


Essay

Use clear and concise questions

Measure effectively complex learning outcomes

Establish scope of expected response

Handling of technical terms

Content versus mechanics of expression

Use descriptive words of precise meaning as

directions

No optional items


24/46


Clear and Concise Questions

Language usage and word choice are

particular important during question

construction.

The language dimension is critical because itcontrols the comprehension level of the item

for the students and also it specifies the

question parameters.

Example


25/46


Clear and Concise Questions (2)

For example, Describe the origins of World War I

A better question is What were the principal

diplomatic events in Europe between 1890 and1913 that contributed directly to the outbreak of

World War I

You have to narrowly specify, explicate, define, or

otherwise clarify what it is that you want from thestudents.


26/46


Clear and Concise Questions (3)

The question set must have an answer that

expert could agree upon, thereby

rendering it object ive. For example,Describe the world and give two examples.

Example


27/46


Complex Learning Outcomes

Complex learning outcomes can bemeasured effectively with essayquestions. These are the abilities to:

Explain cause-effect relationshipsDescribe applications of principles

Present relevant arguments

Formulate tenable hypotheses Formulate valid conclusions


28/46


Complex Learning Outcomes (2)

These are abilities to:

State necessary assumptions

Explain methods and procedures

Produce, organise, and express ideas

Integrate learning in different areas

Evaluate the worth of ideas


29/46


Extended versus Restricted

Response

From the instructors standpoint, an extensiveresponseto a few broadly based questions allowsan in-depth sampling of a students knowledge,thinking processes, and problem-solvingbehaviour relative to a particular topic.

The open-ended nature of the question posed by

the instructor such as Discuss the bond andshare markets in the Malaysian context ischallenging to a student.


30/46


Extended versus Restricted Response (2)

Extensive response question is poorly

structured and its free-response essay item

tends to yield a variety of responses fromthe students, in both content and

organisation, and thus inhibit reliable

grading.

Example


31/46


Extended versus Restricted Response (3)

A restricted-response question is preferable.For example, Discuss the advantages anddisadvantages of essay and short-answer

question with respect to (a) reliability, (b)objectivity, (c) content validity, and (d)usability.

This question presents a better defined task

which lends itself to reliable scoring, yetallows students sufficient latitude to organiseand express their thoughts creatively.


32/46


Content versus Mechanics of

Expression

Factors such as expression, grammar, spelling,

and the like can be evaluated in conjunction with

contents of the essay. Evaluation of such skills is legitimate provided they

are included in the learning outcomes.

You may point out such mistakes in the students

coursework.


33/46


Directive Words

Compare

Defend

Trace

Contrast

Explain

Differences


34/46


Non-directive Words

Review

Tell all that you know

Report your knowledge


35/46


Classic Case

Question (10 marks)

Tell me all you know

about Malaysia.

Answer

It is a beautiful country.


36/46


Example for Objectivity of Question

External Examiners comment:

The students cant answer the question

properly on the basis of the information given

as they dont know how to spread the financecharges and looking at the three main

alternative methods is not part of the question.

I consider the question is either unfair or

unanswerable.


37/46


Example of Extensive Response

The relevance and the effects of accounting

practices, have been the subject of much debate.

Discuss the impact on the accounting practices.

External Examiners comments:

Give source.

I dont think this is a clear question and I would be

amazed if any student produced a response anything

like the outline answer.


38/46


Example of Language Usage

A business or organisation can be positioned

as effective as products and services.

Internal examiners comment:

The sentence does not make sense. I suggestSuccessful marketing of products and

services often depends on effective

positioning in the market place.


39/46


Group Activity 3

Look back at the essay or short-answerquestions you set in Group Activity 2. Do youthink you need to make any adjustments?


Time : 20 minutes.


40/46


Mathematical and Technical Problem

Items

Provide enough information and directions sothat students clearly understand the problem.

Indicate the degree of precision expected in

the response. Clearly specify the units for reporting the

response.

When using regulareveryday words topresent a problem, make sure the connectionbetween the structured situation and the realworld is genuine.


41/46


Enough Information

During the vacation Sue and Tai agreed topaint their house. One side of the house iseight-meters long. They mark a point on theside that is four meters from each other.

Each one starts at one end and paintstoward the center mark, four meters fromeach other. Can we be sure that eachpainted the same area of that side of the

house if each paints to the center mark?


42/46


Degree of Precision

In the following set of exercises for calculating

z-scores from the raw scores, use the mean

and standard deviation values to the nearest

hundredth of a raw score unit. The proportions that you report for the ratios

should be given in decimal fractions to the

nearest hundredth.


43/46


Units

Total elapsed time should be reported in

minutes and seconds.

When reporting the perimeters, give your

responses in meters.


44/46


Everyday Words

What words can you make from the letters in

TEA?

TEA, ATE, EAT, ETA, AET, TAE

Create a permutation for the three letters in

TEA.


45/46


Group Activity 4

Look at the mathematical and technical

questions set in Group Activity 2. Are the

questions ambiguous? Do you want to re-set

them?


Time: 20 minutes


46/46


Thank you

Workshop on the Table of Specifications

Documents

Transcript of Workshop on the Table of Specifications