Assessment – Preparing, Delivering, Assessing Damian Gordon.

Assessment – Preparing, Delivering,

Assessing

Damian Gordon

Last Week: Instructional Design

The Joy of Teaching

Then Jesus took his disciples up to the mountain and gathered them around him, he taught them saying, "Blessed are the poor in spirit, for theirs is the kingdom of heaven. Blessed are the meek. Blessed are they that mourn. Blessed are you when you suffer. Be glad and rejoice, for your reward is great in heaven."

Then Simon Peter said, "Are we supposed to know this?" And Andrew said, "Will we have a test on this?" And Philip said, "I don't have any paper." And Bartholomew said, "Do we have to turn this in?" And John said, "The other disciples didn't have to learn this." And Matthew said, "May I go to the bathroom?" Then one of the Pharisees who was present asked to see Jesus' lesson

plan and inquired of Jesus, "Where are your objectives and your required competencies in the cognitive domain?"

And Jesus wept.

Overview Part 1 (Why?)

Purpose of assessment Formative -v- summative assessment

Part 2 (What?) Learning outcomes Constructive alignment

Part 3 (How?) Types of assessment Grading of assessment

Part 4 (Who?) Self / peer assessment

Assessment

How do you get a new belt in Karate, how accurate does your throws, blocks and punches have to be ? 40% 50% 75% 90% 100%

Assessment

How do you get a new belt in Karate, how accurate does your throws, blocks and punches have to be ? 40% 50% 75% 90% 100%

>>>>>>> How should it be assessed? <<<<<<<<

Assessment

To qualify as a doctor, what average result do you need to get to pass ? 40% 50% 75% 90% 100%

Assessment

To qualify as a doctor, what average result do you need to get to pass ? 40% 50% 75% 90% 100%

>>>>>>> How should it be assessed? <<<<<<<<

What is the purpose of assessment ?

John Biggs

Born in Tasmania (5th Generation)

an educational psychologist and novelist

developed the SOLO Taxonomy and the model of constructive alignment

Discusses “Backwash”

Purpose of assessment I

Reasons for assessment Select students Control / motivate students Direct teaching / learning Identify strengths / weaknesses of teaching /

learning

Purpose of assessment II

Backwash Students learn what they think they will be

assessed on Assessment determines what and how students

learn more than the curriculum does This must be capitalised on and employed

Michael Scriven

Born 1928 in Britain Degree in mathematics

and a doctorate in philosophy

made significant contributions in the fields of philosophy and evaluation

Coined the terms “formative evaluation” and “summative evaluation” in 1967

Purpose of assessment III

Types of assessment Formative assessment

Used for feedback throughout the course Should not (all) contribute to marking Useful for monitoring learning progress

Summative assessment End of course Used to grade students

Peter Knight

Born in Britain Died in 2007 Educational research

and assessment expert Author of “Children’s

Understanding of People in the Past”

Author of “Being A Teacher In Higher Education”

Purpose of assessment IV

Reason for assessment Authorities agree that assessment has two main

roles. It warrants or certifies achievement…….It also influences learning, partly by signaling what is really valued…

Peter Knight, SEFI Annual Conference, (2001)

…and one other expert(s) we

should listen to

Purpose of assessment V

Student’s views

1. “Helps me find out what I know and don’t know” (87.3%)2. “Guides my learning” (75.5%)3. “Helps me link theory to practice” (84.3%)4. “Helps lecturers evaluate student knowledge” (81.4%)5. “Enables lecturers to evaluate their teaching” (75.5%)6. “Used by lecturers to rank students” (44.1%)

Formative -v- Summative Assessment

“When the chef tastes the sauce it is formative assessment, when the customer tastes it, it is summative” Robert Stake (possibly)

Formative Assessment Students’ Top 5 reasons for assessment all

formative Formative assessment cannot be separated

from teaching The more effective the teaching method, the

better its ability to provide formative feedback

1. “Helps me find out what I know and don’t know” (87.3%)

2. “Guides my learning” (75.5%)3. “Helps me link theory to practice”

(84.3%)4. “Helps lecturers evaluate student

knowledge” (81.4%)5. “Enables lecturers to evaluate

their teaching” (75.5%)6. “Used by lecturers to rank

students” (44.1%)

Admitting / recognising error is crucial for learning Error is used to instruct, not punish Student should be able to take the formative role on

himself / herself in the future

Summative assessment

To report a grade!

One line answer

Why assess? We assess to help students learn (formative)

and to judge how much they learned (summative)

Learning Outcomes

Without a clear understanding of the programme goals students are likely to become disorientated and waste time trying to discover what it is we want them to learn

Learning outcomes consist of three parts: the student action the content the standard required to meet the objective

Industry-Oriented Learning Outcomes

Graduates to be equipped with core competencies Who will be able to become productive in a short

time with training Graduates to be equipped with the necessary

skills to adapt to changing needs as appropriate and acquire new skills quickly

Layers of Learning Outcomes

SessionModule

Programme

Constructive Alignment

LEARNING OUTCOMES

TEACHING& LEARNING

ASSESSMENTMETHODS

Constructive Alignment I

Learning outcomes, teaching methods and assessment are all inter-dependant

Only by truly integrating these components together, do we get efficient student learning

Meaning is not imposed or transmitted by direct instruction - it is created by the student's own learning activities

Constructive Alignment II

A. Knowledge and understanding

1. Software Engineering;

2. Problem Solving and Critical Analysis;

3. A range of programming languages and environments;

4. Basic principles of computer hardware, data transmission and local area networking;

5. Operating systems,

6. Business context;

7. Engineering practice;

Teaching/learning methods The knowledge required for the basic topics is

obtained via lectures, exercises, practicals, assignments and project work. Appropriate IT and other software packages are taught.

Practical demonstrators and project supervisors advise students, and feedback is provided on all continually assessed work.

As the course progresses students are expected to show greater initiative.

Assessment Most knowledge is tested through a combination of

practicals, assignments and formal examinations. Students write reports on many assignments, and

also make oral presentations of their work.

Constructive Alignment III

B. Intellectual Skills

1. Select and apply appropriate computer based methods;

2. Analyse and solve problems;

3. Transfer appropriate knowledge and methods from one topic to another;

4. Plan, conduct and write a report on a project or assessment;

5. Prepare an oral presentation.

Teaching/learning methods Appropriate software, mathematical, scientific and IT

skills and tools are taught in lectures, and problems to be solved are given as projects or assignments.

Project planning is part of the Part 4 project, and written and oral presentations are required for various assignments and projects.

Assessment Skills 1-3 are assessed partly by examination, though

sometimes also by project or assignment work. Skills 4 and 5 are assessed as part of project work.

Constructive Alignment IV

C. Practical Skills

1. To design, implement and test well structured and documented computer programs using a wide variety of programming languages, with a focus on object-oriented development,

2. To develop applications using the main web programming technologies;

3. Database manipulation and administration;

4. Research into a computer science problem;

5. Present work in both written and oral form.

Teaching/learning methods Software tools are introduced in lectures

and their use is assessed by examinations and assignments.

Programming assignments are set, and students may write programs to solve other projects.

Practicals and projects are used to teach about skill 3, and projects are used for skills 4, 5, 6 and 7.

Assessment Skills 1 to 4 are tested by assignments and

projects, is addition to coursework. Skills 1, 2 and 3 are assessed in practicals.

Skills 4 and 5 are assessed through project work.

Constructive Alignment V

D. Transferrable Skills

1. use software tools.

2. acquire, manipulate and process data.

3. use creativity and innovation.

4. solve problems.

5. communicate scientific ideas.

6. give oral presentations.

7. work as part of a team.

8. use information resources.

9. manage time.

Teaching/learning methods and strategies Software tools are taught partly in lectures, mainly

through practical sessions and assignments. Data skills are acquired in laboratory and projects. Creativity and innovation and problems solving are

experienced through projects, as are team working, time management and presentations.

Use of information resources, such as the library and IT methods is experienced through projects and assignments.

Assessment Some skills, like the use of software tools and ability

to communicate orally and in written form are directly assessed, in assignments or projects

3D Alignment Horizontal Vertical Constructive

O'Leary, C., Lawless, D., Gordon, D., Carroll, D., Mtenzi, F., and Collins, M., “3D Alignment in the Adaptive Software Engineering Curriculum” 36th Annual Frontiers in Education Conference, San Diego, USA, October 28th - 30th 2006

we recall Bloom’s Taxonomy

Blooms Taxonomy I

6 Levels (Cognitive Domains) Knowledge Comprehension Application Analysis Synthesis Evaluation

Blooms Taxonomy II

Knowledge list name identify show define recognize recall state

Year 1

Blooms Taxonomy III Comprehension

summarize explain put into your own words interpret describe compare paraphrase differentiate demonstrate visualize find more information about restate

Years 1, 2

Blooms Taxonomy IV Application

solve illustrate calculate use interpret relate manipulate apply classify modify put into practice

Years 2, 3

Blooms Taxonomy V

Analysis analyze organize deduce choose contrast compare distinguish

Year 3

Blooms Taxonomy VI Synthesis

design hypothesize support schematize write report discuss plan devise compare create construct

Years 3, 4

Blooms Taxonomy VII

Evaluation Evaluate Choose Estimate Judge Defend Criticize Justify

Year 4

Types of Knowledge

Types of knowledge I

Declarative

Procedural

Conditional

Functioning

Types of knowledge II

Declarative Knowing something Facts – recorded in books / delivered in lectures Accrues from research, not from personal

experience Verifiable, usable

Types of knowledge III

Procedural Skill based knowledge Sequences of actions Having the right competencies

Types of knowledge IV

Conditional Subsumes declarative and procedural Knowing when to do something, what to do, why

to do it… Turns procedural knowledge into functioning

knowledge

Types of knowledge V

Functional Performance of understanding Knowledge within the experience of the learner

who can put declarative knowledge to work by solving problems

Types of knowledge VI

Curricula at third level generally focus on declarative, something reflected in forms of assessment

What is required for professional competence is functioning knowledge

Assessment should reflect this.

SOLO Taxonomy

Levels of understanding I

SOLO Taxonomy Structure of Observable

Learning Outcome

PrestructuralUnistructuralMultistructural Relational

Extended

abstract

Quantitative Qualitative

Levels of understanding II

Prestructural understanding Learner misses the point No observable understanding

Levels of understanding III

Unistructural understanding Simple naming Terminology Focus on one conceptual issue in a complex task Low level understanding

Levels of understanding IV

Multistructural understanding Disorganised collection of items Knowledge telling

Levels of understanding V

Relational understanding Using a concept that integrates a collection of

data Understanding how to apply the concept to a

familiar data set or to a problem

Levels of understanding VI

Extended abstract Relating to existing principle so that unseen

problems can be handled Questioning and going beyond existing principles

Assessment Criteria

Assessment criteria I

State the Type of knowledge Level of understanding

required to meet a particular learning outcome

Assessment criteria II

“the standard required to meet the outcome” Criteria are developed by analyzing the learning

outcomes and identifying the specific characteristics that contribute to the overall assessment

How many criteria are needed that contain neither irrelevant or miss important areas?

Assessment criteria III

Criteria serve two functions For formative feedback, learners can

See how they performed on the various components of the assessment

Identify strengths and weaknesses For summative feedback,

A contract is in place between teacher and learner

Assessment criteria IV

Questions to answer when setting an assessment What learning outcomes do you intend to assess? How does your approach to assessment align with these

outcomes? What are the assessment criteria? What other skills are assumed e.g. skills from previous

years? How do you ensure that your standards are consistent?

Formative Feedback I Feedback helps students to improve and prevent them from

making the same mistakes again. It is useless if feedback comes back too late in the semester Students sometimes wait on the feedback before they feel

capable of going onto the next bit Grades do not equal feedback. Examinations are usually a summative form of assessment and

therefore not used for feedback.

Formative Feedback II

For students, high quality feedback consists of following three things:

1. A clear criteria against which to judge the comments.

2. Comments that are detailed and related to specific aspects of their work.

3. Comments that are improvement focussed.

Formative Feedback III

Students’ views

Agree Disagree Don’t know

Check understanding of course material

81.4% 12.7% 5.9%

Help students improve their work

99.0% 1.0% 0.0%

To explain the grade 83.3% 14.7% 2.0%

Formative Feedback IV

75.5% agree that they compare the grade received to the feedback

Does the feedback reflect the grade given? “I feel lecturers should consult students about their grades

not just tell them the grade. Recently I got some assignments back and the comments were ‘excellent, good conclusion, good reflections etc.’ Nothing bad was written on it, but I only received 50%. I can’t understand why it was so bad if all the comments were so positive.”

Formative Feedback V

Nearly a quarter of the students surveyed indicated that they would not approach the lecturer for clarification of feedback: “I think it would be helpful if lecturers took specific time to

speak with all their students to discuss exam results etc. At present lecturers leave it up to the student to come forward. This doesn’t work because students who haven’t done well are less likely to come forward to speak to their lecturer.”

Formative Feedback VI

1. Feedback should be provided by the assessor to the assessed student

2. High level feedback to the class can be useful

3. Feedback should be correlated to the assessment criteria

4. Asking the student to self-assess against criteria can prove useful

One line answer

What to assess? We assess all learning outcomes so that we can

guarantee that a student who passes a module will know (declarative) or be able to do (functioning) everything listed in the outcomes to a certain level.

General approaches to assessment I

Norm referenced assessment (NRA) Judgements about people Measurement model

Criterion referenced assessment (CRA) Judgements about performance Standards model

General approaches to assessment II

Measurement model Grading on a curve Marking by counting bottom up and averaging Assessment separated from teaching Multistructural, not relational or extended abstract thinking Verbatim responses gain marks Easily fit into most structures

General approaches to assessment III

Standards model Standards or criteria set as objectives Most students should be able to reach standards at an

acceptable level Different performances can reflect the same standards Performances can be judged against the criteria Holistic assessment – functional and declarative

knowledge working together

Types of Assessment

Examinations Normally unseen by students prior to the day they

are attempted Sample a range of student learning and test the

student’s own work Encourage students to memorise information

rather than attempting to understand it as a component of their overall course

Types of Assessment

Take home assignment Students given a period of time ranging from

weeks to months Gives time to reinforce theory delivered in class Allows students develop own model, through trial

and error Requires students to work without assistance for

the majority of the time

Types of Assessment

Laboratory Work Immediate feedback Can observe process Identify strengths and weaknesses

Types of Assessment

Class Presentations Opportunity to assess on both structure and

content The criteria should be made clear to the students Communication skills are key transferrable skills

Types of Assessment

Essays / Research Assess, Analyse, Evaluate ......... Two approaches to grading

Analytic Holistic

Types of Assessment

Online Tests Multiple choice questions Complete objectivity in marking the test Largely used to test factual material and the understanding

of concepts Used for testing large groups Time consuming to set No credit for partial information May encourage guessing

Types of Assessment

Problem Based Learning An instructional method that challenges students to learn to

learn Work cooperatively in groups to seek solutions to real

world problems Used to engage students' curiosity and initiate learning the

subject matter Prepares students to think critically and analytically, and to

find and use appropriate learning resources

Types of Assessment

Reflective Journal Develop an awareness of processes Write in your reflective journal as soon as

possible after an event

Assessment for Learning

Assessment for Learning Students know at the outset of a unit of study what

they are expected to learn. 1. At the beginning of the unit, the teacher will work with the

student to understand what she or he already knows about the topic as well as to identify any gaps or misconceptions (initial/diagnostic assessment).

2. As the unit progresses, the teacher and student work together to assess the student’s knowledge, what she or he needs to learn to improve and extend this knowledge, and how the student can best get to that point (formative assessment).

Assessment for learning occurs at all stages of the learning process.


According to The UK Assessment Reform Group (1999) improving learning through assessment depends of five factors:

1. The provision of effective feedback to the students.2. The active involvement of students in their own learning.3. Adjusting teaching to take account of the results of

assessment.4. A recognition of the profound influence assessment has

on the motivation and self esteem of the students, both of which are crucial influences in learning.

5. The need for students to be able to assess themselves and understand how to improve.


• How do we do this ?

• Sharing learning goals with students.

• Involving students in self assessment.

• Providing feedback which leads to students recognizing and taking the next steps.

• Being confident that every student can improve.


• What barriers are there ?

• A tendency for teachers to assess quantity and presentation of work rather than quality of learning.

• Greater attention given to marking and grading, much of it tending to lower self esteem of students, rather than providing advice for improvement.

• A strong emphasis on comparing students with each other, which demoralizes the less successful learners.

Primary UsersAssessment of

Learning Policy makers Program planners Supervisors

Assessment for

Learning Students Teachers Parents

Typical UsesAssessment of

Learning Certify competence Sort students

according to achievement

Assessment for

Learning Help students see

and hit the target Help teachers

identify and respond to student needs

Teacher’s RoleAssessment of

Learning Follow test administration

procedures Use results to help

students reach standards

Assessment for

Learning Inform students of

targets Build assessments Adjust instruction Involve students in

assessment

Student’s RoleAssessment of

Learning Study to meet standards Take the test Strive for the highest

score possible Avoid failure

Assessment for

Learning Strive to understand the

target Act on classroom

assessment to improve Encourage success

The Assessment Goal

To create and maintain a balanced assessment system that includes high quality assessments of and for student learning.

Question

Assessment of learning = Summative Learning ??

Assessment for learning = Formative Learning ??

Question

Teacher-centred = Summative Learning ??

Student-centered = Formative Learning ??

Student Centred Assessment

Multi-choice Questions As we discussed previously from a teacher-

centred perspective multi-choice questions have little to recommend them, but from a student-centred point-of-view multi-choice questions are telling the student “Here are N questions and I expect you to know the answers to all of them, if you find yourself guessing a lot of answers, you really don’t know the material as well as I want you to.”

Involving Students in Assessment I Ultimately the goal of any university course is for students to be

able to make judgments on their own and others work The more time students spend on higher-level abilities like

analysing and evaluating, the better they will get at assessment One approach to getting students involved in their assessment

by having them develop their own assignment questions If students are concerned with the subject’s assessment they will

be encouraged to engage in their subjects in deeper and more meaningful ways

Involving Students in Assessment II

Peer assessment Students comment on and judge their colleagues

work Vital role to play in formative assessment Can also be used as a component in a summative

assessment package studies have found student ratings of their colleagues

to be both reliable and valid

Involving Students in Assessment III

Peer assessment – advantages1. Helps students to become more autonomous, responsible

and involved2. Encourages students to critically analyse work done by

others, rather than simply seeing a mark3. Helps clarify assessment criteria4. Gives students a wider range of feedback5. More closely parallels possible career situations where

judgment is made by a group6. Reduces the marking load on the lecturer

Involving Students in Assessment IV

Peer assessment – disadvantages1. Students may lack the ability to evaluate each other. 2. Students may not take it seriously, allowing friendships,

entertainment value, etc. to influence their marking. 3. Students may not like peer marking because of the

possibility of being discriminated against, being misunderstood, etc.

4. Without lecturer intervention, students may misinform each other.

Involving Students in Assessment V

Involving Students in Assessment VI

Self assessment the involvement of the students in identifying standards

and/or criteria to apply to their work making judgements about the extent to which they have

met these criteria and standards this form of assessment is commonly a supplement to

teacher assessment of students, but in some cases it may replace it.

BSc Computer Science I

SUMMATIVE

FORMATIVE

“diagnostic use of assessment to provide feedback to teachers and students over the course of instruction”

“generally takes place after a period of

instruction and requires making a judgment about

the learning that has occurred e.g. by grading

a test”

Categories of Assessment

BSc Computer Science II

SUMMATIVE

FORMATIVE

Laboratory Work

Assignments

Assignments

Examination

Categories of Assessment

Learning Outcomes

Programme Learning

Outcomes

Develop and

Evaluate

Learning & Teaching Strategies

Assessment Criteria

Assessment Methods

Outline of Module Content

Module Learning

Outcomes

Module

Aims

Programme Aims and Content

One line summary

Assessment is the senior partner in the three way relationship also involving learning outcomes and teaching and learning strategy, because it heavily influences how learners approach a course and what they will ultimately know and do.

Have a look at…

John Biggs, Teaching for Quality Learning at University, Open University Press, UK, 2003

Assessment – Preparing, Delivering, Assessing Damian Gordon.

Documents

Transcript of Assessment – Preparing, Delivering, Assessing Damian Gordon.