Assessment Feedback Systems and Interactive Learning Environments The Open University, Walton Hall,...

Assessment Feedback Systems and Interactive Learning Environments

The Open University, Walton Hall, Milton Keynes MK7 6AA

Professor Denise Whitelockdenise.whitelock@open.ac.uk

DMW Congress Colombia November 2014

I hate marking but want the tasks and feedback to assist student learning

The e-Assessment and automatic feedback Challenge

• Constructivist Learning – Push

• Institutional reliability and

accountability – Pull

DMW Congress Colombia November 2014DMW Congress Colombia November 2014

www.storiesabout.com

www.storiesabout.com/creativepdp

c.mckillop@rgu.ac.uk

MCQs: Variation on a theme

Example of LAPT Certainty-Based Marking

UK cabinet ministers demo exercise

showing feedback,

University College, Tony Gardner-Medwin

Drug Chart Errors and Omissions

Medicines Administration Assessment,

Chesterfield Royal Hospital

MCQs: High Stakes Assessment

Example of practice “Thinking Skills Assessment" (TSA) question, Admissions Interview, Cambridge Assessment, Steve Lay

Example of practice “Thinking

Skills Assessment"

(TSA) feedback, Admissions Interview,

Cambridge Assessment, Steve Lay

Scaffolding and High Stakes assessment

• Math for Science

• Tutor less course

• Competency led

• No point to cheat

• Web home exam

• Invigilation technologies

Self diagnosis

• Basic IT skills, first year med

students (Sieber, 2009)

• Competency based testing

• Repeating tests for revision

• Enables remedial intervention

Theoretical drivers for Assessment and Feedback

Piaget (1930) Individual manipulating surroundings --

mental representations can be tested Induction of rules and

their testing ... Nuffield science

Cognitive psychology (70s 80s) How are these mental

representations stored? Classification development and how

this can go wrong. .... Misconceptions and mal rules........

Student modelling .... AI

Bruner (1982) assessment tasks will match competency

levels depending on level of help...SCAFFOLDING

Pask conversation model used by Laurillard

Scaffolding and Interactivity; The Open University Science Foundation Course

• Interaction, Feedback loops

• Tell, Explore, Check

• Predict, Look and Explain

• Entering the discourse of a subject via audio feedback

• Scaffolded text feedback (Bruner & Woods)

• SHOW ME button

Interactive Tasks

• Games

• Simulations

• Making the abstract concrete

• Directing the sequence of an

animation

Interactivity and Cognitive Change Scores

Theoretical drivers: Social Constructivism

• Vygotsky (1978) individuals shape cultural settings

which shapes minds no longer individual

• Activity theory (Engstrom 1987) Tool mediation

• Situated Cognition (Lave and Wenger 1991)

Authentic assessment

• Peer interaction and assessment

• Learning conversations Laurillard (2002)

Characteristics Descriptor

Authentic Involving real-world knowledge and skills

Personalised Tailored to the knowledge, skills and interests of each student

Negotiated Agreed between the learner and the teacher

Engaging Involving the personal interests of the students

Recognise existing skills Willing to accredit the student’s existing work

Deep Assessing deep knowledge – not memorization

Problem oriented Original tasks requiring genuine problem solving skills

Collaboratively produced Produced in partnership with fellow students

Peer and self assessed Involving self reflection and peer review

Tool supported Encouraging the use of ICT

Elliott’s characteristics of Assessment 2.0 activities

Authentic Assessment: Building e-portfolios on a chef’s course

Food preparation for e-portfolio

Modern Apprenticeship in Hospitality and

Catering, West Suffolk College, Mike Mulvihill

Evidence of food preparation skill for e-portfolio

Modern Apprenticeship in Hospitality and Catering,

West Suffolk College, Mike Mulvihill

Peer Assessment and the WebPA Tool

• Loughborough (Loddington et al, 2009)

• Self assess and peer assess with given criteria

• Group mark awarded by tutor

• Students rated:

• More timely feedback

• Reflection

• Fair rewards for hard work

• Staff rated:

• Time savings

• Administrative gains

• Automatic calculation

• Students have faith in the administrative system

Mobile Technologies and Assessment

• MCQs ,PDAs Valdiva &

Nussbaum(2009)

• Polls,instant surveys

• Simpson & Oliver (2007)

• Draper (2009) EVS

Gains from Formative Assessment

• Mean effect size on standardised tests between 0.4 to 0.7

(Black & Williams, 1998)

• Particularly effective for students who have not done well

at school http://kn.open.ac.uk/document.cfm?docid=10817

• Can keep students to timescale and motivate them

• How can we support our students to become more

reflective learners and engage in formative assessment

tasks?

Collaborative formative assessment with Global Warming

DMW, Institute of Educational Technology, September 1997DMW, Institute of Educational Technology, September 1997

Global Warming

Next: ‘Yoked’ apps via BuddySpace

Student A

Student B(‘yoked’, butwithout full screen sharingrequired!)

Global Warming: Simlink Presentation

LISC: Aily Fowler

Kent University ab-initio Spanish module

•Large student numbers

•Skills-based course

•Provision of sufficient formative assessment meant

unmanageable marking loads

•Impossible to provide immediate feedback

• leading to fossilisation of errors

The LISC solution: developed by Ali Fowler

A call system designed to enable

students to:

•Independently practise sentence

translation

•Receive immediate (and robust) feedback

on all errors

•Attend immediately to the feedback

(before fossilisation can occur)

How is the final mark arrived at in the LISC System?

The two submissions are unequally weighted

•Best to give more weight to the first attempt

• since this ensures that students give careful

consideration to the construction of their first

answer

• but can improve their mark by refining the answer

•The marks ratio can vary

(depending on assessment/feedback type)

• the more information given in the feedback, the

lower the weight the second mark should carry

Heuristics for the final mark

If the ratio is skewed too far in favour of

the first attempt…

• students are less inclined to try

hard to correct non-perfect

answers

If the ratio is skewed too far in favour of

the second attempt…

• students exhibit less care over the

construction of their initial answer

What about emotional support in the feedback?

• Difficult at times to receive written feedback

• Not just a cognitive response

• How can Bales help?

Coding into Categories

• Bales analysis

• Psychology 1950s

• Analyses talk

• Includes socio-emotive categories

• Flander’s (1970) categories

inappropriate as also includes

classroom control

Bales Categories

• Four main groupings

• A. Positive reactions; agreeing

and boosting the other person

• B. Directing/teaching

• C. Questions: requesting

information, clarification etc

• D. Negative reactions:

disagreement

Coding the tutor comments

Categories Specific Examples

Positive Reactions

1. Shows solidarity

2. Shows tension release

3. Shows agreement

Jokes, gives help, rewards others

Laughs, shows satisfaction

Understands, concurs, complies, passively accepts

Attempted Answers

4. Gives suggestion

5. Gives opinion

6. Gives information

Directs, proposes, controls

Evaluates, analyses, expresses feelings or wishes

Orients, repeats, clarifies, confirms

Questions

7. Asks for information

8. Asks for opinion

9. Asks for suggestion

Requests orientation, repetition, confirmation, clarification

Requests evaluation, analysis, expression of feeling or wishes

Requests directions, proposals

Negative Reactions

10. Shows disagreement

11. Shows tension

12. Shows antagonism

Passively rejects, resorts to formality, withholds help

Asks for help, withdraws

Deflates others, defends or asserts self

Bales’ Interaction Process

Identifying trends: H801

Graph to show conflated Bale’s

categories against mean number

of incidences in H801 scripts

0 5 10 15 20 25

A Pass 1

A Pass 2

A Pass 3

A Pass 4

B Pass 1

B Pass 2

B Pass 3

B Pass 4

C Pass 1

C Pass 2

C Pass 3

C Pass 4

D Pass 1

D Pass 2

D Pass 3

D Pass 4

Number ofIincidences

Identifying trends: H801

Pie Chart to show the mean

number of incidences per pass per

conflated Bales' Interactional

Category for all four levels of pass

in H801 scripts

A = Positive reactions

B = Responses

C = Questions

D = Negative reactions

Identifying trends

Pie Charts to show the mean number of incidences per conflated Bales Interactional

Category for ‘Pass 1’ and ‘Pass 4’ in the following courses:

Key:A = Positive reactions C = QuestionsB = ResponsesD = Negative reactions

Pass 4

Pass 1

B820 S103

What is Open Mentor?

“An open source mentoring tool for tutors”

“Open source” = free and easy to use, and to embed in an

institutions infrastructure and working practices

“mentoring” = designed to help people learn how to give

feedback effectively, through reflection and social networks

“tutors” = primarily intended for teaching staff, but with clear

applications for those involved in quality

How Open Mentor handles comments

“Good work”

“Yes, well done”

“Yes, but is this useful?”

“Can you explain what you mean”

“This does not follow”

A = positive reactions

B = attempted answers,

and not a positive reaction

C = questions

D = negative reactions

Explaining OpenMentor’s Rules

Four categories

•A – Positive Reactions

•B – Attempted Answers

•C – Questions

•D – Negative Reactions

‘A’ - Positive Reactions

Category Examples of Rules Examples of commentsA - Positive Reactions1. Shows solidarity A1 ...excellent... Excellent Conclusions.

A1 ...(good|comprehensive)... Good, you are drawing on hard facts here.A1 ...nicely... Very nicely stated. Your analysis is thorough and your

conclusions consistent regarding the attractiveness of the budget airline sector. This is a good example of critical thinking.

A1 ...well presented... Very well presented diagram with interesting information.A1 ...effective use... Effective use of the case material here.A1 …well (structured|stated)… Report very well structured.A1 ...(well|clear)(ly)*

The corporate vs. business unit strategy is well presented and nicely tied to strategies.

A1 ...reasonable.... A reasonable structure as listed in your table of contents.A1 ...useful point(s)... Generally useful points in this section.

2. Shows tension release A2 ... a helpful...

A2 …(thanks|thank you)…3. Shows agreement A3 ...yes... Yes, the intellectual reactions are both real.

A3 ...indeed... Indeed – if it has one basic strategy it is surely differentiation, though it still has to control costs.

4. Praise then direction A4 good...but... Good model, good quote, but be careful about what industry you analyse ??

‘B’ Attempted Answers

Category Rules Examples of commentsB - Attempted Answers

B4 …perhaps… Perhaps even better here to explain the link in your mind between "analysis of stratgeies" and "strategic issues".

B4 …requires…B4 …take care… Take care with your STEP analysis not to make it too industry

focussed.B4 …useful to… Innovation is closely linked to structure and culture- it would be

useful to see some book 6-8 concepts here too.

B4 …you (might|ought)… You ought to have explicitly stated these.

B4 Don’t|never … Don’t introduce new frameworks just for the sake of it in the conclusion. The conclusion should be pulling together what went before.

B4 Please (see|refer to|look at)... Please make sure to read and understand the question correctly

5. Gives opinion B5 I (am|think)... I think I can see where you are going, though a numbered report format might have demonstrated the approach better

B5 This is.... this is an introduction rather than a “summary”B5 ...sounds...like... This sounds as if it could be very popular!!B5 ...not sure... I am not sure about the balance between the environmental

analysis and the review of the resources, capabilities (power, culture, structures and systems) as raised in the question.

B5 I (thought|agree|suggest)… I thought it was because they did not need any external input and saw a significant market sector they could address themselves.

B5 I (do|don’t) think… I don’t think this exercise has helped to develop your analysis. I also think that the development of the perspectives is superficial

6. Gives information B6 …(demonstrates|shows) this….B6 Also… Also, cross link to Leadership issues, Pettigrew on Strategic

Thinking tooB6 ...Q1... etc Q1 = 59/100

Is the rule set generic?

Comments Classified from Test Data

0102030405060708090

B820 TestSet 1

B820 TestSet 2

B820 TestSet 3

A850 M878 S809

Test Data Set

What can we learn from modelling tutors marking to construct a formative e-assessment tool?

• Open Comment project builds on

the work of OpenMentor

• Free text entry for History and

Philosophy students

• Immediate feedback (in context)

to students

• Influenced by ELIZA

(Weizenbaum, 1963)

Open Comment addresses the problem of free text entry

• Automated formative assessment tool

• Free text entry for students

• Automated feedback and guidance

• Open questions, divergent assessment

• No marks awarded

• For use by Arts Faculty

Causal models of explanation

First step:

• Identification of question types where

students exhibit causal reasoning

Looked for questions with:

• Salient variables

• Description of these variables

• Identification of trends

• Identification of relationship between

the variables i.e. causality

Praise for effort and not just ability

• Praise for ability per se can hinder

learning (Mueller & Dweck, 1998)

• Praise = being clever

• Negative feedback now

without ability

• Disempowering and demoralising

Praise and motivation: feedback and self regulation

• “Feeling good” feedback part of many guidelines

• Mueller & Dweck (1998) Praise for effort as well as

performance

• Raven’s test given to USA school children

• First test feedback was praise

• Second test most difficult. Half praised for effort,

half praised for ability

• Third test medium difficulty

• On the third test pupils praised effort increased score by

1. Praised for ability decreased 1 mark (scale 0-10)

How does feedback effect mindsets?

1. Your intelligence is something very basic about you

that you can’t change very much

2. You can learn new things but you can’t really change

how intelligent you are

3. No matter how much intelligence you have you can

always change it quite a bit

4. You can always substantially change how intelligent

you are

Mindsets (Dweck, 2006)

• Fixed mindset

• Super sensitive about being

• Always trying to prove

themselves

• Growth mindset

• Stretch themselves

• Confront obstacles as

challenges

• Lack of tension when

learning as they know they

are novices and can

improve

Stages of analysis by computer of students’ free text entry for Open Comment: Advice with respect to content (socio-emotional support stylised example)

Stage 1a: Detect errors

E.g. Incorrect dates, facts. (Incorrect inferences and

causality is dealt with below)

•Instead of concentrating on X, think about Y in order to

answer this question Recognise effort (Dweck) and

encourage to have another go

•You have done well to start answering this question but

perhaps you misunderstood it. Instead of thinking about X

which did not…….. Consider Y

Computer analysis continued

Stage 2a: Reveal first omission

Consider the role of Z in your answer Praise

what is correct and point out what is missing

Good but now consider the role X plays in your

answer

Stage 2b: Reveal second omission

Consider the role of P in your answer Praise

what is correct and point out what is missing

Yes but also consider P. Would it have

produced the same result if P is neglected?

Final stages of analysis

• Stage 3:request clarification of key point 1

• Stage 4: request further analysis of key point 1

– stages 3 and 4 repeated with all the key points

• Stage 5:request the inference from the analysis

of key point 1 if it is missing

• Stage 6:request the inference from the analysis

of key point 1 if it is not complete

• Stage 7:check the causality

• Stage 8:request all the causal factors

are weighted

Open Comment drivers for reflection

• Students are able to find facts similar to X

• Know how X might be disputed

• Are able to make predictions about X

• Know how to use X in an argument

• Know how far X can be pushed

• Supported with tools and strategies for effort

SAFeSEA

Professor Denise Whitelock

Professor John Richardson

Professor Stephen Pulman

An automated tool supporting

online writing and assessment

of essays providing

accurate targeted feedback

SAFeSEA: Supportive Automated Feedback for Short Essay Answers

About SAFeSEA

• No tutor support for drafts of first assignment

• Reduce dropout rate with automatic feedback?

• Effect of summaristion

• What are the beneficial factors?

• Correlate measures of learner activity and

essay improvement

• http://www.open.ac.uk/researchprojects/safesea/

Talk Back

• Checking understanding by ‘talk back’

• Summaries in OpenEssayist

• Key words = key ideas

• www.open.ac.uk/researchprojects/safesea/

OpenEssayist: What it tells you

The system’s focus is to present summaries of your own work

in different ways, to encourage you to reflect constructively on

what you have written.

In other words Open Essayist tells you from its analysis what

are the most important or key points in your essay. You can

then think about whether that was what you intended to

emphasis in your essay. If not then you can make the

appropriate changes.

A very important aspect of the OpenEssayist system is that it

will not tell you what to write, or how to rewrite sections of

your essay, or even what is correct or incorrect in your essay.

GRAPHICS

ANALYSIS

OpenEssayist: How it gives feedback

Three aspects of your essay are analysed by the system:

• the structure of the essay (which paragraphs constitute

the introduction, the conclusion, the discussion

sections, etc.),

• the key words and key phrases of your essay (which

are the most important words and phrases, the ones

that are most representative of your essay's overall

meaning)

• the key sentences of your essay (which are whole

sentences that are most representative of your essay's

overall meaning).

Sample key phrases dispersion plot

Visualisations as a Thinking Tool

• Visualising text (Bertin, 1981;

Johnson et al, 1993)

• Free text visualisation still

problematic

• Do users need training with

visualisations?

• Convention vs. Instruction?

Short text for illustration of Rainbow Diagrams

Sentence graph of short text

Pretend essay: 10 identical paragraphs

Pretend essay: 50 identical sentences

Stanford University Boothe Prize essay

OU essay awarded high grade

OU essay awarded low grade

Good vs. bad?

•not densely connected•red nodes (conclusion)

not central•few links between violet (intro)

and red nodes

•densely connected•red nodes (conclusion)

central•close links between violet

(intro) and red nodes

Findings

“One of the clues was talking about the way the color nodes, that the red ones are at the end. If they are a good connection with a good explanation for each paragraph. They should be connected together but the darker ones should be in the middle. But you see the color groups together so for me it automatically pulls my eye to this page because all the colors are closer together and more in the middle. So that would be the student course assignment essay with the highest mark.”

“You don’t show me anybody’s text. You are not revealing anyone else’s essay. So students cannot plagiarize. But you are saying ‘Look hang on, this is the way this essay connects together.’ That’s what telling a good story is about this linking.”

Rainbow diagrams related to mark awarded

• Multivariate analysis of variance on marks

awarded to 45 students

• Submitted two essays

• Rainbow diagrams produced from these essays

and rated as high, medium or low attainment

• Covariate showed a significant relationship with the marks

• F(1, 43) = 5.92, p = .01 using a directional test

• Essays rated as high would be expected to receive 8.56

percentage points more than essays rated as medium

• 17.2 percentage points higher than essays rated

from rainbow diagrams as low

Is NLP the bridge between Computer Assisted Assessment and Learning Analytics?

• Automatic marking

• Recognising text

• Refining text

How about feedback first?

• Hints before writing?

• R.C.T.

• 2 essays

• F(1,41) = 3.23 p = 0.04 for hints

Creating teaching and learning dialogues: towards guided learning supported by technology

• Learning to judge

• Providing reassurance

• Providing a variety of signposted

routes to achieve learning goals

• Provide socio-emotive support

Feedback: Advice for Action

• Students must decode feedback and

then act on it Boud (2000)

• Students must have the opportunity to

act on feedback Sadler (1989)

• Gauging efficacy through student action

Characteristics Descriptor

Authentic Involving real-world knowledge and skills

Personalised Tailored to the knowledge, skills and interests of each student

Negotiated Agreed between the learner and the teacher

Engaging Involving the personal interests of the students

Recognise existing skills Willing to accredit the student’s existing work

Deep Assessing deep knowledge – not memorization

Problem oriented Original tasks requiring genuine problem solving skills

Collaboratively produced Produced in partnership with fellow students

Peer and self assessed Involving self reflection and peer review

Tool supported Encouraging the use of ICT

Elliott’s characteristics of Assessment 2.0 activities

Advice for Action

The 4Ts Pyramid

Tool Development

Training of staff

TransformationTasks

TransferLearning

Training

• Question development

• Quality REAQ

• Plagiarism software

• Advice for Action

• Socio e-emotive content

• Maintain empathy with the Learner

e-Assessment Futures

• Research issues

• Adaptive testing

• Automatic marking

• Learning Analytics Data mining

• Web 2.0, 3.0 …

Feedback to prompt self reflection

• Analysis must prompt “Advice for

Action”

• Self reflective discourse with

computer feedback

• Visual representation of feedback

can open a discourse between

tutor and student

• Prompt peer to peer discourse

References 1

Whitelock, D. (2010) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global. pp. 319–342.

Whitelock, D. (2008) ‘Accelerating the assessment agenda: thinking outside the black box’. In Scheuermann, F. & Pereira, A.G. (eds.) Towards a Research Agenda on Computer-Based Assessment: Challenges and Needs for European Educational Measurement, published by the European Commission, Joint Research Centre. pp15-21 ISSN. 1018-5593

References 2

Five assessment special issues

Assessment Feedback Systems and Interactive Learning Environments The Open University, Walton Hall,...

Documents

Transcript of Assessment Feedback Systems and Interactive Learning Environments The Open University, Walton Hall,...

CDE Conference 09/02/2009. D Whitelock

1847~t~1 S20. Henry RObert son of Josepn and …whitlockfamilyassociation.com.s3.amazonaws.com/sources/...1656-1657--Payment into the '.rreas'lry.Allthen.tic Whitelock signatilre.

TELECOMMUNICATIONS RECEPTION SET AR-88 · 2020. 12. 18. · ELECTRICAL AKD MECKLNICAL ENGINEERING REGULATIONS TELECOMMUNICATIONS G¥-6aa-RECEPTION SET AR-88 DESCRIPTION (Provisional)

Psychological Effects of Stroke - NHS England...Psychological Effects of Stroke 26 June 2019 / Dr Claire Whitelock, St George’s Hospital and Wandsworth Community Neurorehabilitation

Salhouse Road, Rackheath, NR13 6AA - OnTheMarket · Salhouse Road, Rackheath, NR13 6AA A handsome, individual four bedroom detached house with a large garden in a non-estate village

Whiteloc harles Ray Delaware - Amazon Web Serviceswhitlockfamilyassociation.com.s3.amazonaws.com/sources/... · Charles Ray Whitelock J,.. b. February 5, 1885 Wilmington, Delaware

Learning Analytics @open.edu + @ou · learners educators Denise Whitelock: Can we provide personalised feedback to free text answers? ?!*?!* ?!*?!* A learning analytics ecosystem

INLEC · bs 5411-11 bs 3900-c5-6aa bs en iso 1461 din 50981 iso 2178 iso 2808-6aa pren iso 19840 ... elcometer 456 integral gauge - specifications and part numbers

Descendants of Charles WHITELOCKwhitlockfamilyassociation.com.s3.amazonaws.com/sources/miscella… · Descendants of Charles WHITELOCK sp: John Sherman FOWLER (m.(Div» ~ 10. Sebastian

Writing motivating feedback Professor Denise Whitelock The Open University, Walton Hall, Milton Keynes MK7 6AA, UK denise.whitelock@open.ac.uk.

Assessment for Learning:Where are we on the e-Assessment spectrum? Denise Whitelock The Open University, Walton Hall, Milton Keynes MK7 6AA d.m.whitelock@open.ac.uk.

Lafayette Orinda Presbyterian Church- Connections WEEKLY … · Kristi Haigh, Foundation Trustee at akhaigh3@comcast.net, or call (925) 254-6023. Peter Whitelock, Pastor / Head of

Dr Jacqueline Baxter The Open University Walton Hall Milton Keynes MK7 6AA Jacqueline.Baxter@open.ac.uk.

sealang.netsealang.net/sala/archives/pdf8/riddle1994relativization.pdf · los tuab or thick 'Do you plant rice spaced far apart or close together?' (Whitelock 1982:86, with amended

Water governance in England - Open Universitymcs.open.ac.uk/CADWAGO/programme.pdfOur mailing address is: The Open University, Walton Hall, Milton Keynes, MK7 6AA, UK CADWAGOstands

By Ron Whitelock Under the Hood of GPS: Introduction.

NOW LEASING VINEYARD AT MADEIRA SEC …images4.loopnet.com/d2/gNK7mBVSDlXGGNM3zhcUVI7sTAjiLZFztJnFybg5tCA/...VINEYARD AT MADEIRA SEC BRUCEVILLE ROAD & WHITELOCK PARKWAY ELK GROVE ::

Celebratingover 120 years - Walshe & Whitelock€¦ · full legal recognition of auctioneers in the early 80’s and for Walshe & Whitelock business was brisk, very brisk, as the

Dr Jacqueline Baxter The Open University Walton Hall Milton Keynes MK7 6AA

DMW, IET, and ME, KMi, Open University, 2005 Exploring Pedagogy, Presence and Participation in Interactive Formative Assessment Denise Whitelock and Marc.

Learning Analytics @open.edu + @ou · learners educators Denise Whitelock: Can we provide personalised feedback to free text answers? ?!?! ?!?! A learning analytics ecosystem