Fine Tuning Assessment for Teaching and Learning

Roy Williams

Michael McCabe

Context

Mathematics student numbers 2002 – 2010:

• 30 to 150 in first year (plus 2nd year?). • Negligible staff increases• Standard Wiley textbook• In-house assessments (Qmark)• PRS feedback (good question bank)• Link to personal tutors (most staff)• Support from Maths Cafe

Questionmark Graphics

2010 Semester 2

• Staffing Changes• New Responsibilities• Short notice for new delivery of course

• Regional Maths Seminar (Sigma-Hub)• Awareness of Wiley Plus resources

• Opportunity to rethink course delivery

QMark: Outdated for Mathematics

• No algorithmic-randomised questions• Technical, authoring and display issues• Answering in mathematical notation

(Limits questions formats)• Cumbersome editing and graphics

updating

Development

• Meeting with Wiley Plus within month• ? Redevelopment in MapleTA?• Decision to do ‘live pilot’ – book plus

e-assessment, e-book, etc. • Two week start-up• Wiley support and resources• Wiley meeting with staff• Wiley meeting with students • Wiley assistance at launch

Advantages

• E-book provided for all • Large banks of pre-prepared questions• Links from assessment to full-text, hints,

solutions, tutorials• Learning design underlying the structure• Pre-prepared lecturer ppts:

– Supplementary PRS questions– PPT Summary (2 screen options)– Applets – interactive activities – Instructor guidance and resources– Exportable MCQ’s >> Victory

Disadvantages

• Learning rather than examination focus• Pre-prepared ppts – formulaic• Limitations on customisation, and in-house Q’s

(MapleTA underlies Wiley Plus)• Error reporting, not correction• 3/7 tests in MapleTA – (different text book)

Assessment

Formative• Pre-prepared• Selected formative (10, mixed difficulty)• Unlimited attempts on algorithmic questions• Varied question types, including interactive graphics

Assessment

Summative• Replaced repeated assessments with

repeated questions • Cumulative scoring

– Builds on success– Promotes confidence– Avoids repetition

• Q. Classification – Easy/ medium/ hard.– By learning objective

• Varied Q. types, including interactive graphics

Disadvantages

• No password facility• Administrative overheads and work-arounds• Long-term continuity, control, stability

– Year-on-year use of same tests– Upward compatibility of software

• Non-uniformity/ comparability of questions– Scoring

– Length

– Amount of work required

– Recommended time-on-task, and marks

• Discriminating between formative and summative

Evaluation Victory

• One-stop shop for information

• Hand-written lecture notes– Lectures as process?– Lectures as mathematical thinking?– Digestible summary of contents

But• Unreliability and GUI Issues• Mathematical limitations

Evaluation

MapleTA

• Ease of Use• Many practice questions

But• Input format ?• Lack of partial credit, inflexible marking

Evaluation Wiley Plus

• Multiple attempts at questions

• Cumulative scoring • Many practice questions • Feedback and links: solutions, hints,

tutorials. • Invigilators calm and helpful

But

• ? Input format• GUI?

Evaluation Useability

• MapleTA very good• Wiley good• Victory: issues

MTH 204

Confirms findings, broadly

Evaluation Method

• Survey on 6 point scale (e.g. essential to learning >> Of no use at all

• Open-text questions (positive and negative)

• MTH 155: 68/150• MTH 204: 11/120

Questions Arising

• To ASP or not to ASP, if so, 100% or not? • Alternative philosophies (OS, Commercial) • Sustainability • Viable hybrid model?• Learning (L&T model – TEL&T)?• Scaffolding – lecturer, interactive resource, both?• Educational benefits

– Repeat assessments or questions?– Integrated multi-mode resources?

• Trade-off: security and admin. load.

• Customisation of resources