TA in Edinburgh

Toby Bailey

June 23, 2011

Maple TA

I Maple TA is a system for e-assessment particularly aimed atMathematics.

I It allows questions with numbers or formulas as answers,not just multiple choice.

I It allows algorithmically generated questions, so everystudent gets a different example with the same underlyingpattern.

I It uses Maple (a computer algebra system) as an underlyingengine for both the algorithmic generation, generation offeedback and marking.

Some questions

Some questions

Some questions

Some questions

Some questions

Some questions

Algorithmically generated questions

Each student getting their own version of a question hasadvantages:

I Collaboration has to be more active. (Need to understand wellenough to change the numbers.)

I It makes it more reasonable to allow multiple attempts at atest.

I One question provides lots of versions for practice.

I One can (and we do) write feedback in the system using theactual parameters of the students questions.

I Reasonable to allow students to print out a copy of theirtest and come back later to answer it.

But not all things you want to test (particularly outsidemathematics) are amenable.

The Maple engine

For maths purposes the ability to do advanced computer algebra isvery important.

I Grading the answer to “Evaluate∫

3(x − 1)2dx”

I (x − 1)3 + C and x3 − 3x2 + 3x + C both correct.I Let S equal simplify( d/dx(students answer) - 3(x − 1)2 )

If this is non-zero then 0 marksElse, if students answer contains precisely 2 indeterminatesthen full marksElse, if students answer contains one indeterminate then fullmarks −1 (forgot constant of integration).

Also enables one to be much smarter about setting questions.

How have we used it?

I Around half of the “weekly hand-ins” for Y1 maths forsci/eng for the last 3 years.

I 10-point Y1 Calculus assessed entirely by 4 TA tests.I Each test can be taken multiple times before a final “due

date”.I A very different assessment dynamic from the “on-off exam”.

I In Studio tutorials.I Group needs to agree an answer before entering it. “Buzz”

from instant response.

Course results

I Average of best 5 of 6 TA assessments. Average of best 3 of4 written assessments.

I Exam mean ≈ 60. Both forms of CA mean ≈ 70.

I TA correlates better with exam (0.7 versus 0.6).

I Fewer students have very high TA scores but poor exam.

Reflections

It’s tempting to think of e-assessment as just a labour-savingpartial alternative to conventional CA but:

I A big weakness of the “usual” cycle of lecture - problems -hand-in - mark - return is that by the time the feedbackarrives the course has typically moved on.

I Research suggests that attempt followed by immediate,specific feedback and a new similar problem should be moreeffective.

E-asessment provides a mechanism for implimenting what shouldbe a rather better learning model.

Negatives

I Students may perceive e-assessment as a money-savingexercise.

I Have to sell it to students as something in addition to thefinite human resources that we have available.

I Students find it unforgiving: “I just left out a few bracketsand it gave me zero marks.”

I Question authoring is very time-consuming.

The Future

I E-assessment is here to stay, but I am a long way fromunderstanding the “dynamics” of e-assessment.

I How does repeated attmpts at (algorithmically generated)e-assessment differ from the “usual exam system”?

I Should we have infinitely repeatable practice tests and one-offassessed tests?

I Can we make doing the same question 10 times with differentnumbers appealing?

I Can we achieve novel things with e-assessment - e.g. “Just InTime Teaching”

I We give credit for what ought to be formative assessment.The “chocolate cake and fruit salad problem”.

And in future, how do we balance TA with use of the systems thatcome with textbooks?