Nidhi Parikh March 12, 2012 Automated Assessment and Question Banks.

Nidhi ParikhMarch 12, 2012

Automated Assessment and Question Banks

References

• Joint Information Systems Committee, Effective Practice with e-Assessment, 2007.

• http://docs.moodle.org.• http://classes.sscnet.ucla.edu/docs/

Using_GIFT_format_to_create_multiple_questions_at_once.

• North Carolina A&T State University, Blackboard Support, Standard Respondus Format for Microsoft Word Questions.

Outline• Automated Assessment• What is e-Assessment?• Forms of Assessment• Motivation• Important Considerations • Assessment and Learning• Case Study 1• Case Study 2• Case Study 3• Challenges

• Question banks• Requirements• Standard Question Formats• GIFT: Moodle Format Examples• Case Study 4

• Conclusion

What is e-Assessment?

• Broad definition:• Any assessment activity that uses digital

technology including• Design and delivery of assessment• Marking by computers or by scanners• Reporting, storing and transferring of assessment

data

• JISC definition:• e-Assessment is the end-to-end electronic

assessment processes where ICT is used for the presentation of assessment activity, and the recording of responses.

Forms of Assessment• Based on at what stage of learner’s activity, it is

taken:

• Diagnostic • Assessment of a learner’s knowledge and skills at

the outset of a course.

• Formative • Assessment that provides developmental

feedback to a learner on his current understanding and skills.

• Also known as assessment for learning.

• Summative • The final assessment of a learner’s achievement,

usually leading to a formal qualification or certification of a skill.

• Also known as assessment of learning.

Motivation

• Can lead to more effective learning• Perhaps the best way to identify support

needs of learner.• Provide good quality, timely feedback and

appropriate resources.• Simulating and challenging ways to

demonstrate understanding.• Interactive and engaging tests could help

people with disabilities.

• Supports Personalization• On demand assessment can help progress at

their pace.• Students can be fast tracked.

Motivation• Ability to capture aspect of learning

previously impossible to capture.

• Reduced workload.

• Timely feedback/evaluations.

• Improved quality of feedback/evaluations.

• Multiple tests are possible.

Important Considerations• Closer alignment of assessment with the

pedagogic approach used.• Timeliness of assessment and quality of

feedback are important.• Interactive and engaging assessment.

Assessment and Learning

The relationship between assessment and effective learning

Effective Practice for e-Assessment, JISC 2007

The TRIADS: Case Study 1• Background:• Tripartite Interactive Assessment Delivery

System (TRIADS).• Developed at the University of Derby in 1992

and further developed in partnership with the University of Liverpool and The Open University.

• In 2005-2006, it delivered assessments to 10000 students in Derby.

The TRIADS: Case Study 1• System:• Flexible design structure.• Capability to build in simulation and

multimedia.• Screen layout developed over many years

according to current guideline and accessibility.• Code templates for generic question styles.• Code template can be used in combination.• Supports randomized selection and sequencing

of questions.• Contain full error trapping routines.• Facilitate partial credits for methods, stages in

answer and final answer.• Detail candidate performance reporting is

provided helping to pinpoint atypical result.• Delivered over web, LAN or CD.

The TRIADS: Case Study 1

The TRIADS: Case Study 1• Outcome:• Despite sophistication offered, 70% of use in

1st year, 20% in 2nd year and 10% in 3rd year.• May be due to workload so a centralized

support unit is set up.• Frequency and timing of access to test reveal

that students retake tests.• Suggests that students are more likely to test

themselves with e-assessment than with pen-paper test.

Summative Assessment: Case Study 2

• Background:• At the School of Pharmacy and

Pharmaceutical Science, at the University of Manchester.

• A subject with emphasis on accuracy.• Have experienced with diagnostic and

formative tests before trying summative tests.

• 240 students, some with disabilities taking first year module in cell biology and biochemistry.


• System:• Tests delivered on WebCT.• Questions written in Microsoft Word,

imported into WebCT using Respondus.• Student identity authenticated by university

username and password and photo ID check.• Timed release of exam.• Students could enter response online or on

an optical marker.


• Challenges:

• Problem with importing questions with decimal points.• Editing has to be carried out in VLE.

• WebCT’s inability to work around spelling mistakes.• List of subject specific keywords provided

to students to paste correct spelling.• Reduced discrepancies between human

and computer based marking to less than 1%.


• Outcome:• Comparison with pilot study indicate that

computer based tests are more accurate compared to slower manual marking.

• Cascading style sheet allowed to customize display and helped students with dyslexia.

Formative Assessment: Case Study 3

• Developed for an online BTEC Intermediate and Advanced Award in IT skills for learners in small scale business accredited by Edexcel.

• Delivered via the Internet with online tutor support.

• Assessment ongoing and incorporated into course structure and content.

• True of false questions.• Wrong answers advise learners to revisit

activity. • Learner progress at their own pace.

Challenges• Provide customized views.• Need staff for technical support.• Increased effort before test instead of

after.• Interoperability.• Preparing question banks that can be

shared between departments and schools.• Need of standard question formats.• To make sure all students get test of equal

measure.• Ability to mark free text answers.

Question Banks• Question format requirements:• Question types• Standardization• Shared and personalized question banks• Question versioning• Import/Export to update existing questions

• Question Interface Requirements:• Easy but powerful searching• Rendering questions of equal measures• Easy organization• Customized Views

Standard Question FormatsQuestion

Type/File Format

Respondus

Moodle(GIFT)

Moodle(XML)

Aiken (simple)

Cloze

Multiple choice

√ √ √ √ √

Multiple answer

√ √ √

True/False √ √ √ √Essay √ √ √

Fill in the blanks

√ √

Short answer

√ √ √

Match √ √Number match

√ √ √

Multi question

√

File format Textdocrtf

text xml text xml

GIFT: Moodle Format Example 1

Multiple Choice Question::Question 1 - Multiple Choice::Where is UCLA located?{=Westwood ~San Diego ~Downtown Los Angeles}


Multiple Answer Question::Question 3 - Multiple Choice with multiple correct answers::Identify two former UCLA basketball players.{~%50%Kareem Abdul-Jabbar~%50%Bill Walton~Kobe Bryant~LeBron James}


Matching Question::Question 8 - Matching::Match the following countries with their capitals. {=United States -> Washington DC=Japan -> Tokyo=Germany -> Berlin=Venezuela -> Caracas}


Number Range Question ::Question 9 - Number range::What year was UCLA founded in? {#=1919:0=%50%1919:2}

The COLA Project: Case Study 4

• Background:• COLEG Online Assessment project (COLA) is

run by Scotland's College Open Learning Exchange Group (COLEG).

• The aim is to create a bank of online formative assessment across wide range of courses and levels in support of SQA qualifications.

• Assessments are objective tests containing up to 20 questions available in 17 subject areas.


• Challenges:• Developed assessment for four Virtual Learning

Environments:• Blackboard• Moodle• Serco (Teknical) Virtual Campus• WebCT

• The final question bank was delivered in IMS Question and Test Interoperability (QTI) standard compliant form.

• None of the VLEs complied with IMS QTI standard. • Solution was that question bank holds only IMS

QTI format questions.• Conversion tools were developed for different

platforms.


• Challenges:• Question types:• Multiple choice• Multiple response• True or false• Match• Fill in the blanks

• Later due to difficulties with fill in the blank they were removed from question bank.

• Problem with user tracking and scoring in some VLEs.

• Accessibility of questions also affected by the way VLEs render them.

• Influenced some colleges to upgrade or change VLE.


• Outcome:• Around 500 formative tests have been

written between 2005 o 2007.• Practitioners and learners have found them

useful.• Raised the profile of e-learning in Scotland.• Highlighted need of innovation in

interoperability of systems.

Conclusion• Studies show following benefits for

automated assessment:• More interactive and engaging environment.• Quality feedback.• Immediate feedback.• Help in effective learning.• Helps students with disabilities.• Reduces workload on academic staff.

• But there are some challenges:• Need better ways to incorporate in learning.• Need more sophisticated question formats.• Need better converters for question formats.• Interoperability.

Thank you !

Nidhi Parikh March 12, 2012 Automated Assessment and Question Banks.

Documents

Transcript of Nidhi Parikh March 12, 2012 Automated Assessment and Question Banks.