ECEL 2010 - Presentation Enhanced Approach of Automatic Creation of Test Items to foster Modern...

ECEL 2010 - Presentation

Enhanced Approach of Automatic Creation of Test Items to

foster Modern Learning Setting

Institute for Information Systems and Computer Media (IICM)Faculty of Informatics – Graz University of Technology, Austria

Christian Gütl, Klaus Lankmayr, and Joachim Weinhofer

Version October 24th, 2010

© 2010 - Christian Gütl19/04/23 2

Advanced Educational Media Technologies Group

• Remote & Collaborative Learning• Adaptive e-learning systems• Virtual learning environments

• E-Assessment• Computer-based assessment• Peer assessment

• Virtual Environments & Setups • Virtual 3D World Infrastructure• End devices (immersive … mobile)

• Multimedia IR and NLP• Access to learning repositories• Pre-processing of content

© 2010 - Christian Gütl19/04/23 3

Agenda

• Introduction

• Background & Related Work

• Requirements & Solution

• A First Qualitative Study

• Current & Future Work

Introduction

© 2010 - Christian Gütl19/04/23 4

• Knowledge-driven and globalized world requires continuous adaption (Gütl, 2010)

• New forms and settings of learning… informal learning, self-guided learning, …

• Assessment and feedback (Bransford et al., 2000)

• Time consuming, in particular for adaptive learning environments

• Almost impossible for self-directed and life-long / informal learning settings

Flexible e-assessment models & toolsAutomatic or semi-automatic text-item

creation from learning content

Background & Related Work

© 2010 - Christian Gütl19/04/23 5

• Not much has be done so far• Multiple choice test item (Coniam, 1997)

part of speech tagging for word candidatesword distractors by statistical methods from text corpus

• Multiple choice test item (Mitkov et a., 2003)test items by rule-bases approachdistractors’ estimation by using WordNet

• Multiple choice test item (Hoshino et al., 2005)machine learning + training set to determine blanksapplication domain: news article

• Factual question test items (Rus et al., 2007)applying patterns, templates and special markup language

• Limited-choice & completion items (Gütl, 2008)automatic summarization, part of speech tagging

No flexible support of different test items and learning settings

Objectives & Requirements

© 2010 - Christian Gütl19/04/23 6

• Goal for Automatic Question Creator is• … support (semi-)automatic test item creation• … use stand alone and integrate in LMS• … flexible design to adjust tool to specific setting

• High-level requirements• Support various file formats, local & Internet• Domain knowledge and document structure

independent• Extraction of concepts from learning content• Creation of test items and reference answers

based on identified concepts• Support of various test items types:

open ended, single/multiple choice, completion exercise

Conceptual Architecture

© 2010 - Christian Gütl19/04/23 7

• File format conversationWord, OD, PDF HTML

• Text cleaningHTML clean up

• Internal RepresentationXML structure (HTML XML)

• Analysis & Weightingtokenizing, POS, stemming, named entities, co-occurrence, semantic & lexical relations

• Concept Selectionnoun phrases concept

• Preprocessingrelevant concepts per chaptermost appropriate content

distractors & reference answer

• Item Creation & Export

Test Item Creation

© 2010 - Christian Gütl19/04/23 8

• Completion Exercises (Fill-in blank)• Find relevant sentence for concept (weighting)

and semantically important additional text • Remove concept (phrases)

• Multiple Choice• Find relevant sentence for concept• Find relevant distractors for concept

• Single Choice (true/false questions)• Find antonyms for single words of concepts• Create sentences with wrong meaning

• Open Ended Questions• Using pattern and rules for question creation• Creating reference answer from content by finding

most appropriate text block (text tiling)

Prototype 1/4

© 2010 - Christian Gütl19/04/23 9

Prototype 2/4

© 2010 - Christian Gütl19/04/23 10

Prototype 3/4

© 2010 - Christian Gütl19/04/23 11

Prototype 4/4

© 2010 - Christian Gütl19/04/23 12

User Study - Setup

© 2010 - Christian Gütl19/04/23 13

• Research Question:Performance of Tool (concepts, test items)

• First qualitative study• 5 subjects (1 college & 4 university students)

• Evaluation Procedure1. Read learning content carefully

“Project Management” (MIT courseware)2. Identify 5 most expressive concepts for each of

the 3 chapters3. Construct per chapter 2 test item for each type4. Rate quality of given concepts for each chapter5. Rate quality (multiple dimensions) of given test

items for each chapter

User Study - Quality of Concepts

© 2010 - Christian Gütl19/04/23 14

• For each chapter 10 highest ranked concepts

• Appropriateness rating (0 …. 100 %)

• Ratings above 50 % counted as relevant

User Study - Quality of Test Items

© 2010 - Christian Gütl19/04/23 15

• For each chapter 2 automatic and 2 manual crated test items provides for assessment

• Assessment according to following criteria:• Pertinence: relevance of question in given context• Level: Level of difficulty of a test item• Distractor: Quality of listed distractors (multiple choice)• Answer: quality of reference answer (open ended questions)

Conclusion & Future Work

© 2010 - Christian Gütl19/04/23 16

• Surprisingly good quality of created test items• Subject domain indepentend• High time complexity, processing time

• Further improvements• Multilinguality (German, Spanish, …) • Improvements of methods to identify concepts and

create questions (quality and performance)• Web service & integration in assessment framework

• Further Evaluation and practical use of the tool• Detailed evaluation with 30 subjects• Usage and evaluation for test item creation support on

medical content (Medical University Graz)• Usage and evaluation for self-directed learning

activities and complex learning experiences (part of EC funded project ALICE)

ReferencesBransford, J.D., Brown, A.L., & Cocking; R.R. (Eds.) (2000) “How People Learn: Brain, Mind, Experience, and

School. Expanded Edition. Washington DC: National Academies Press.

Coniam, D. (1997). A Preliminary Inquiry Into Using Corpus Word Frequency Data in the Automatic Generation of English Language Cloze Tests. Calico Journal, Vol 14, No. 2–4, pp 15-34.

Gütl, C. (2008). Automatic Limited-Choice and Completion Test Creation, Assessment and Feedback in modern Learning Processes. Paper read at LRN Conference 2008, Guatemala, February 12th – 16th.

Gütl, C. (2010) “The Support of Virtual 3D Worlds for enhancing Collaboration in Learning Settings. In Francesca Pozzi and Donatella Persico (Eds.) Techniques for Fostering Collaboration in Online Learning Communities: Theoretical and Practical Perspectives, IGI Global, 2010 (accepted)

Hoshino, A. and Nakagawa, H. (2005). A real-time multiple-choice question generation for language testing: a preliminary study”, Proceedings of the second workshop on Building Educational Applications Using NLP, pp 17-20.

Mitkov, R. and Ha, A. L. (2003). Computer-Aided Generation of Multiple-Choice Tests. Proceedings of the HLT-NAACL 2003 workshop on Building educational applications using natural language processing, pp 17-22.

Rus, V., Cai, Z. and Graesser, A. C. (2007). Experiments on Generating Questions About Facts. Computational Linguistics and Intelligent Text Processing, 8th International Conference, CICLing 2007, Mexiko City, Mexiko, February 18-24, 2007, Proceedings, pp 444–455.

© 2010 - Christian Gütl19/04/23 17

© 2010 - Christian Gütl19/04/23 18

Questions & Contact Information

Thank you for your Attention!Questions are welcome!

Further Information:

Christian Gütl

http://www.iicm.edu/guetl

[email protected]

Weighting Algorithm

© 2010 - Christian Gütl19/04/23 19

Test Item Example 1/4

© 2010 - Christian Gütl19/04/23 20

Fill in the blank 2-2question: Hence, management functions can be organized into a hierarchical structure

designed to improve operational efficiency. ______________ are performed by all managers, regardless of enterprise, activity or hierarchical levels. Finally, the development of a management philosophy results in helping the manager to establish relationships between human and material resources.

correct answer The basic management functions

criteria Pertinence Level Concept

value


© 2010 - Christian Gütl19/04/23 21

Multiple Choice 2-2question: Hence, management functions can be organized into a hierarchical

structure designed to improve operational efficiency. ______________ are performed by all managers, regardless of enterprise, activity or hierarchical levels. Finally, the development of a management philosophy results in helping the manager to establish relationships between human and material resources.

distractors The basic documentation functions

The basic serviceablenes functions

The basic chores functions

The basic management functions

correct answer The basic management functionscriteria Pertinence Level Distractors

value


© 2010 - Christian Gütl19/04/23 22

Single Choice 2-1question: The optimization or suboptimization is often achieved by the use of operations

research techniques, such as linear programming, quadratic programming, graph theory, queuing theory and Michael porter simulation.

correct answer falsecriteria Pertinence Levelvalue


© 2010 - Christian Gütl19/04/23 23

Open Ended 2-3question: Which operation research techniques are used for optimization?

Answer region A topic of major interest in management science is the maximization of profit, or in the absence of a workable model for the operation of the entire system, the suboptimization of the operations of its components. The optimization or suboptimization is often achieved by the use of operations research techniques, such as linear programming, quadratic programming, graph theory, queuing theory and Monte Carlo simulation

criteria Pertinence Level Answervalue

ECEL 2010 - Presentation Enhanced Approach of Automatic Creation of Test Items to foster Modern...

Documents

Transcript of ECEL 2010 - Presentation Enhanced Approach of Automatic Creation of Test Items to foster Modern...