Towards automatic generation of

e-assessment using semantic web technologies * †

1

*Presentation based on Cubric,M.&Tosic,M.(2010/2011) ‘Towards automatic generation of e- assessment using semantic web technologies; International Conference on CAA, University of Southampton, July 2010; Also to appear in International Journal of E-assessment 2011†University of Hertfordshire Business School research Seminar 2/11/2011

Marija Cubric, m.cubric@herts.ac.ukMilorad Tosic, mbtosic@acm.org

AbstractSemantic Web technologies have been increasingly used as a tool for generating, organizing and personalizing e-learning content. In this presentation we will discuss and demonstrate an innovative approach to automated generation of computer-assisted assessment (CAA) from Semantic Web–based domain ontologies. The primary application domain of this work is in the automated assessment, and in particular, the development of intelligent CAA systems and question banks, but the ideas can be further generalized in the context of ontology engineering and evaluation.

2

ContentTerminology ……….. Research Problem………... ……….. Related Work……….. Methodology ………..Our Contribution ……….. Implementation ………. Benefits and Limitations ……….. Future Work …..……….. Other research ……….. Q&A

3

E-learning…is the use of innovative technologies and learning models to transform the way individuals and organisations acquire new skills and access knowledge (Moeng, 2004)

4

.

… comprises all forms of electronically supported learning and teaching. The information and communication systems, whether networked learning or not, serve as specific media to implement the learning process (Tavangarian et al.,2004)

… a broad combination of processes, content, and infrastructure to use computers and networks to scale and/or improve one or more significant parts of a learning value chain, including management and delivery (Aldrich, 2004) Adrich, C. (2004) Simulations and the Future of Learning. San Francisco: Pfeiffer, p.240Moeng, B. (2004). IBM tackles learning in the workplace. IBM Management Development Solutions, Nov 8, 2004.Tavangarian D., Leypold M., Nölting K., Röser M.,(2004). Is e-learning the Solution for Individual Learning? Journal of e-learning

E-learning

5

.

Information Systems

Information Systems EducationEducation

This research

6

.

EducationEducationInformation

SystemsInformation

Systems

Information TechnologyInformation Technology

AssessmentAssessment

Objective tests and CAA

7

Objective tests require a user to choose or provide a response to a question whose correct answer is predetermined.

Such a question might require a student to :- select a solution from a set of choices (MCQ, true-false, matching)- identify an object or position (graphical hotspot) or- supply brief numeric or text responses (text input).

Because the correct answers to objective test questions are pre-determined, they are well suited to the many forms of CAA (Computer-assisted assessment) or e-assessment (CAA Centre Resources)

MotivationStudents’ needs

– Prompt and frequent feedback (NSS)– Constant simulation, fun, new stuff (Bean, 2010)

Universities’ agendas– Increase opportunities for flexible learning (“anytime anywhere”)– Cost reduction (“more for less”)

Good teaching practice (Chickering and Gamson, 1987) – Encourages active learning– Gives prompt feedback– Emphasises time on task– Respects diverse talents and ways of learning

Teachers’ needs– Questions are difficult and time-consuming to write– Professional writers plan one hour or more per item (Van Hoozer

quoted in Collins, 2006)

8

Bean M (2010) JISC 2010 Opening keynote: The learning journey: From informal to formalChickering A.W. and Gamson Z.F (1987) "Seven principles for good practice in undergraduate education. The American Association for Higher Education Bulletin, March 1987Van Hoozer H, et al. The teaching process: theory and practice in nursing. Norwalk, CT: Appleton-Century-Crofts, 1987:279-280

Research Problem

9

Develop algorithms and tools for automated generation of objective tests, leveraging on increasing body of domain ontologies (“knowledge conceptualisations”) and other advances in the area of semantic web.

.

Information Systems

Information Systems EducationEducation

Information TechnologyInformation Technology

AssessmentAssessment

Semantic Web Primer

Semantic Web - “The Web of data with meaning in the sense that a computer program can learn enough about what the data means to process it” ("Weaving the Web" by Tim Berners-Lee, 1999)

Ontology - A formal specification of a conceptualization of a knowledge domain in terms of classes, instances, relations, properties and annotations relevant to modeling of the domain (Gruber, 1993)

W3C standards define specific formalisms for encoding ontologies, such as RDF (Resource Description Framework) and OWL (Ontology Web Language)

10

http://www.w3.org/

Gruber T. R. (1993), A Translation Approach to Portable Ontology Specifications. Knowledge Acquisition, 5(2):199-220

11

An Example Ontology from Business Domain (e-commerce)

12

E-commerce ontology (class view)

13

classes classes

properties

properties

instances instances

annotations annotations

Related Work- Ontologies in LearningOntological formalization of learning content, instructional processes and learning designs for the purpose of automated discovery, composition and presentation of knowledge

– Holohan & Pahl , 2003; Sicilia & Barriocanal, 2005; Knight, Gaševic & Richards,2006

Related data mining techniques for the discovery of ontologies from arbitrary text corpora

– Montoyo, et al. 2005Ontologies for fostering collaborative learning (Allert, Markkanen, Richter 2006)

– Collaborative concept mapping– Using ontologies to organize or annotate shared artifacts– Using existing ontologies to carry out an activity– Collaborative ontology development as part of an overarching task

14

Chung, Niemi & Bewley (2003): assessment authoring support system for “aiding assessment authors to populate the assessment ontology with values specific to the users’ purposes”

Holohan, et al, (2005): semi-automatic generation of simple learning objects such as slide shows and objective tests, in the context of an adaptive learning environment

Holohan, et al, (2006) & Zualkernan et al, (2009) dynamic problem generation, using domain-specific algorithms: the examples considered was from the domain of relational databases and software engineering and the resulting problems were database queries and questions related to the UML models of software artifacts

Mitkov, R. et al, (2006): NLP techniques such as shallow parsing, term extraction, sentence transformation and computation of semantic distance, aided by WordNet

Related Work- Automated Generation of E-assessment

15

Papasalouros A., Kotis K., Kanaris K. (2008): defined 11 ontology-based strategies for automatic generation of ‘distractors’ for Multiple Choice Questions (MCQ), from arbitrary knowledge domains.

The generation is based on the basic meta-ontology relations between a ‘class’ and ‘individual’, as well as between two individuals (binary ‘role’)

Related Work- Automated Generation of E-assessment (cont)

16

A(a) and not A(b) R(a,b) and not R(c,b) A<B and not D<B

Our ContributionFocus on ontology-based automatic generation of assessment of an arbitrary knowledge domain

Developed Protégé-based plugin (Tosic&Cubric,2009)–Extended and optimized strategies introduced by Papasalouros et al (2008)

In Cubric & Tosic (2010) :Added annotations to the meta-ontology used for question

generation

Added question templates for generating question text

Added semantic interpretation to the question templates in terms of learning theory - different types of questions can be generated based on different semantic interpretation

Most recent: : Developed open source prototype for evaluation purposes (www.opensemcq.org ) 17

@code.google.com

18

opensemcq.org

Design as a MethodologyDesign can be thought of as a mapping from function space - a functional requirement constituting a point in this multidimensional space - to attribute space, where an artifact satisfying the mapping constitutes a point in that space (Takeda, et al, 1990). Design then, is knowledge in the form of techniques and methods for performing this mapping – the know-how for implementing an artifact that satisfies a set of functional requirements. (DESRIST.org)

Design Science Research – “learning through building - artifact construction” and “using design as a research method or technique” (DESRIST.org) Subset of Design Research (research into or about design) whereas DSR is research using design as a research method or technique

19Takeda, H., Veerkamp, P., Tomiyama, T., Yoshikawam, H. (1990). "Modeling Design Processes." AI MagazineWinter: 37-48.

Information SystemsInformation Systems

Information TechnologyInformation Technology

Design Science Research Method

Design science research is most effective when its practitioners shift between pragmaticand critical realist perspectives, guided by a pragmatic assessment of progress in the design cycle( Bunge,1984)

20

Bunge, M. (1984). Philosophical Inputs and Outputs of Technology. History and Philosophy of Technology. G. Bugliarello and D. Donner. Urbana, IL, University of Illinois Press: 263-281.

Knowledge gained from construction

Philosophical assumptions of Design Science Research

(source: DESRIST.org)21

Protégé Plugin(“suggestions”)

Features• Variety of questions generated

based on different pluggable ontologies (19 test ontologies)

• Optimized existing 11 (Papasalouros et al, 2008) and added 4 new strategies for generating distracters

• Adjustable number of questions and types of strategies used.

• Adjustable difficulty level of the generated question set.

Architecture

22

MCQ Ontology (IMS, 2002; CAA,2010)

23

n m*

*

CAA Centre Resources (2002) http://www.caacentre.ac.uk/resources/ (accessed 12/5/2010)IMS Global Learning Consortium, Inc. IMS Question and Test Interoperability Version 2.1 Public Draft

Generation ProcessInput : Any domain ontologyOutput: MCQ ontology populated with instances of MultipleChoiceQuestion class

24

(any) domain ontology

target (MCQ) ontology

question templatequestion template

Stems, question templates & semantic interpretation

25

Stems using ontology annotations

Stems generated from pre-defined parameterized question templates

Templates based on different levels of Bloom’s (1956) taxonomy of cognitive domains

Strategies for generating distracters

26

Based on ‘text similarity measures’ - the more similar distracters are, the more difficult question becomes (Mitkov et al, 2008)

Mitkov, R. et al, 2006. A computer-aided environment for generating multiple-choice test items. Natural Language Engineering, Vol. 12, No. 2, pp. 177-194.

Which of the following definitions describes the concept Quantitative value?  a) This is a conceptual entity that holds together all aspects of the n-ary relation hasWarrantyPromise. A ________ is an entity representing the duration and scope of services that will be provided to a customer free of charge in case of a defect or malfunction of theProductOrService. b) An instance of this class is an actual _________ for a quantitative property of a product. This instance is usually characterized by a minimalvalue, a maximal value, and a unit of measurement. c) A _____ is a numerical interval that represents the range of a certain quantitative Product Or Service Property in terms of the lower and upper bounds for a particular Product Or Service . It isto be interpreted in combination with the respective unit of measurement . d) An instance of this class is an actual _______for a quantitative property of a product. This instance is usually characterized by a minimal value, a maximal value, and a unit of measurement.Examples: The intervals between 10.0 and 25.4 kilogramms" or "10.2 and 15.5 milimeters".e) A __________ is a predefined value for a product characteristic. Examples: the color "green" or the power cord plug type "US"; the garment sizes "S", "M", "L", and "XL”.

Which of the following definitions describe the concept <A> : (x,y,…)

27

Class A

x

Annotation

Class B

yHigh Similarity

Annotation Annotation

Annotation

✔ ✖

Strategies for distractors: Most similar annotation

Which one of the following examples demonstrates the concept <A>:(a,b,c,...)

28

aa

Class A

✔

instanceinstance

instanceinstance

bb

Class B

✖

instanceinstance

instanceinstance

cc

instanceinstance

instanceinstance

Class C

✖

Which one of the following examples demonstrates the concept Payment method credit card? a) Check in advance b) Direct debit c) Pay Swarm d) JCB e) Google Checkout

Strategies for distractors: Instance of a Parent Class, Instance of a Sibling Class …

Read the paragraph <x> and decide which one of the following concepts generalize the concept defined by <x>: (A,B,C,D,E,F,…)

29

Read the paragraph and decide which one of the following concepts generalize the concept defined by it:  "A _____ is a conceptual entity that specifies the additional costs asked for settling the payment after accepting a given Offering using a particular Payment Method . A ____ is characterized by (1) a monetary amount per order specified as a literal value of type float in combination with a Currency , (2) the payment method , and (3) a whether this charge includes local sales taxes , namely VAT”

a) Product or serviceb) Actual product or service instance c) Business functiond) Price specificatione) Location of sales or service provisioning

Strategies for distractors: Sibling concept, Child Concept, Parent’s sibling, Grandparent…

Class B

Class A

x

✔

✖Class

E

Class F ✖

✖

Class C

Class D

✖

✖

Read the paragraph and decide which one of the following concepts it defines:"The ________ represents types of services that will be provided free of chargeby the vendor or manufacturer in the case of a defect (e .g . labor and parts ,just parts ), as part of the warranty included in an Offering . The actualservices may be provided by the Business Entity making the offering , bythe manufacturer of the product , or by a third party . Examples: Parts andLabor , Parts" <d>a) Quantitative valueb) Delivery methodc) Payment methodd) Warranty scopee) Warranty promise

Read the paragraph <x> and decide which one of the following concepts it defines: (A,B,…)

30

Class A

x

Annotation

Class B

Annotation High Similarity

Annotation Annotation

Annotation

✔ ✖

Strategies for distractors: Most similar concept

Which one of the following response pairs relates in the same way as <a> and <b> in the relation <R>? : (c,d),(c,b),(a,d),…

31

Which one of the following response pairs relates in the same way as Sunday and Saturday in the relation has previous?

a)Wednesday and Tuesday b) Friday and Saturdayc) Wednesday and Sundayd) Sunday and Mondaye) Monday and Wednesday

aa

bb

R

cc

dd

R ✔

✖✖

Strategies for distractors: Non-existent relation with the same domain, Non-existent relation with the same co-domain

Analyze the following text and decide which one of the following words is acorrect replacement for the blank space in the text: "This subproperty specifies that the upper and lower limit of the given ________ are identical and have the respective value . It is a shortcut for suchcases where a quantitative property is (at least practically ) a single pointvalue and not an interval ."a) Warranty promiseb) Quantitative value integerc) Quantitative valued) Quantitative value floate) Qualitative value

Analyze the text <x> and decide which one of the following words is a correct replacement for the blank space in <x>: (B,C,…)

32

Class B

annotation

annotation

annotation

Class CClass C

annotation

annotation

annotation

High

Similarity

Class A

annotation

x

annotation

Describes

✔✖

Strategies for distractors: Most Similar Annotation

Implementation: opensemcq: generate test

33

Implementation: opensemcq: view test

34

Implementation: opensemcq: evaluate test

35

BenefitsAdding annotations

– Four more question types and three more strategies Adding questions templates

– Avoiding the NLP complexity problemAdding semantic interpretation

– Questions ordered according to increased level of educational objectives (Bloom, 1956)

– Enables adaptive test creation (Lilley & Barker, 2002)Decoupling semantic interpretation from question generation

– Enables change of the underlying learning theory – Kolb’s experiential learning (Kolb, 1984) as a basis for defining

question templates (Barker, 2008) Useful ‘seed’ for further question enhancements

36

Bloom, B. S., Krathwohl, D. R. (1956). Taxonomy of educational objectives. Handbook 1. Cognitive domain. New York: Addison-Wesley.Lilley, M., & Barker, T. (2002). The development and evaluation of a computer-adaptive testing application for English language. 6th Computer assisted assessment conference, July 2002, Loughborough.Kolb, D. (1984) Experiential Learning. Englewood Cliffs. N.J.: Prentice Hall

Issues and LimitationsLack of annotated (test) ontologies

Lack of questions templates for assessing higher order skills (e.g. synthesis and evaluation)

Generation of distracters based on text similarity bear no necessary semantically structured relation to the subject matter domain

– Should be extended with meta-ontological relations, such as: subclass, superclass, ‘friend’ class, ‘related’ classes

Scope of assessment limited to testing the knowledge described in the ontology and the quality of the ontology.

37

Current WorkInvitation for evaluation to be sent to authors of ontologies used for testing

Further invitations to be sent to University of Hertfordshire academics

Improvements to be implemented based on the feedback from the evaluation.

Want to participate? – Go to www.opensemcq.org – Select a Test to evaluate– Click on Evaluate !

Want to develop a new ontology?– Talk to us!

38

Future WorkIntegrate text similarity with more advanced ontological approaches to generation of distracters

– “The distance between concepts in the hierarchy can be used to generate challenging answers covering closely related false answers (distracters)” (Pahl & Holohan, 2009)

Extend and enrich the template base

Add other ontology components, such as rules, axioms, restrictions, events etc. to the meta ontology used for question generations

– Explore Horridge’s et al. (2009) ideas on ontology justifications, for generating question feedback

Extend the empirical dataset to include some larger ‘real-life’ ontologies

Generalize the MCQ ontology to include other types of objective tests e.g. Multiple Response Questions

Further collaborative work planned with the Semantic Web groups from University of Southampton and Manchester; CAA group group University of Dublin , Agean and CMU, and Ontology engineering group from Stanford …

39