WP5: Validation

Anne De RoeckDiane Evans

The Open University, UK

WP5

• Who?• Objectives• Challenges & Risks• Scenario Based Evaluation• Why now?• Progress• Plan

Who?

• All Partners• Open University (WP5 Lead)

– Distance: 200,000 students – 90% on-line– Research and Teaching

• OU Centre for Research in Computing– CRC = Computing + KMi + IET UserLab– NLP: Anne De Roeck; Donia Scott, …– E-learning: Josie Taylor; Patrick McAndrew;

Diane Evans, …

WP5 Objectives

• Aim:– to validate if eLearning systems (LMS) are

improved by adding new functionalities based on language technology tools and ontology-based (multilingual) retrieval.

• Objectives – to assess the extent to which the integration of

these new functionalities affects the effectiveness of the eLearning process.

Challenges

• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development

Challenges

• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development

Challenges

• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development

• Challenge 1: Methodology:– Scenario based validation– Functionality & user context centred– User questionnaires & interviews

Challenges

• Focus:– Language Technology Tools– Ontology retrieval– Improve effectiveness!

• Challenge 2: Search & Retrieval Scenarios– Embed functionality (teachers & learners?)– Establish base-line– Refine scenarios

Challenges

• Focus:– Multi-lingual experience

• Challenge 3: Multilingual Scenarios– Multiple communities– Multi-lingual scenario– Chart the territory!

Risks

• Content!• Assessment??• Performance of Components?• Interface vs Functionality• Cross-cultural issues

Risks

• Content!• Assessment?• Performance of Components?• Interface vs Functionality• Cross-cultural issues

• Technical deliverable vs user requirements

Methodology

• User Scenarios– ‘a story focused on a user or group of users,

which provides information on the nature of the users, the goals they wish to achieve and the context in which the activities will take place’.

Methodology

• User Scenarios– Step beyond Software Development Use Cases– Qualitative evaluation of functionalities …– …in context of user roles– Accessible to all stakeholders – Developed from high level descriptions into

scenario instantiations during implementation– Used in learning applications (eg mobile

learning) & Software Development

Why now?

• Functionalities & System development• LT4eL = New technology• Co-evolve system & validation

– Process of scenario instantiation:– Explore/refine ideas on possible usage– Clarify user requirements– Establish common ground between pedagogists,

developers, NLPers, users– Concurrent with design & development

Progress

• Aim:– Prepare & support user scenario validation– Clarify objectives– Mitigate risk

• Background literature search and survey: – Keyword extraction & validation methods– Ontologies and deployment in search– Scenario-based validation

Progress

• Contribution WP1:– Identifying and supplying English Content within

the domain of e-learning in educational establishments.

• Contribution WP2/3:– running early version of keyword extractor with

English content and feeding back to WP2 leader. – liasing with Malta on English content– identifying qualitative validation tests for Keyword

Extractor application.

Progress

• Start WP5:– Monitor content (ongoing: high risk!)– Draft validation plan for ILIAS + new

functionality– Delivered high level user scenario (starting

point of instantiation)– Preliminary Questionnaire (establish qualitative

base-line)

Progress/Plan

• Questionnaire:– slide presentation supporting 'focus group' – questionnaire (in Word format) covering search

motivation and behaviours – 81 individual student responses from 6

partners

Progress/Plan

• Questionnaire Findings on:– Reasons to search in non Native language– Difficulties encountered when searching– Selecting search criteria – Handling search results– Other issues

Plan

• Process Questionnaire• Challenge2

– Refine search & retrieval scenarios (Workshop – March)

– Integration (WP4)

• Challenge 3– Multi-lingual dimension to functionalities– Uncharted territory !

Plan

0. Preliminary Questionnaire (M12) - Partners

1. Pilot Validation (M16-18) – OU2. Main Validation (M19-23) – Partners (ILIAS Community part)3. Final Validation (M26-28)

• Reports M24, M30