CALICO 2014

Seven ways to make CALL more intelligent. Towards the effective integration of NLP techniques

Piet Desmet

in collaboration with

Frederik Cornillie, Ruben Lagatie, Sien Moens, Maribel Montero Perez, Hans Paulussen & Serge Verlinde

CALICO 2014

0. Introduction

0.1. Terminology

Parser-based CALL (Holland et al. 1993) NLP-enhanced CALL (Nerbonne 2005) Intelligent CALL or iCALL “ICALL – Intelligent CALL – is a field within CALL which applies concepts, techniques, algorithms and technologies from AI to CALL (…). Most relevant to CALL is research in four branches of AI: (1) natural language processing, (2) user modelling, (3) expert systems and (4) intelligent tutoring systems”. (Schulze 2010: 65)

-> progressively larger scope

CALICO 2014

0.2. Challenges and opportunities

Most of the CALL environments do not use NLP -> use of NLP in classrooms is hardly mainstream practice “The development of systems using NLP technology is not on the agenda of most CALL experts,

and interdisciplinary research projects integrating computational linguists and foreign language teachers remain very rare” (Amaral & Meurers 2011: 6). cf. also report by the Dutch Language Union: Onderzoek taal- en spraaktechnologie en onderwijs (Van den Heuvel, T’Sas & Verberne 2012)

Technological concerns Pedagogical concerns

CALICO 2014

Technological concerns Can NLP account for the full complexity of natural human languages

“I am pessimistic about the possibility of ICALL” (Nyns 1989: 46)

+ in educational settings, no room for erroneous analyses -> need for nearly error-free applications But: too limited accuracy of NLP-tools -> risk of mislearning of L2

http://www.flickr.com/photos/maphutha/1303854829/

CALICO 2014

“It is therefore of the utmost importance to warn the users of the limitations of the tools. Ideally, inadequate outputs provided by the NLP tools should make the learners reflect even more on the language they are learning and on its structure. Therefore, even NLP tool errors should help the learners master the target language because they require more thinking on their part. Thus, NLP tools can be useful for CALL and usefully used in CALL. The present, imperfect state of technology should not be a hindrance, although there is obviously room for improvement. However, improvements are often triggered by remarks and studies on how tools effectively work in context. Thus, if one waits to see improvements before using NLP tools in CALL software, one might have to wait for a long time, while using the tools in their present state will encourage improvements to be made according to the needs of the language learners” (Vandeventer 2003 – Linguistik online 17 – Learning and Teaching (in) Computational Linguistics)

-> users are intelligent and undemanding (BUT: for all language levels?) + ICALL stimulates reflective practice -> Perhaps the best is yet to come, but let’s choose not to wait…

CALICO 2014

The Penrose impossible stairs

Computerpower doubles every 2 years

Technological progress vs Pedagogical progress!

Pedagogical concerns

CALICO 2014

Pedagogical concerns Initial NLP-enhanced CALL is mainly form-focused <-> Also ‘focus on meaning’ & meaning-based activities in Foreign language learning and teaching (FLLT) ICALL allows for the use of authentic materials and skills-oriented activities (cf. communicative approach) BUT: still appropriate task design needed in ICALL “From a computational perspective, a well-defined task design with its clear set of relevant language constructions facilitates the restriction to a linguistic domain which is ‘manageable’ for a system’s natural language processing modules” (Schulze 2010: 79)

<-> In FLLT focus on authentic language tasks with fully open & unpredictable interaction in real life settings (cf. task-based language teaching) -> challenges for ICALL!

CALICO 2014

0.3. Towards a typology (a) Linguistic perspective: type of language involved

Written vs spoken, native vs learner language, etc.

(b) Technological perspective: NLP & AI-technologies involved Parsing, NER, topic detection, sentiment mining, text summarization, etc.

(c) Pedagogical perspective: type of learning activities involved Reception, production, interaction, mediation

(d) CALL perspective: type of technology-based learning or teaching activities

CALICO 2014

CALL perspective: type of technology-based learning or teaching activities From receptive to productive activities with focus on written language input & output

1. Input provider

2. Reading companion

3. Exercise and test generator

4. Error detector, feedback generator and automatic scoring tool

5. Writing aid

6. Adaptive item sequencer

7. Resource generator

-> Conceptual outline + applications from academic R&D

CALICO 2014

1. Input provider

What? (Semi-)automated selection of comprehensible & authentic text material How? a. analysis of readability and formal complexity + syntactic and lexical text simplification/elaboration b. analysis of meaning or text categorization (e.g. subject categorization, topic detection, text categorization, etc.)

Seven roles for ICALL applications

CALICO 2014

a. Text retrieval on the basis of readability evaluation of input REAP-project (Carnegie Mellon) http://reap.cs.cmu.edu “Reader-Specific Lexical Practice for Improved Reading Comprehension”: support learners in searching for texts that are well-suited for providing vocabulary and reading practice. + analysis of formal complexity of input SATO-project (François Daoust) http://www.ling.uqam.ca/sato Système d’analyse de texte par ordinateur + SATO-Calibrage Automated formal analysis of a corpus (existing or personal)

CALICO 2014

b. analysis of meaning & text categorization

© Sien Moens (2009)

CALICO 2014

2. Reading companion

What? helping learners understand foreign-language input How? annotation layers, both formal and semantic GLOSSER-project (John Nerbonne) http://www.let.rug.nl/glosser/Glosser lemmatization of the inflected forms dictionary entry (cf. Van Dale) examples of the word from corpora iRead+ project (ITEC) Lemmatization & POS-tagging Named entity recognition (persons, organisations, locations)

CALICO 2014

GLOSSER-project

Lemmatization of inflected forms

Dictionary entry (Van Dale)

Examples from corpora

CALICO 2014

CALICO 2014

Vb Anderlecht Taalk. /3

CALICO 2014

CALICO 2014

3. Exercise and test generator

What? (semi-)automated generation of exercise and test items How? based on the analysis of L1 text materials and/or on the analysis of learner errors - morpho-syntactical activities: iRead+ project (ITEC) VIEW-project (Detmar Meurers) http://sifnos.sfs.uni-tuebingen.de/VIEW Visual Input Enhancement of the Web

- lexical activities: Alfalex-project (Serge Verlinde) http://ilt.kuleuven.be/publicaties/tools.php - semantic activities: cf. Sien Moens - semantic frame labeling

CALICO 2014

iRead+ Exercise generator: Three step model

– Explore

1. Application highlights target items

2. Learner recognizes target items and clicks on target items

– Practice

• Fill gaps or multiple choice activities

• Feedback

– Play

• Target items, drill & practice

• Time constraint

CALICO 2014

CALICO 2014

Alfalex (KU Leuven – Serge Verlinde)

CALICO 2014

CALICO 2014

Semantic frame labeling © Sien Moens (2009)

CALICO 2014

4. Error detector, feedback generator and automatic scoring tool

What? Automated analysis of learner output and generation of feedback Not limited to closed exercises (MC, fill-in-the-blank, etc.) but also (semi-)open practice tasks (translation, correction, rephrasing, etc.) How? 1) Approximate (or fuzzy) string matching (ASM) -> anticipate all potential well-formed and ill-formed learner responses

(with inclusion of regular expressions)

2) Parser-based (malrules, constraint relaxation, etc.) and/or statistical and machine learning methods

CALICO 2014

0

5

10

15

20

197

8

197

9

198

0

198

1

198

2

198

3

198

4

198

5

198

6

198

7

198

8

198

9

199

0

199

1

199

2

199

3

199

4

199

5

199

6

199

7

199

8

199

9

200

0

200

1

200

2

200

3

200

4

200

5

200

6

200

7

200

8

200

9

201

0

201

1

201

2

Nu

mb

er o

f p

ub

lica

tio

ns

Year

Parser-based ICALL Statistical ICALL

Looking at the number of publications shown in this figure, parser-based CALL has dominated research up until 2005, but there seems to have been a shift in research interest and statistical error detection systems have since then taken the upper hand.

© PhD Ruben Lagatie

CALICO 2014

BUT: fully open-ended learner output hard to manage -> constrain learner output (reduce possible answers) by reducing the search space of the processing tools How? ° task: structured towards specific type of output - require the learner to use certain language material e-tutor-project (Trude Heift) http://www.e-tutor.sfu.ca translation Tagarela-project (Detmar Meurers) http://sifnos.sfs.uni-tuebingen.de/tagarela answer should include certain words - the task design offers clues towards a limited set of possible answers Dialog Dungeon (ITEC) gamified written dialog tasks ° correction: focus on specific set of problems (e.g. past tenses, subject-verb agreement, etc.)

CALICO 2014

E-tutor project

CALICO 2014

Tagarela-project

CALICO 2014

‘social’ log-in

Dialog Dungeon gamified written dialogue tasks

@Frederik Cornillie

semi-open written exercise

learning support: link to responses of peers

CALICO 2014

learning support: responses of peers

CALICO 2014

focused exercise on tenses

CALICO 2014

‘typical’ error

CALICO 2014

contrastive analysis learner reponse – closest correct match (based on approximate string matching, POS tagging, lemmatization)

CALICO 2014

supportive prompt with metalinguistic hints (based on POS tagging)

CALICO 2014

5. Writing aid

What? Support the learner in writing a functional, well-formed tekst How? No automated correction, but suggestions to help the learner improve himself his output Post-writing checks (or on-the-fly prompts) Assistance on lower-order skills (spelling) but also on lexical (including MWE), lexico-grammatical and grammatical skills Interactive Language Toolbox-project (Serge Verlinde) https://ilt.kuleuven.be/inlato Bon patron, SpellCheckPlus & SpanishChecker (Nadaclair Language Technologies) http://bonpatron.com; http://spellcheckplus.com; http://spanishchecker.com + semantic and pragmatic analysis Glosser-project (Univ. of Sydney) http://www.glosserproject.org/en/

CALICO 2014

Interactive Language Toolbox

CALICO 2014

CALICO 2014

CALICO 2014

Glosser project (Sydney)

CALICO 2014

6. Adaptive item sequencer

(a) Adaptivity: to what? - difficulty level of the tasks - learner profile (prior knowledge, motivation, cognitive load, interests & preferences) - context (time and place, device, etc.) (b) What to adapt? - adaptive form representation (form in which content is presented to the learner, e.g. dynamically

generated hypermedia pages) - adaptive content representation (e.g. with or without learner support) - adaptive curriculum sequencing (e.g. selection of items in function of difficulty level, learner

profile, etc.)

(c) How to implement adaptivity? - full program control (via reasoning component ‘if.. then…’ rules) - full learner control - shared control

Wauters, K., Desmet, P., Van Den Noortgate, W. (2010). Adaptive Item-Based Learning Environments Based on the Item Response Theory: Possibilities and Challenges. Journal of Computer Assisted Learning, 26 (6), 549-562.

CALICO 2014

What about AI? (a) Adaptivity: to what? - linguistic complexity -> language agent (expert system) - item difficulty -> IRT - learner profile (prior knowledge, motivation, cognitive load, interests &

preferences) -> student agent (b) What to adapt? - adaptive curriculum sequencing (e.g. selection of items in function of difficulty

level, learner profile, etc.) -> tutor agent

(c) How to implement adaptivity? full or shared control -> tutor agent

Beuls, Katrien (2013) Processing, learning and tutoring of Spanish verb morphology. Brussels: VUB. (PhD)

CALICO 2014

7. Resource generator

What? creating reference materials: concordancing on bilingual corpora REBECA-project (ITEC) Ressources électroniques bilingues extraites de corpus alignés & more advanced search engines DPC-project (ITEC) http://www.kuleuven-kulak.be/DPC/ Dutch Parallel Corpus

corpus-enriched learner dictionaries & grammars BLF-project (Serge Verlinde) http://ilt.kuleuven.be/blf/ Base Lexicale du Français

How? Annotation & exploitation of corpora

For whom? (Intermediate or) advanced learners with well developed linguistic awareness

CALICO 2014

REBECA-project

CALICO 2014

DPC-project

CALICO 2014 16 June 2012 45

Example: “en réalité” French-Dutch

CALICO 2014 16 June 2012 46

Example: “en réalité” French - Dutch

CALICO 2014

Extension: video search

Exploitation of XML-based captions (transcriptions and translations) to render video selection more versatile and attractive: lemmas are time tags indexes which form reference points to video selections

CALICO 2014

à 00:00:09.6800

a 00:00:24.8900|00:00:28.8400|00:00:59.9000|00:01:12.7000

alors 00:00:24.8900

assez 00:00:18.0300|00:00:31.9000

au 00:00:38.3000

aussi 00:00:38.3000

aux 00:00:09.6800|00:01:05.8100

bas 00:00:35.6500

beaucoup 00:00:24.8900|00:00:55.0000|00:00:59.9000

bonnes 00:00:31.9000

calais 00:00:51.1200

ce 00:01:05.8100

celle 00:00:42.0800

c’est 00:00:09.6800|00:00:24.8900

cette 00:00:06.6400|00:01:05.8100|00:01:12.7000

chefs 00:01:16.0000

chercher 00:00:18.0300|00:00:31.9000

chou 00:00:04.0000

choux 00:01:05.8100

club 00:01:16.0000

comme 00:00:06.6400

comment 00:00:18.0300

composer 00:00:42.0800

cuisine 00:00:04.0000|00:00:14.1800|00:01:12.7000

curiosité 00:00:38.3000

dans 00:00:18.0300|00:00:18.0300|00:00:24.8900|00:00:28.8400|

00:00:31.9000|00:00:35.6500|00:00:47.2800|00:01:16.0000|00:01:19.9500

Viewing video extracts based on lexical

search in transcription and/or translation

(challenge: search on semantically related

words)

CALICO 2014

Conclusion… Digital (learning) is the new normal a must trivial mainstream It’s not about technology. It’s about usage & added value! Usage: ICALL should enter our classrooms + be integrated into applications Added value: ICALL is a (potentially) powerful means to realise the main objective: improve learning -> qualified optimism about NLP-enhanced CALL