LAK14 Doctoral Consortium

Learning Analytics for Scaffolding Academic Writing

through Automatic Identification of Meta-discourse

Duygu Simsek

Doctoral Consortium, 4th Learning Analytics and Knowledge Conference, Indianapolis, USA 25th March, 2014

people.kmi.open.ac.uk/simsek

[email protected]

simsekduygu_Supervisors: Prof. Simon Buckingham Shum, Dr. Rebecca Ferguson, & Dr. Anna De Liddo Dr. Ágnes Sándor, Xerox Research Centre Europe

2

Research Aim

To investigate

whether computational techniques can automatically identify the attributes of good academic writing in as correlated with grades of the essay and as identified in the literature

if this proves possible, how best to feed back actionable analytics to support students and educators

whether this feedback has any demonstrable benefits

25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium

3

Where this research sits?

ACADEMIC WRITING

LEARNING ANALYTICS

COMPUTATIONALTEXT ANALYSIS

Rhetorical Parsers

DiscourseCentric

Learning Analytics

Meta-discoursein Student writing


4

Where this research sits?- Academic Writing

ACADEMIC WRITING

LEARNING ANALYTICS


Rhetorical Parsers

DiscourseCentric

Learning Analytics


Key aim of academic writing is to convince

readers about the validity of the claims and arguments put forward through an effective narrative.


5

Where this research sits?-Meta-discourse

ACADEMIC WRITING

LEARNING ANALYTICS


Rhetorical Parsers

DiscourseCentric

Learning Analytics


This effective narrative is signalled through meta-discourse!


6

Meta-discourse

Meta-discourse refers to the features of text that convey the author’s intended meaning and intention. It provides cues to the reader which explicitly express a viewpoint, argument and claim, and signals the writer's stance.


Fig. 1 Meta-discourse that convey summary statements

Cu

es t

o Summary

sta

tem

en

ts

7

Examples of meta-discourse cues thatsignal academic/analytical rhetorical moves

BACKGROUND KNOWLEDGE:

Recent studies indicate …

the previously proposed …

… is universally accepted

NOVELTY:

New insights provide direct evidence……suggest a new approach…Results define a novel role ...

OPEN QUESTION:

Little is known …

… role … has been elusive

Current data is insufficient…

TENDENCY:

... emerging as a promising approach

Our understanding ... has grown exponentially ...

Growing recognition of the

importance ...

CONTRASTING IDEAS:

In contrast with previous hypotheses ...

... inconsistent with past findings ...

SIGNIFICANCE:

studies ... have provided important advances

... is crucial for ... understanding

valuable information ... from

SURPRISE:

We have recently observed ... surprisingly

We have identified ... unusual

The recent discovery ... suggests intriguing roles

SUMMARISING:

The goal of this study ...

Here, we show ...

Our results ... indicate


8

Where this research sits?- Meta-discourse

ACADEMIC WRITING

LEARNING ANALYTICS


Rhetorical Parsers

DiscourseCentric

Learning Analytics


In order to assess students’ writing therefore, educators will be examining students’ use of meta-discourse which make their students’ thinking visible.

However, students find it challenging to learn to write in an academically sound way.

They need to learn how to make their thinking visible by recognising and deploying meta-discourse.


9

Where this research sits?- Computational Text Analysis

ACADEMIC WRITING

LEARNING ANALYTICS


Rhetorical Parsers

(XIP)

DiscourseCentric

Learning Analytics


Meta-discourse cues are automatically identifiable.

This PhD investigates whether it is possible to provide automatic meta-discourse analysis of student writing through the use of a particular rhetorical parser, XIP.


10

Example of a rhetorical parser: Incremental Parser (XIP)

Natural Language Processing (NLP) product which includes a rhetorical parser detecting meta-discourse in academic texts.

XIP extracts salient sentences based on their rhetorical functions: Background Knowledge Summarising Tendency Novelty Significance Surprise Open Question Contrasting Ideas


11

Student Writing Analysed by XIP


CONTRAST

SUMMARY

12

Rhetorical functions classified by XIP

BACKGROUND KNOWLEDGE:

Recent studies indicate …

the previously proposed …

… is universally accepted

NOVELTY:

New insights provide direct evidence……suggest a new approach…Results define a novel role ...

OPEN QUESTION:

Little is known …

… role … has been elusive

Current data is insufficient…

TENDENCY:

... emerging as a promising approach

Our understanding ... has grown exponentially ...

Growing recognition of the

importance ...

CONTRASTING IDEAS:

In contrast with previous hypotheses ...

... inconsistent with past findings ...

SIGNIFICANCE:

studies ... have provided important advances

... is crucial for ... understanding

valuable information ... from

SURPRISE:

We have recently observed ... surprisingly

We have identified ... unusual

The recent discovery ... suggests intriguing roles

SUMMARISING:

The goal of this study ...

Here, we show ...

Our results ... indicate


13

Fine for researchers or machines but it

is not learner/educator

friendly

XIP’s Output


14

Why XIP? – Key Features of Academic Writing?


RelevanceUnderstanding & KnowledgeStructure & OrganisationLinguistic AccuracyIllustrations ReferencingArgumentation

15

There is a mapping between good and strong features of academic writing and the XIP’s rhetorical functions.

Why XIP?


16

Where this research sits?- Learning Analytics

ACADEMIC WRITING

LEARNING ANALYTICS


Rhetorical Parsers

(XIP)

DiscourseCentric

Learning Analytics


XIP is a parser with potential, if it can be embedded in a more complete learning

analytics (LA) approach. It has

potential for formative feedback

to writing through LA.


17

Where this research sits?- Discourse-centric Learning Analytics

ACADEMIC WRITING

LEARNING ANALYTICS


Rhetorical Parsers

(XIP)

DiscourseCentric

Learning Analytics

(DCLA)


How should a DCLA approach be validated?


18

Main Research Question

To what degree can computational text analysis and visual analytics be used to

support the academic writing of students in higher education?


19

To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good academic writing within student writing, as judged by the grade, and by educators?


XIP

Evaluates Accuracy & Sufficiency

Any correlation between Grades &

XIP output?

XIP’s Highlights vs. Marker’s

RQ1

20



RQ1

XIP Highlighted Student Writing

Any correlation between the final grade of writing & XIP

findings?Pearson for Total number of salient sentences vs. GradeGeneralised Multiple Regression How strongly each rhetorical sentence type influences the final grade

Grades

21



RQ1

What is the overlap between XIP’s output and how tutors judge quality?

Tutor Highlighted Student WritingXIP Highlighted Student Writing

22

In what ways should XIP output be delivered to end users (students and educators)?


XIP



XIP output?


Output

RQ2

23

1st Year Pilot study

In what ways should XIP output be delivered to end users (students and educators)?RQ2


24

To what extent do educators value the results of XIP’s analysis of an individual student or cohort’s work when

the primary focus is on assessment?


XIP



XIP output?


Output

What educators think

RQ3

25


XIP



XIP output?


Output


To what extent do educators value the results of XIP’s analysis of an individual student or cohort’s work when

the primary focus is on assessment?RQ3

Semi-structured interviews

26

To what extent do students value the results of XIP’s analysis as formative feedback on their writing?


XIP



XIP output?


Output


What students think

RQ4

System for OU students.

27

1. For my quantitative study, do I have the right approach? Are there any alternative approaches? How could I make my study stronger?

2. What qualitative & quantitative methods could I use to evaluate the quality of the comparison between XIP & marker highlights?

3. Are there any available well-developed methodologies on assessing visualisations to elicit user reactions?

Feedback?


LAK14 Doctoral Consortium

Education

Transcript of LAK14 Doctoral Consortium