LAK15 Short Paper Talk

Correlations between

Automated Rhetorical Analysis and Tutors’ Grades

on Student Essays

http://oro.open.ac.uk/42042/

Speaker: Duygu Simsek

Co-Authors: Ágnes Sándor, Simon Buckingham Shum,

Rebecca Ferguson, Anna De Liddo,

Denise Whitelock

5th Learning Analytics and Knowledge Conference, Poughkeepsie, NY, USA 20th March, 2015

people.kmi.open.ac.uk/simsek

[email protected]

simsekduygu_

http://oro.open.ac.uk/42042/

To investigate

whether computational techniques can automatically identify the attributes of good academic writing in higher education as correlated with grades of the essay and as identified in the literature

if this proves possible, how best to feed back actionable analytics to support students and educators

whether this feedback has any demonstrable benefits

20/03/2015 | http://oro.open.ac.uk/42042/LAK'15 Poughkeepsie, NY, USA

Research Aim


Where is this research located?

ACADEMIC

WRITINGLEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

Key aim of academic writing is to convince

readers about the validity of the claims and

arguments put forward through an effective

narrative.


Where is this research located?-

Academic Writing

ACADEMIC

WRITINGLEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

One of the key requirements of academic writing in

higher education is that students must develop a critical mind, make their

thinking visible, and learn to construct sound arguments

in their discipline.



Student Writing

ACADEMIC

WRITINGLEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

This effective narrative is often articulated

through meta-discourse!



Meta-discourse

ACADEMIC

WRITINGLEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

Meta-discourse


Meta-discourse refers to the features of text that convey the author’s intended

meaning and intention. It provides linguistic cues to the reader which explicitly express a viewpoint, argument and claim, and signals the writer's stance.

Fig. 1 Meta-discourse that convey summary statements

Cu

es

to Summary

sta

tem

en

ts

When assessing their students’ writing therefore,

educators will, among other features, be looking for

scholarly meta-discourse as an indicator of argumentation.



Meta-discourse

ACADEMIC

WRITINGLEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

Powerful computational language technologies for extracting meta-discourseautomatically are becoming available.

But since they are originally developed in non-educational contexts, there is a need to validate them in a higher education framework.



Computational Text Analysis

ACADEMIC

WRITINGLEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

(XIP)

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

Natural Language Processing (NLP) product which includes a

rhetorical parser detecting meta-discourse in academic texts.

XIP detects rhetorically salient sentences in scholarly writing based

on the identification of meta-discourse and labels them based on

their rhetorical functions in seven categories.


The Incremental Parser (XIP)

SUMMARY: summarising the goals or results of the article

EMPHASIS: emphasising the importance of ideas

BACKGROUND: describing background knowledge necessary for

understanding the article’s contribution

CONTRAST: describing tensions, contrasts between ideas, models or

research directions

NOVELTY: conveying that an idea is new

TENDENCY: describing emerging research directions

OPEN QUESTION: describing problems that have not been solved


The Incremental Parser (XIP) labels:

Student Writing Analysed by XIP


CONTRAST

SUMMARY

BACKGROUND:

Recent studies indicate …

the previously proposed …

… is universally accepted


Rhetorical functions classified by XIP

NOVELTY:

New insights provide direct evidence…

…suggest a new approach…Results define a novel role ...

OPEN QUESTION:

Little is known …

… role … has been elusive

Current data is insufficient…

TENDENCY:

... emerging as a promising approach

Our understanding ... has grown exponentially ...CONTRAST:

In contrast with previous hypotheses ...

... inconsistent with past findings ...

EMPHASIS:

studies ... have provided important advances

... is crucial for ... understanding

valuable information ... from

SUMMARY:

The goal of this study ...

Here, we show ...

Our results ... indicate

Analysing written texts manually is a labour-intensive process.

Academic writing analytics research is burgeoning especially in the field of automated analysis of student writing.



Learning Analytics

ACADEMIC

WRITINGLEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

(XIP)

Discourse

Centric

Learning

Analytics

Meta-

discourse

in Student

writing

Learning analytics offer the potential for automated, timely, and formative feedback.

Computational rhetorical parsing technology barely deployed in educational contexts.



Discourse-centric Learning Analytics

ACADEMIC

WRITINGLEARNING

ANALYTICS

COMPUTATIONAL

TEXT ANALYSIS

Rhetorical

Parsers

(XIP)

Discourse

Centric

Learning

Analytics

(DCLA)

Meta-

discourse

in Student

writing


Aim of the Study

Explore the possibilities of applying the XIP rhetorical parser

in an educational tool.

Investigate to what extent XIP is accurate and sufficient for

detecting good academic writing in students’ essays given

the tutors’ grade as an evaluation measure.


Research Questions

Is there a correlation between the salient sentences

extracted by XIP and final grades?

What are the rhetorical markers out of the salient sentences

detected by XIP that are most promising as indicators of

good academic writing in students’ essay?

How accurate is the XIP output?


Dataset

1307 samples of student writing

Final year undergraduate education and arts module

The Open University, UK

EA300 Children’s Literature, students: study novels, picture books, poems produced for children.

read a selection of related critical material.

consider major themes, issues and debates in the field.

write 3000 word long essays at the end of the module.


Methodology

XIP Highlighted 1307 samples of Student Essays

1. XIP Analysis Results

quantified by calculating the total number of salient

sentences extracted by the parser and the numbers

of each rhetorical sentence type

2. Correlational Study

was conducted with these analysis results based on

the essays’ marks.

3. Generalised Multiple Regression

In order to understand the effect, if any, of each

rhetorical sentence type on essay marks.

Grades


Multiple Regression Analysis Results

Dependent Variable: Essay MarkIndependent Variable: Each XIP Category

The regression model proved to be highly significant (p≤0.001)

4.8% of the total variability in mark was explained by the independent variables (adjusted R2=0.048)


Multiple Regression Analysis Results

• for a one unit increase in the number of CONTRAST sentences within essays, the model predicts that the dependent variable, essay mark, will increase between 0.498 and 1.078 points calculated as B±2*Std.Error), holding all other independent variables fixed/constant.

Contrast (p≤0.001)

• for a one unit increase in the number of BACKGROUND sentences within essays, the model predicts that the dependent variable, essay mark, will increase between 1.075 and 3.431 points, holding all other independent variables fixed/constant.

Background (p≤0.001)


DISCUSSION- The XIP Performance

225 sentences were evaluated.

49 (22%) of them did not play the role of the scholarly

argumentation in the essay.

An important source of errors is related to the subject-

specific structures and terminology that the current version of XIP does not account for.


DISCUSSION- Relationship between the

tutors’ marking grid and the salient sentences

Marking Grid:

Approach to alternative explanations Construction of academic argument

CONTRAST (capture the expression of tensions, contrasts between ideas or research directions)

BACKGROUND (make reference to relevant other work)student is aware of

alternative analyses of young adult literature &

constructs her own argument!


DISCUSSION- Relationship between the

tutors’ marking grid and the salient sentences

SUMMARY: summarising the essay (do not contribute to any of the

evaluation aspects)

EMPHASIS: emphasising ideas as surprising or important

NOVELTY: referring to new research directions

describing research TENDENCY

Raising OPEN QUESTIONs.

Not usual

discourse

moves in

literature

analysis


DISCUSSION- Some Outliers

High grades to essays with few salient sentences

Literary style, which does not strictly follow the patterns of

concise scholarly communication.


DISCUSSION- Some Outliers

Low grades to essays with a relatively great number of

salient sentences

Simple and schematic style, and sometimes their syntactic

structure is not clear


CONCLUSION

Understanding the power and effectiveness of XIP in educational contexts. Output of XIP is related to tutors’ expectations in student essays.

(BACKGROUND & CONTRAST)

The parser’s performance is reasonably good although it has not been customized for this particular domain.

Further work is needed to recognise special literary writing style and integrate these features to the tool.

Follow up studies with student essays from various other disciplines.

LAK15 Short Paper Talk

Education

Transcript of LAK15 Short Paper Talk