Significant increase across proficiency levels (ω2=.33). · Web view2020. 6. 8. · # clauses...
Transcript of Significant increase across proficiency levels (ω2=.33). · Web view2020. 6. 8. · # clauses...
Table 1: Part-of-SpeechSub-category Measure Notes on operationalisation Study Findings
Verbs Verbs # all verbs (tagged with the Biber tagger)
Grant & Ginther 2000*a Increases with essay grade.
Mean frequency of all verbs (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase with proficiency level.
Verb forms % verb form types from the total # words
Llanes, Tragant & Serrano 2015
Significant increase over 3-week study abroad programme (d=.09).
Modal verbs # modal verbs (tagged with the ICE tagger)
Espada-Gustilo 2011a No clear increases across proficiency levels. N.B ANOVA is reported as significant but increase is not linear across the 3 proficiency levels.
# modal verbs (tagged with the Biber tagger)
Grant & Ginther 2000*a Increases with essay grade.
reference to Biber 1988 implies counts are per 1,000 words
Ferris 1994a No significant differences across proficiency levels.
Auxiliary modal verbs # auxiliary modal verbs Torras and Celaya 2001*a Increases over time.
Primary verb types # primary verb types Torras and Celaya 2001*a Increases over time.
Lexical verb types # lexical verb types Torras and Celaya 2001*a Increases over time.
Tense/aspect # present tense verbs(reference to Biber 1988 implies counts are per 1,000 words)
Ferris 1994a Significantly more used by higher proficiency group.
# present tense verbs (tagged with the Biber
Grant & Ginther 2000*a Increases with essay grade.
tagger)# simple present verb phrases
Verspoor, Schmid & Xu 2012a
Significant negative correlation with essay score (r=-.73).
% present tense verbs Santos, Nerbonne & Verspoor 2012
Amongst the strongest correlates of essay score in pooled model (Ranked 5th out of 8th in strongest correlates).
# simple past verb phrases Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.73).
Frequency counts of past-tense verbs
Guo, Crossley & McNamara 2013
Significant negative correlation with holistic essay scores for integrated essays (r=-.17) but no correlation reported for independent essays.
Frequency counts of verbs in non- third person singular present form
Guo, Crossley & McNamara 2013
Significant negative correlation with holistic essay scores for integrated essays (r=-.34) and independent essays (r=-.44).
Frequency counts of verbs in third person singular present form
Guo, Crossley & McNamara 2013
Significant positive correlation with holistic essay scores for integrated essays (r=.19) but no correlation reported for independent essays.
# progressive verb phrases Verspoor, Schmid & Xu 2012a
Non-significant positive correlation with essay score (r=.05).
# past tense/perfect aspect Ferris 1994a Significantly more verbs
verbs(reference to Biber 1988 implies counts are per 1,000 words)
produced by higher proficiency group.
# perfect aspect verbs (tagged with the Biber tagger)
Grant & Ginther 2000*a No clear increase across grades.
# present perfect verb phrases
Verspoor, Schmid & Xu 2012a
Non-significant negative correlation with essay score (r=-.00).
# past perfect verb phrases Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.17).
Verbs in base form Frequency counts of verbs in base form
Guo, Crossley & McNamara 2013
Significant negative correlation with holistic essay score for integrated essays (r=-.40) and independent essays (r=-.28).
# Non-finite verbs # non-finite verb types Torras & Celaya 2001*a Increases over time.
Non-finite verb ratio # non-finite verbs ÷ words Hou, Verspoor & Loerts 2016
Significant increase over time for higher proficiency university group.
Finite verb ratio # finite verbs ÷ words Hou, Verspoor & Loerts 2016
No significant change over time for higher proficiency university group.
Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Significant positive correlation with holistic writing score (r=.59).
Past participle verbs Frequency counts of past participle verbs
Guo, Crossley & McNamara 2013
Significant positive correlation with holistic essay scores for integrated
essays (r=.44) and independent essays (r=.46).
Gerund or present participle verbs
Frequency counts of gerunds or present participle verbs.
Guo, Crossley & McNamara 2013
Significant positive correlation with holistic essay scores for integrated essays (r=.19) but no correlation reported for independent essays.
Nouns # Nouns # nouns (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase with proficiency level.
# noun types Torras & Celaya 2001*a Increases over time.
reference to Biber 1988 implies counts are per 1,000 words
Ferris 1994a No significant differences across proficiency levels.
adjectives # Adjectives Mean frequency of adjectives (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase with proficiency level.
# adjective types Torras & Celaya 2001*a Increases over time.
# Participials (as part of combined variable)
reference to Biber 1988 implies counts are per 1,000 words
Ferris 1994a Significantly more used by higher proficiency group.
Lexical density # Adjectives + nouns +verbs/all words
total # lexical words/total # orthographic words x 100
Nihalani 1981 No significant differences between proficiency levels.
total # lexical words/total # words in essay
Engber 1995 Non-significant positive correlation with writing scores (r=.23).
total # content words / total # words per essay
Becker 2010 Significant increase across proficiency levels (ω2=.33).
N.B. No post-hoc differences between the 3 proficiency levels are made clear or reported.
total # content words / total # words per essay
Vidakovic & Barker 2010 No clear differences between CEFR levels.
total # lexical words/total # words in essay
Kim 2014 Significant positive correlation with proficiency level (η²=.21).
total # lexical words/total # words in essay
Lorenzo & Rodriguez 2014 No significant difference across school years.
NB. We are assuming that this is the measure used, though the description in the article appears to confuse density and diversity; specifically with type-token ratio.
Gregori-Signes & Clave-Arroitia 2015*
Decreases across time and proficiency level.
Total # lexical words/total # words
Hou, Verspoor & Loerts 2016
Significant increase over time for low proficiency university group and a reported non-significant difference for the high school group and the high proficiency university group.
Content word ratio Zheng 2016 No significant difference over time.
Weighted lexical densityContent words/functions words, with high-frequency items given half the weight
Banerjee, Franceschina & Smith 2007*
Increases across proficiency levels [NB. Though the authors make this claim, it's not clear that the data really
of low-frequency items support it. Significant differences across band levels go down as well as up].
articles #Articles # of articles (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase with proficiency level.
# articles per 100 words Castro 2004 Non-significant increase across proficiency levels.
#Singular definite and indefinite articles
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More frequent in higher rated essays.
# Indefinite articles # indefinite articles (tagged with the Biber tagger)
Grant & Ginther 2000*a Increases with essay grade.
# Definite articles # definite articles (tagged with the Biber tagger)
Ferris 1994a Significantly more used by higher proficiency group.
# definite articles (tagged with the Biber tagger)
Grant & Ginther 2000*a Increases with essay grade.
# Singular or plural determiners
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More determiners used in lower rated essays.
# Demonstrative determiners
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More determiners used in lower rated essays.
# Post-determiners Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More determiners used in higher rated essays.
# Pre-qualifiers Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More pre-qualifiers used in higher rated essays.
# Pre-quantifiers Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More pre-quantifiers used in higher rated essays.
adverbs #Adverbs # Intensifying adverbs Perez-Paredes & Diez-Bedmar 2012a
Significant differences in the use of “very” between adjacent grades only for 7 vs 8, with grades 8 vs 9 or 9 vs
10.Significant differences in the use of “very” between non-adjacent grades 7 vs 9 and grades 7 vs 10
# adverbs (tagged with the Biber tagger)
Grant & Ginther 2000*a Increases with essay grade.
Mean frequency of (general) adverbs (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase with proficiency level.
# adverb types Torras & Celaya 2001*a Increases over time.
Types of connectors # Connectors Normed counts per 200 words
Shaw & Liu 1998 Significant increase over time.
# Conjunctions # conjunctions per 100 words
Castro 2004 Non-significant decrease across proficiency levels.
Incidence of and Coh-metrix index Crossley, Kyle & McNamara 2016
Significant positive correlation with organisation (r=.13) and overall score (r=.17).
Incidence of intentional connectives
Coh-metrix index Crossley, Kyle & McNamara 2016
Significant negative correlation with overall score (r=-.15).
Incidence of causal connectives
Coh-metrix index Crossley, Kyle & McNamara 2016
Significant negative correlation with organisation (r=-.16) and overall score (r=-.16).
# Positive connectives Coh-metrix index Crossley, Kyle & McNamara Significant positive
2016 correlation with organisation (r=.24) and with overall score (r=.33).
Significant increase over time (n2=.20)
# Positive intentional connectives
Coh-metrix index Crossley, Kyle & McNamara 2016
Significant negative correlation with organisation score (r=-.12) but no reporting for overall score.
# Positive causal connectives Coh-metrix index Crossley, Kyle & McNamara 2016
Significant positive correlation with organisation score (r=.21) and overall score (r=.30).
# Negative connectives Coh-metrix index Crossley, Kyle & McNamara 2016
Significant negative correlation with overall score (r=-.11) but no result reported for organisation scores.
pronouns # All pronouns # all pronouns (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase with proficiency level.
# pronouns per 100 words Castro 2004 No clear pattern across proficiency levels.
# Personal pronouns Frequency counts of personal pronouns(not explicitly normalized for text length)
Guo, Crossley & McNamara 2013
Significant negative correlation with holistic essay scores for integrated essays (r=-.32) and independent essays (r=-.30).
# 1st/2nd person pronouns # 1st/2nd person pronouns(tagged with the Biber tagger)
Ferris 1994a Significantly more used by higher proficiency group.
Grant & Ginther 2000*a Increases with essay grade.# 1st person pronouns (singular)
Normed counts per 200 words
Shaw & Liu 1998 Significant decrease over time.
# 1st person pronouns (plural)
Normed counts per 200 words
Shaw & Liu 1998 No significant decrease over time.
# 3rd person pronouns # 3rd person pronouns(tagged with the Biber tagger)
Ferris 1994a No significant difference across proficiency levels.
Grant & Ginther 2000*a Increases with essay grade.# Impersonal pronouns # impersonal pronouns
(tagged with the Biber tagger)
Ferris 1994a Significantly more used by higher proficiency group.
Pronoun: noun ratio Coh-metrix index Crossley, Kyle & McNamara 2016
Significant negative correlation with overall score (-.15) but no result reported for correlation with organisation score.
Prepositions # Prepositions # all prepositions(tagged with the Biber tagger)
Grant & Ginther 2000*a Increases with essay grade.
# all prepositions (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase with proficiency level.
# Prepositional phrases Normed per 1,000 words Mazgutova & Kormos 2015 No significant change over time for either the high school or undergraduate group.
Coh-metrix index Guo, Crossley & McNamara 2013
Significant positive correlation with holistic essay scores for integrated
essays (r=.17) but no correlation reported for independent essays.
Number of conjunctions Coh-metrix index Normed incidences of conjunctions
Crossley, Kyle & McNamara 2016
Significant positive correlation with organisation (r=.18) and overall scores (r=.24).
Number of subordinating conjunctions
# Subordinating conjunctions
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More conjunctions used in lower rated essays
Normed incidences of simple subordinators (Coh-metrix index)
Crossley, Kyle & McNamara 2016
Significant positive correlation with overall scores (r=.11).
Pooled POS measures Pooled POS measure Pooled POS measure consists of measures from Lu’s (2011) Syntactic Complexity Analyser and count based measures for: nouns (various types), pronouns (various types), adjectives, conjunctions, interjections, determiners, prepositions, verbs (various types)
Vajjala 2018 Significant positive correlation with proficiency level for TOEFL essays (r=.74) and FCE texts (r=.49).
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
Table 2: Phrase structure2a: Noun Phrase structure
Measure Notes on operationalisation Study FindingsMean length of noun phrase # words per noun phrase
(manual counts)Bulté & Housen 2014 Significant increase with time (Cohen’s d=.41).
Significant positive correlation with overall essay score (r=.36) and language use score
(r=.37).Lahuerta-Martinez
2018 Significant positive correlation with holistic
essay score (r=.63). Significantly longer noun phrases produced in
year 4 than in year 3 (p=<0.001).Verspoor, Lowie, Chan
& Vahtrick (2017, study 1)
Significant positive correlation with holistic essay score (r=.47).
Number of Modifiers per noun phrase
Coh-metrix index Crossley & McNamara 2014
Significant increase over time (n2=.12). Significant positive correlation with language
use score (r=.21) and combined essay score (r=.21).
Guo, Crossley & McNamara 2013
Significant positive correlation with holistic essay scores for integrated essays (r=.26) and independent essays (r=.34).
Mazgutova & Kormos 2015
Significant increase across time for the undergraduate group (r=-.48) but no change for the high school group.
Dependents per nominal Average dependents per nominal Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.31)
Number of complex nominals # Complex nominals (SCA index) Lorenzo & Rodriguez 2014
Significant decrease from 10th-11th grade (Hedge's g = .40)
No significant differences between 9th-10th or 11th-12th grades.
Number of complex nominals per clause
# Complex nominals ÷ # clauses (SCA index)
Hou, Verspoor & Loerts 2016
Significant increase over time for senior high school group.
Significant increase over time for lower proficiency university group.
No significant change over time for higher proficiency university group.
Kim 2014 Significant increase with proficiency level.
Kim & Crossley 2018 Non-significant positive correlation with source-based essay scores (r=.11).Significant positive correlation with independent essay scores (rs=.22).
Kyle & Crossley 2017 Significant correlation with essay grade but removed from analysis due to multicollinearity.
Kyle & Crossley 2018 Significant correlation with essay grade but removed from analysis due to multicollinearity with mean length of clause.
Lorenzo & Rodriguez 2014
Significant decrease from grade 10-11 (Hedge’s g=-.44) and grade 11-12 (hedge’s g=-.86).
Non-significant increase from grade 9-10 (Hedge’s g=.28).
Lu 2011 Significant increase from years: 1-2; 2-3; 1-3; 2-4; 1-4.
Paquot 2018 No significant differences across proficiency levels.
Shih & Ma 2012 Significant differences between grades 7-11, 8-11, 9-11 and 7-12.
Yang, Lu & Weigle 2015
Significant positive correlation with writing quality scores for ‘appearance’ topic (r=.12) and ‘future’ topic (r=.20).
Yau 1991 Higher ESL proficiency group produce significantly more complex nominals per clause than lower ESL proficiency group.No significant increase in complex nominals per clause between higher ESL proficiency group and L1 native speaker group.
Yoon & Polio 2016 No significant increase over time for narrative and argumentative essays.
Yoon 2017 Significant differences between proficiency levels (η2=0.03).
Post-hoc tests reveal non-significant differences between adjacent proficiency levels and non-adjacent B1.1-B2 levels.
Significant differences between non-adjacent levels CEFR A2-B1.2 and A2-B2.
Number of complex nominals per t-unit
# Complex nominals ÷ # t-units (SCA index)
Hou, Verspoor & Loerts 2016
Significant increase over time for senior high school group.
Significant increase over time for lower proficiency university group.
Significant decrease over time for higher proficiency university group.
Kim 2014 Significant increase with proficiency level (η2 =.23)Kim & Crossley 2018 Non-significant positive correlation with source-based
essay scores (rs=.09) and independent essay scores (rs=.10).
Kyle & Crossley 2018 Removed from analysis as it either violated normality assumption or failed to reach a small effect size for correlation with essay score.
Lorenzo & Rodriguez 2014
Significant decrease from grade 10-11 (Hedge’s g=-.33) and grade 11-12 (Hedge’s g=-.72).Non-significant increase from grade 9-10 (Hedge’s g=.16).
Lu 2011 Significant increase from years: 1-2; 1-3; 1-4Paquot 2018 No significant differences across proficiency levelsYoon & Polio 2016 No significant increase over time for narrative and
argumentative essays.# Complex nominals ÷ # t-units (manual counts)
Shih & Ma 2012 Significant differences between grades 7-10, 8=10, 7-11, 8-11, 9-11, 7-12 and 8-12.
Yau 1991 Higher ESL proficiency group produce significantly more complex nominals per t-unit than lower ESL proficiency group.
No significant increase in complex nominals per t-unit between higher ESL proficiency group and L1 native speaker group.
Number of ‘Of phrases’ as post-modifiers (concrete/locative meaning)
# per noun phrase Parkinson & Musgrave 2014
No significant difference between MA and EAP groups.
Number of ‘Of phrases’ as post-modifiers (abstract meaning)
# per noun phrase Parkinson & Musgrave 2014
No significant difference between MA and EAP groups.
Number of participial pre-modifiers
# per noun phrase Parkinson & Musgrave 2014
Significantly more pre-modifiers used by MA group than EAP group.
Number of appositive noun phrases as post-modifiers
# per noun phrase Parkinson & Musgrave 2014
Significantly more noun phrases used by MA level than EAP group.
To clauses as post-modifiers # per noun phrase Parkinson & Musgrave 2014
No significant difference between MA and EAP groups.
Simple postmodifiers NPs with one postmodifier only. Normed per 1,000 words
Mazgutova & Kormos 2015
No significant change across time.
Complex postmodifiers NPs with more than one postmodifier only. Normed per 1,000 words
Mazgutova & Kormos 2015
Significant increase over time for undergraduate group (r=-.37) but no significant change for high school group.
Number of complex nominals in subject position
Mean # complex nominals in subject position
Mazgutova & Kormos 2015
No significant change over time for either the undergraduate or high school groups.
Dependents per object of the preposition
Average dependents per object of the preposition
Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.30).
Prepositions per nominal Prepositions per nominal Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.29).
Prepositions per object of preposition
Prepositions per object of preposition
Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.28).
Adjectival modifiers per object of Adjectival modifiers per object of Kyle & Crossley 2018 Significant positive correlation with holistic essay
preposition preposition grade (r=.26).Dependents per object per preposition
Dependents per object per preposition
Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.26).
Dependents per nominal (SD) Standard deviation of dependents per nominal
Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.25).
Dependents per direct object (SD) Standard deviation of dependents per direct object
Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.25).
Dependents per direct object Average dependents per direct object
Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.23).
Dependents per nominal subject Dependents per nominal subject Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.21).
Determiners per nominal Determiners per nominal Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.15).
Adjectival modifiers per direct object
Adjectival modifiers per direct object
Kyle & Crossley 2018 Significant positive correlation with holistic essay grade (r=.15).
Number of attributive adjectives # Attributive adjectives (tagged with the Biber tagger)
Grant & Ginther 2000*a
Increases with essay grade.
Normed per 100,000 words Staples & Reppen 2016
Significant positive correlation with language score for L1 Chinese (r=.24) but not L1 Arabic (r=.16) writers.
# per noun phrase Parkinson & Musgrave 2014
Significantly more adjectives used by EAP group than MA group.
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More used in high rated essays.
Number of common attributive adjectives
# per noun phrase Parkinson & Musgrave 2014
Significantly more attributive adjectives used by EAP group than MA group.
Number of less common attributive adjectives
# per noun phrase Parkinson & Musgrave 2014
Significantly more less common adjectives used by EAP group than MA group.
Number of Pre-modifying nouns # pre-modifying nouns Grant & Ginther 2000*a
Increases with essay grade.
Normed per 100,000 words Staples & Reppen 2016
Significant positive correlation with language score for L1 Arabic (r=.23) but not L1 Chinese (r=-.10) writers.
# per noun phrase Parkinson & Musgrave 2014
Significantly more pre-modifying nouns used by MA group than EAP group.
Possessive nouns as pre-modifiers # per noun phrase Parkinson & Musgrave 2014
Significantly more possessive nouns used by MA group than EAP group.
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
2b: Verb phrase/clause structureMeasure Notes on
operationalisationStudy Findings
Number of verb phrases # Verb phrases Crossley & McNamara 2014
Significant decrease with time (η2=.10). Significant negative correlation with language use score
(r=-.23) and combined essay score (r=-.24).
# Verb phrases (SCA index)
Lorenzo & Rodriguez 2014
Significant decrease from grade 9-10 (Hedge’s g=-.40). Non-significant decrease from grade 10-11 (Hedge’s g=
-.10).Significant increase from grade 11-12 (Hedge’s g=.51).
Number of conditional verb phrases
# Conditional verb phrases
Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.20).
Number of passive verb phrases
# Passive verb phrases Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.18).
Verb phrase density Incidence of verb phrase scores
Aryadoust 2016 Predicts language scores only at pre-course essay stage. Reported as having greatest score predictive power at pre
and post course levels.
Verb argument construction: Delta P
TAASSC index Kyle & Crossley 2017 Significant positive correlation with holistic essay score (r=.25).
Verb argument construction: faith score
TAASSC index Kyle & Crossley 2017 Significant positive correlation with holistic essay score (r=.17).
Average verb lemma construction frequency
TAASSC index Kyle & Crossley 2017 Significant negative correlation with holistic essay score (r= -.23).
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
2: AdverbialsMeasure Notes on operationalisation Study Findings
# Adverbials # adverbials (tagged with the Biber tagger)
Ferris 1994a No significant differences across proficiency levels
# Adverb wh- # adverbs (wh adverbs) (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase across proficiency levels
# Adverbial clauses # of adverbial clauses (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
Ratio of adverbial finite clauses
% of adverbial finite clauses over all clause types. Adverbial finite clause is taken to be a clause with a finite verb functioning as an adverbial
Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Non-significant negative correlation with holistic essay score (r=-.29).
Adverbial subordination # adverbials that indicate subordination (e.g. ‘because she was late’)(tagged with the Biber tagger)
Grant & Ginther 2000*a Increases with essay grade.
Ratio of adverbial clauses to secondary clauses
# adverbial clauses ÷ # secondary clauses (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
# Causative adverbial clauses
# causative adverbial clauses normed per 100,000 words
Staples & Reppen 2016 No significant correlations with language ratings for L1 Arabic or L1 Chinese groups
# Conditional adverbial clauses
# conditional adverbial clauses normed per 100,000
Staples & Reppen 2016 Significant positive correlation with language
words rating for L1 Chinese (r=.24) but not L1 Arabic (r=-.08) writers.
normed per 1,000 words Mazgutova & Kormos 2015 Significant increase over time for undergraduate group (r=-.44) but no significant change for high school group.
# if clauses Normed count per 200 words
Shaw & Liu 1998 No significant increase over time.
# Other adverbial clauses # other adverbial clauses normed per 100,000 words
Staples & Reppen 2016 Significant positive correlation with language rating for L1 Chinese (r=.24) but not L1 Arabic (r=-.04) writers.
# Finite adverbial clauses # finite adverbials Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.38)
2d: Prepositional phrase structureMeasure Notes on operationalisation Study Findings
Number of prepositional phrases # Prepositional phrases (Coh-metrix index)
Crossley & McNamara 2014
Non-significant increase over time (η2 =.06).
Non-significant positive correlation with language use score (r=.18) and combined essay score (r=.23).
# Prepositional phrases (tagged with the Biber tagger)
Ferris 1994a Significantly more prepositional phrases used by higher proficiency group.
Number of total prepositional phrases with a locative/concrete meaning
# per noun phrase Parkinson & Musgrave 2014
No significant difference between MA group and EAP group.
Number of total prepositional phrases with abstract meaning
# per noun phrase Parkinson & Musgrave 2014
Significantly more prepositional phrases used by MA group than EAP group.
Number of post noun modifying prepositional phrase
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More prepositional phrases used in higher rated essays.
Number of prepositional phrases with prepositions other than ‘of’ (concrete/locative meaning)
# per noun phrase Parkinson & Musgrave 2014
No significant difference between MA group and EAP group.
Number of prepositional phrases with prepositions other than ‘of’ (abstract meaning)
# per noun phrase Parkinson & Musgrave 2014
Significantly more prepositional phrases used by MA group than EAP group.
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
Table 3: Clause structureSub-category Measure Notes on operationalisation Study Findings
Number of words before the main verb
# Words before main verb Coh-metrix index Crossley & McNamara 2014 Significant increase over a semester ¿¿=.09).
Non-significant positive correlation with language use scores (r=.12) and combined essay score (r=.17).
Diversity of clauses # Different clauses # different clauses Casanave 1994 Non-linear increase over time.
All complement clauses # Complement clauses # that complement and infinitive complements(features tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase across proficiency levels.
Noun clauses # Noun clauses # noun clauses (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
Nominal clause ratio Ratio of nominal clauses to all secondary clauses (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
# Finite nominals clauses # finite nominal clauses Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.34).
Ratio of finite nominal clauses
% finite nominal clauses over all clause types. A finite nominal clause is taken to be a clause with a finite verb functioning as a nominal
Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Non-significant positive correlation with holistic essay score (r=.21).
# Nominal subjects per clause
Kyle & Crossley 2018 Significant negative correlation with holistic essay score (r=-.17).
Relative clauses # Relative clauses # relative clauses (tagged with the Biber tagger)
Ferris 1994a Significantly more relative clauses used by higher proficiency group.
normed per 1,000 words Mazgutova & Kormos 2015 Significant increase over time for undergraduate group (r=-.37) but no significant change for high school group.
# per noun phrase Parkinson & Musgrave 2014 Significantly more relative clauses used by MA group than at EAP group.
Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.37).
# relative clauses (tagged with the ICE tagger)
Espada-Gustilo 2011a Significant increase across proficiency levels.
Ratio of relative clauses to all clause types.
Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Non-significant positive correlation with holistic essay score (r=.25).
# Subject, object and preposition relative clauses
# relative clause types (tagged with the Biber tagger)
Grant & Ginther 2000*a Increases with essay grade.
# That relative clauses Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More relative clauses used in lower rated essays.
Normed frequency counts per 200 words
Shaw & Liu 1998 No significant increase over time.
# Wh relative clauses Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
Slightly more relative clauses used in lower rated essays.
Normed frequency counts per 200 words
Shaw & Liu 1998 No significant increase over time.
# Subject relative clauses Coh-metrix index Crossley & McNamara 2014 Non-significant increase over time (η2 =.04).
Non-significant positive correlation with language use score (r=.08) and combined essay score (r=.10).
Finite complement clauses # Noun + that clauses per noun phrase Parkinson & Musgrave 2014 No significant difference between MA level and EAP groups.
Normed # per 100,000 words
Staples & Reppen 2016 No significant correlation with language rating for L1 Chinese or L1 Arabic writers.
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
Slightly more clauses used in higher rated essays.
Normed count per 200 words
Shaw & Liu 1998 Significant increase over time.
# ‘Of + ing’ noun complement clauses
# per noun phrase Parkinson & Musgrave 2014 No significant difference between MA level and EAP groups.
# Verb + that clauses Normed # per 100,000 words
Staples & Reppen 2016 No significant correlation with language rating for L1 Chinese or L1 Arabic writers
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More clauses used in higher rated essays.
That clause adjective complements
# That clause adjective complements (manual counts)
Normed frequency counts per 10,000 words
Taguchi, Crawford & Wetzel 2013*
More clauses used in lower rated essays.
Adjectival clauses = relative clauses)
# Adjectival clauses (manual counts)
# adjectival clauses (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
ratio of adjectival clauses to secondary clauses (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
Non-finite clauses # Non-finite clauses # non-finite clauses Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.41).
# Non-finite elements per clause
# non-finite elements ÷ # clauses
Yang, Lu & Weigle 2015 Non-significant positive correlation with writing quality scores for ‘appearance’ topic (r=.03) and significant positive correlation with writing quality score for ‘future’ topic (r=.20).
# Non-finite constructions Manual frequency counts Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Non-significant positive correlation with holistic
essay score (r=.13)Participial clauses -‘ed’ participial clauses as
post-modifiers# per noun phrase Parkinson & Musgrave 2014 Significantly more clauses
used by MA group than EAP group.
‘ing’ participial clauses as post-modifiers
# per noun phrase Parkinson & Musgrave 2014 No significant difference between MA and EAP group.
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
Table 4: CoordinationSub-category Measure Notes on operationalisation Study Findings
Indices of coordination
Coordination index # of independent clauses ÷ # of combined clauses
Bardovi-Harlig 1992*
Decreases as proficiency level increases.
Casanave 1994 Non-linear increase over time
Gene-Gil, Juan-Garau & Salazar-Noguera 2015
Significant increase for CLIL group (p=0.01) but not for the non-CLIL group over 3 school years.
All Coordination
Implies coordination combines phrasal and clausal coordination
# clauses and phrases that indicate coordination (tagged with the Biber tagger)
Ferris 1994a No significant differences across proficiency levels
# clauses and phrases that indicate coordination (tagged with the ICE tagger)
Espada-Gustilo, 2011a
Significant increase across proficiency levels
Sentence coordination
Ratio of sentences containing coordination
Ratio of sentences beginning with ‘and’, ‘but’, ‘or’ to all sentences (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
# Sentences containing coordination
# sentences containing coordination (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
Phrasal coordination
# Coordinate phrases per clause
SCA index Hou, Verspoor & Loerts 2016
Significant increase over time for senior high school group.
No significant change over time for lower proficiency university group.
Significant increase over time for higher proficiency university group.
Kim 2014 No significant difference across proficiency levels.
Kim & Crossley 2018
Non-significant positive correlation with source-based essay scores (rs=.03).
Significant positive correlation with independent essay scores (rs=.19).
Kyle & Crossley 2017
Significant positive correlation with holistic essay score (r=.19)
Kyle & Crossley 2018
Significant positive correlation with holistic essay score (r=.19).
Lu 2011 Significant increase from year: 1-3; 1-4 Paquot 2018 No significant differences across proficiency levels
Yang, Lu & Weigle 2015
Non-significant positive correlation with essay scores for ‘appearance’ topic (r=.03) and significant correlation with essay scores for ‘future’ topic (r=.21).
Yoon & Polio No significant increase over a semester for
2016 narrative and argumentative essays.
Yoon 2017 Significant differences between proficiency levels (η2=0.02)
# coordinate phrases per clause (manual counts)
Shih & Ma 2012 Significant differences between grades 7-10, 8-7 and 10-7.
# Coordinate phrases per T-unit
# coordinate phrases ÷ # t-units (SCA index)
Hou, Verspoor & Loerts 2016
Significant increase over time for senior high school group.
No significant change over time for lower proficiency university group.
Significant increase over time for higher proficiency university group.
Lu 2011 Significant increase from years: 1-3.Kim 2014 Significant increase with proficiency level.Kim & Crossley 2018
Non-significant positive correlation with independent essay scores (rs =.12).Non-significant negative correlation with source-based essay scores (rs= -.03).
Kyle & Crossley 2017
Measure either did not meet normality assumption or its effect size did not meet the minimum ‘small’ effect size required for further analysis.
Kyle & Crossley 2018
Measure either did not meet normality assumption or its effect size did not meet the minimum ‘small’ effect size required for further analysis.
Paquot 2018 No significant differences across proficiency levels
Yoon & Polio 2016
No significant increase over time for narrative and argumentative essays.
# coordinate phrases ÷ # t-units (manual counts)
Shih & Ma 2012 Significant differences between grades 7-10, 8-7 and 10-7.
# Coordinate phrases # coordinate phrases (SCA index) Lorenzo & Non-significant decrease from grade 9-10
Rodriguez 2014 (Hedge’s g=.-.09). Non-significant decrease from grade 10-11
(Hedge’s g=.-.10). Significant increase from grade 11-12
(Hedge’s g=.37).Clausal coordination
# Coordinate clauses # coordinate clauses Torras and Celaya 2001*
Increases over time.
coordinate clause ratio # of coordinate clauses ÷ # sentences
Bulté & Housen 2014
Significant increase over a semester (Cohen’s d=.31).
Non-significant positive correlation with overall essay score (r=.11) and language use score (r=.09).
Lahuerta-Martinez 2018
Significant positive correlation with holistic essay scores (r=.38).
Significantly more clausal coordination in year 4 than year 3 texts (p=<0.001).
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
Table 5: SubordinationMeasure Notes on operationalisation Study Findings
Subordinate clause ratios
# Dependent clauses ÷ # clauses (manual counts)
Banerjee, Franceschina & Smith, 2007*
Non-linear increase with proficiency level.
Santos, Nerbonne & Verspoor 2012
Amongst the strongest correlates of essay score (Ranked 4th out of 8 strongest correlates in pooled prediction model).
Bulté & Housen 2014 Non-significant increase over time (Cohen’s d=.04) Significant positive correlation with holistic essay score
(r=.24) and language use score (r=.29).
Hirano 1991 Significant positive correlation with test scores: texts with specified audience (r=.48); texts with no specified audience (r=.32).
Homburg 1984 Overall significant increase across levels but individual level comparisons are not significant.
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.Knoch, Rouhshad, Oon & Storch 2015
No significant change over time.
Lahuerta-Martinez 2018
Significant positive correlation with holistic essay scores (r=.24).
Significantly more dependent clauses in year 4 texts than year 3 texts (p=<0.001).
Shih & Ma 2012 Significant differences between grades 7-10, 8-10, 9-10, 7-11, 8-11, 7-12 and 8-12.
Storch & Hill 2008 Significant increase over time.Storch 2009 No significant change over time.Struc & Wood 2011 Significant increase over time for narrative essays.
No change over time for argumentative essays.SCA index Hou, Verspoor &
Loerts 2016 No significant change over time for senior high school
group or lower proficiency university group. Significant decrease over time for higher proficiency
university group.Kim 2014 Significant increase with proficiency level. Lorenzo & Rodriguez 2014
Non-significant decrease from grade 9-10 (Hedge’s g=-.13).
Non-significant increase from grade 10-11 (Hedge’s g=.08).
Significant decrease from grade 11-12 (Hedge’s g=-.47).
Lu 2011 Significant decrease from years 2-4.Paquot 2018 No significant differences across proficiency levels.Yoon & Polio 2016 No significant increase over a semester for narrative and
argumentative essays.# Dependent clauses ÷ # t-units (SCA index)
Hou, Verspoor & Loerts 2016
No significant change over time for senior high school group or lower proficiency university group.
Significant decrease over time for higher proficiency university group.
Kim 2014 Significant increase with proficiency level.Lorenzo & Rodriguez 2014
Non-significant decrease from grade 9-10 (Hedge’s g=-.12).
Non-significant decrease from grade 10-11 (Hedge’s g=-.02).
Non-significant decrease from grade 11-12 (Hedge’s g=-.25).
Lu 2011 Significant decrease from years 2-4.Mazgutova & Kormos 2015
No significant change across time for either the undergraduate or high school groups.
Paquot 2018 No significant differences across proficiency levels.
Yang, Lu & Weigle 2015
Significant positive correlation with essay score for ‘appearance’ topic (r=.14) and ‘future’ topic (r=.08).
Yoon & Polio 2016 No significant increase over time for narrative and argumentative essays.
# Dependent clauses ÷ # t-units Shih & Ma 2012 Significant differences between grades 7-10, 8-10, 9-10, 7-11, 8-
(manual counts) 11, 7-12 and 8-12.Struc & Wood 2011 Non-significant increase over time for narrative and
argumentative essays.
Overall subordination
# Complement clauses, relative clauses and adverbial subordination
Grant & Ginther 2000*a
Increases with essay grade.
Embedded clauses Coh-metrix index Guo, Crossley & McNamara 2013
Significant negative correlation with holistic essay scores for independent essays (r=-.34) but no correlation reported for integrated essays.
Number of subordinate clauses
# Subordinate clauses Torras & Celaya 2001a Increases over time for late-starter group only.Espada-Gustilo 2011a Significant increase across proficiency levels.
S-bars Incidences of S-bars Crossley & McNamara 2014
Non-significant development over time ¿¿=.00). Non-significant negative correlation with language use
score (r=-.09) and combined essay score (r=-.13).Infinitives Incidences of infinitives Crossley & McNamara
2014 Non-significant increase over time (η2=.00). Significant positive correlation with language use score
(r=.23) and combined essay score (r=.32).Infinitive clause per 1,000 words Mazgutova & Kormos
2015Significant decrease over time for the high school group but no significant change over time for undergraduate group.
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
Table 6: Information structureSub-category Measure Notes on operationalisation Study Findings
Passive measures
# Passive sentences # sentences with one or more passives
Kameen 1979 Significantly more passives used by 'good' writers
Kawata 1992 Significant differences between proficiency levels
# Passives per sentence # passives ÷ # sentences Kameen 1979 Significantly more passives used by 'good' writers
# Sentences with at least 1 passive # sentences with at least 1 passive Kameen 1979 Significantly more passives used by 'good' writers.
# Passives # by and agentless passives(tagged with the Biber tagger)
Ferris 1994a Significantly more passives used by higher proficiency group.
Grant & Ginther 2000*a
Increases with essay grade.
# by and agentless passives(tagged with the ICE tagger)
Espada-Gustilo, 2011a
Significant increase across proficiency levels.
# passives (manual counts) Verspoor, Schmid & Xu 2012a
Increases across proficiency groups 1-4 (dropping from 4-5).
Normed counts per 200 words Shaw & Liu 1998 Significant increase over time.# Dynamic passives # dynamic passives (manual counts) Kameen 1979 Significantly more passives
used by 'good' writers.# Passives per clause # of passives per clause (manual
counts)Kameen 1979 Significantly more passives
used by 'good' writers.# Passives in main clause # passives in main clauses (manual Kameen 1979 Significantly more passives
counts) used by 'good' writers.# Passives in secondary clauses # passives in secondary clauses
(manual counts)Kameen 1979 Significantly more passives
used by 'good' writers.# Passives per sentence # of passives per sentences (manual
counts)Kameen 1979 Significantly more passives
used by 'good' writers.# Passives per T-unit # of passives per T-unit (manual
counts)Kameen 1979 Significantly more passives
used by 'good' writers.Active measures
# Actives in secondary clauses # actives in secondary clauses (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ proficiency writers
# Actives in main clauses # actives in main clauses (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ proficiency writers
nominalisations Frequency counts of words ending in ‘-ment’, ‘-ity’, ‘-ness’.
Frequency counts (tagged with the Biber tagger)
Grant & Ginther 2000*a
Increases with essay grade.
Mean frequency of nominalisations Frequency counts (tagged with the ICE tagger)
Espada-Gustilo 2011a
Significant increase with proficiency level.
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
Table 7a: Generic indices: T-unit-based measures
Measure Notes on operationalisation Study FindingsLength in words # Words per T-unit (manual counts) Arnaud 1992 Significant positive correlation with essay
scores (r=.32).
Arthur (1979, study 1) Non-significant decrease between 12 timed essays over time.
Becker 2010 Significant increase between proficiency levels (ω2=.60).
Significant increases between proficiency levels 1-2, 1-3 and levels 2-3.
Bulté & Housen 2014 Significant increase over a semester (cohen’s d=.47).
Significant positive correlation with overall essay score (r=.40) and language use score (r=.43).
Casanave 1994 Increases over time.
Cumming, Kantor, Baba, Erdosy,
Eouanzoui, & James 2005
Significant increase across proficiency levels (η2=.11).
Flahive & Snow 1980 Significantly increases across proficiency levels.Gene-Gil, Juan-Garau
& Salazar-Noguera 2015
For the CLIL group there is a significant increase in t-unit length (p=0.04) and a non-significant increase for the non-CLIL group.
Gyllstad, Granfeldt, Bernardini, & Kallkvist
2014
Significant positive correlation with proficiency level (Kendall's tau=.48)
Hirano 1991 Significant positive correlation with test scores: texts with specified audience (r=.60), texts with
no specified audience (r=.58).Homburg 1984 Significant increase across proficiency levels.
Significant from level 5 to level 6 onlyHo-Peng 1983 Significantly increases across proficiency groups
for both writing tasks.Intaraprawat &
Steffensen 1995*Increase in “good” essays
Ishikawa 1995 No significant change over time.
Kameen 1979 Significantly longer in ‘good’ writers' texts.
Kawata 1992 Significant increase across proficiency levels.Knoch, Rouhshad, Oon
& Storch 2015Significant increase over time (p=0.05).
Knoch, Rouhshad & Storch 2014
Non-significant increase over time.
Larsen-Freeman & Strom 1977
Linear increase across proficiency level. However, none of the increases are statistically significant.
Larsen-Freeman 1978 Significant linear increase across proficiency levels.
Larsen-Freeman 1983 Study 1: Significant increase across proficiency levelsStudy 2: Significant increase across time
Larsen-Freeman 2006 T-unit length increases over time (reported as a group average of 1+ standard deviation over time)
Nihalani 1981 Non-significant increase across proficiency levels.
Perkins 1980 Non-significant differences between
proficiency levels. Non-significant positive correlation
with writing score (r=.33).
Plakans, Gebril and Bilki 2016
Significant positive correlation with integrated essay scores (r=.32).
Shih & Ma 2012 Significant difference between grades 7-10, 8-10, 9-10, 7-11, 8-11 and 9-11.
Storch 2009 No significant change over timeStorch & Tapper 2009 No significant change over time
Struc & Wood 2011 Significant increase over time for narrative essays.
Non-significant increase over time for argumentative essays.
Tedick 1990 Significant increase across proficiency levels for general and field specific topics.
Xudong, Cheng, Varaprasad & Leng
2010
Non-significant increase over time.
Yau 1991 Higher ESL proficiency group produce significantly longer t-units than lower ESL proficiency group.
No significant increase in mean length of t-unit between higher ESL proficiency group and L1 native speaker group.
Yau & Belanger 1984 Significant increase with proficiency level for narrative and expository essays.
Youn 2014 Significant positive correlation with proficiency level (r s=.42) and pragmatic performance score (r s=.44).
Zyad 2012* Increase over time. Longer t-units used by more proficiency writers
# Words per T-unit (SCA index) Hou, Verspoor & Loerts 2016
Significant increase over time for senior high school group.
No significant change over time for lower proficiency university group
Significant decrease over time for higher proficiency university group.
Kim 2014 Significant increase with proficiency level (η2=.24)
Kim & Crossley 2018 Significant positive correlation with source-based essay scores (rs=.16).
Non-significant correlation with independent essay scores (rs =.08).
Lu 2011 Significant increase from years: 1-2; 1-3; 1-4
Mazgutova & Kormos 2015
No significant change across time for either the undergraduate or high school groups.
Paquot 2018 No significant differences across proficiency levels
Yang, Lu & Weigle 2015
Significant positive correlation with writing quality for ‘appearance’ topic (r=.21) and ‘future’ topic (r=.22).
Yoon & Polio 2016 Significant increase over a semester for narratives essays (ηρ2=.10) but not argumentative essays.
Yoon 2017 Significant differences between proficiency levels (η2=0.03).
Post-hoc tests reveal non-significant differences between adjacent proficiency levels.
Significant differences between CEFR A2-B2 and B1.1-B2 levels.
# Long t-units (21 words or more) (manual counts) Kameen 1979 Significantly more long t-units used by good writers
Ratio of long t-units to all t-units (manual counts) Kameen 1979 Significantly higher ratio used by good writersRatio of short t-units to all t-units (short t-units are 1-8 words long) (manual counts)
Kameen 1979 Significantly lower ratio used by good writers
# Short t-units (manual counts) Kameen 1979 Significantly lower number of short t-units used by good writers
# Medium t-units (9-20 words)(manual counts) Kameen 1979 Non-significant differences between ‘poor’ and ‘good’ proficiency writers.
Length in clauses # Clauses ÷ # t-units Banerjee, Franceschina & Smith,
2007*
Non-linear increase with proficiency level
Bardovi-Harlig & Bofman 1989
No significant difference across proficiency groups
Becker 2010 Significant increase across proficiency levels (ω2=.37)
Significant increase between levels 1-2 and 1-3.
Casanave 1994 Non-linear increase over timeCumming, Kantor,
Baba, Erdosy, Eouanzoui, & James
2005
No significant difference across proficiency levels.
Flahive & Snow 1980 Significantly increases across proficiency levelsGene-Gil, Juan-Garau
& Salazar-Noguera 2015
Significant increase over time for CLIL group (p=0.01) although the non-CLIL group also have a non-significant increase in subordination.
Gyllstad, Grenfeldt, Bernadini & Kalkvist
Significant positive correlation with proficiency (Kendall's tau=.46)
2014Hirano 1991 Significant positive correlation with test scores:
texts with specified audience (r=.46), texts with no specified audience (r=.30).
Hou, Verspoor & Loerts 2016
No significant change for senior high school students or lower proficiency university group.
Significant decrease over time for higher proficiency university group.
Ishikawa 1995 No significant change over timeKameen 1979 Non-significant differences between ‘good’ and
‘poor’ writers.Kim 2014 Significant increase with proficiency level.
Knoch, Rouhshad, Oon & Storch 2015
No significant change over time.
Knoch, Rouhshad & Storch 2014
Non-significant increase over time.
Larsen-Freeman 2006 Increase in clauses per t-unit over time (reported as a group average of 1+ standard deviation over time)
Llanes, Tragant & Serrano 2015
No significant increase over time on abroad programme.
Lu 2011 No significant differences across year groups.Paquot 2018 No significant differences across proficiency
levelsPerkins 1980 Non-significant differences between
proficiency levels. Non-significant positive correlation
with writing score (r=.10)
Qin & Uccelli 2016 Non-significant negative correlation
with writing quality scores for argumentative essays (r=-.13).
Non-significant positive correlation with writing quality for narrative essays (r=.09).
Shih & Ma 2012 Significant difference between grades 7-10, 8-10, 8-11.
Storch & Hill 2008 Significant increase over timeStorch 2009 No significant change over time
Struc & Wood 2011 Non-significant increase over time for argumentative and narrative essays.
Yau & Belanger 1984 Significant increase over proficiency levels for narratives but not for expository essays.
Youn 2014 Significant positive correlation with proficiency level (r s=.25) and pragmatic performance score (r s=.30).
Yoon 2017 Non-significant differences between proficiency levels
Yoon & Polio 2016 Significant increase over a semester for narrative essays (ηρ2=.18) but not argumentative essays.
Zyad 2012* Increases over time. Longer length in clauses in more proficient texts.
# Verb phrases ÷ # t-units Hou, Verspoor & Loerts 2016
Significant increase over time for high school group.
No significant change over time for lower proficiency university group.
Significant decrease over time for higher proficiency university group.
Lorenzo & Rodriguez 2014
Non-significant decrease from grade 9-10 (Hedge’s g=-.09)
Non-significant decrease from grade 10-11 (Hedge’s g=-.13).
Non-significant decrease from grade 11-12 (Hedge’s g=-.16).
Lu 2011 No significant development across year groups.Yoon & Polo 2016 Non-significant increase over a semester for
narrative and argumentative essays (ηρ2=.00).Kim 2014 Significant increase with proficiency level.
Kyle & Crossley 2018 Measure either did not meet normality assumption or did not reach small effect size and was therefore removed from analysis.
Lu 2011 No discrimination ability for any of the proficiency levels.
Paquot 2018 No significant differences across proficiency levels
Shih & Ma 2012 Significant differences between grades 7-10, 8-10, 9-10, 10-12, 7-11, 8-11 and 9-11.
Complexity index Sum of weighted scores divided by number of T-units. Weighted score = score of weighted structures plus number of words in t-unit divided by number of words in t-unit:1 = derivational morphemes and adjectives; T-units with no embedded or complex morphological forms2 = relatives, passives, embedded clauses, possessives, comparatives3 = adverbial and noun clauses
Perkins 1980 Non-significant correlation with essay score (r=.19).
Flahive & Snow 1980 Significant increase across proficiency levels.
Complex t-unit ratio (CT/T)
# Complex T-units ÷ # t-units (SCA index) Hou, Verspoor & Loerts 2016
No significant change over time for senior high school group or the lower proficiency university group.
Significant decrease over time for higher proficiency university group
(p=0.001)
Kim 2014 Significant increase with proficiency level.Lu 2011 No significant development across year groups.
Paquot 2018 No significant differences across proficiency levels.
Number of complex t-units
% Complex t-units Casanave 1994 Non-linear increase over time # Complex t-units (SCA index) Lorenzo & Rodriguez
2014 Non-significant decrease from grade 9-
10 (Hedge’s g= -.23). Non-significant decrease from 10-11
(Hedge’s g=-.07). Non-significant increase from grade 11-
12 (Hedge’s g=.26).Number of t-units with more than one level of embedding
# t-units with more than one level of embedding Casanave 1994 Non-linear increase over time
Number of t-units joined by (,)
# t-units joined by (,) (manual counts) Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
Ratio of t-units joined by (,) to all t - units
# t-units joined by (,) ÷ # t-units (manual counts) Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
Number of t-units joined by ‘and’, ‘but’, ‘or’
# t-units joined by ‘and’, ‘but’, ‘or’ (manual counts) Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
Ratio of number of t-units joined by ‘and’, ‘but’, ‘or’ to all t-units
Ratio of t-units joined by ‘and’, ‘but’, ‘or’ to all t-units (manual counts)
Kameen 1979 Non-significant differences between ‘good’ and ‘poor’ writers.
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
Table 7b: Generic indices: : clause-based measuresMeasure Notes on
operationalisationStudy Findings
Mean length of clause
# Words ÷ # clauses Casanave 1994 Non-linear increase over timeGyllstad, Grenfeldt, Bernadini & Kalkvist
2014
Significant positive correlation with proficiency (Kendall’s tau=.31)
Hirano 1991 Significant positive correlation with test scores: texts with specified audience (r=.40), texts with no specified audience (r=.38).
Hou, Verspoor & Loerts 2016
Significant increase over time for high school group. Significant increase over time for lower proficiency university group No change over time for higher proficiency university group.
Ishikawa 1995 No significant change over timeKameen 1979 Significantly longer clauses used by good poor writers
Kim 2014 Significant increase with proficiency levelKim & Crossley 2018 Significant positive correlation with source-based essays (r=.14) and independent
essays (r=.25).Knoch, Rouhshad, Oon
& Storch 2015No significant change over time.
Knoch, Rouhshad & Storch 2014
Non-significant increase over time.
Kyle & Crossley 2017 Significant positive correlation with holistic essay score (r=.24).Lorenzo & Rodriguez
2014 Non-significant increase from grade 9-10 (Hedge’s g=.19). Significant decrease from grade 10-11 (Hedge’s g=-.43). Significant decrease from grade 11-12 (Hedge’s g=.-.70).
Lu 2011 Significant increase from years: 1-2; 1-3; 2-4; 1-4.
Paquot 2018 Significant increase across proficiency levels.
Qin & Uccelli 2016 Significant positive correlation with essay scores for argumentative essays (r=.32) and narrative essays (r=.19)
Shih & Ma 2012 Significant difference between grades 9 & 10, and 9-11.
Struc & Wood 2011 Non-significant increase over time for narrative and argumentative essays.
Yang, Lu & Weigle 2015 Non-significant positive correlation with writing quality score for ‘appearance’ topic (r=.13) and significant positive correlation with writing quality score for ‘future’ topic (r=.23).
Yau & Belanger 1984 Significant increase with proficiency level for expository essays but not for narratives.
Yoon & Polio 2016 No significant increase over a semester for narrative and argumentative essays. Yoon 2017 Significant differences between proficiency levels (η2=0.03).
Post-hoc tests reveal non-significant differences between adjacent proficiency levels and non-adjacent CEFR A2-B1.2.
Significant differences between CEFR B1.1-B2 and A2-B2 levels.Youn 2014 Significant positive correlation with proficiency level (r s=.33) and pragmatic
performance score (r s=.30).Zyad 2012* Increases over time. Longer clauses used by more proficiency writers.
Mean length of finite clause
Mean length of finite clause
Bulté & Housen 2014 Significant increase over a semester (Cohen’s d=.50). Significant positive correlation with overall essay score (r=.48) and
language use (r=.49).
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length
Table 7c: Generic indices : sentence-based measuresMeasure Notes on operationalisation Study FindingsMean length of sentence
# Words per sentence Bulté & Housen 2014
Significant increase over time (Cohen’s d=.44).
Significant positive correlation with overall essay score (r=.41) and language use score (r=.42).
Espada-Gustilo 2011a
Significant increase with proficiency level.
Homburg 1984 Significant increase from level 5 to level 6 but not from levels 6 to 7.
Hou, Verspoor & Loerts 2016
Significant increase over time for senior high school student group.
No significant change over time for the lower proficiency university group.
Significant decrease over time for higher proficiency university group
Ishikawa 1995 No significant change over time.Kawata 1992 No significant differences between
proficiency levels.Kim & Crossley
2018 Significant positive correlation
with source-based essays (rs =.14).
Non-significant positive correlation with independent essays (rs=.05).
Kim 2014 Significant increase with proficiency
level.Kobayashi & Rinnert 2013
Decreases over time.
Lahuerta-Martinez 2018
Significant positive correlation with holistic essay scores (r=.52).
Significantly longer sentences produced in year 4 than year 3 (p=<0.001).
Lorenzo & Rodriguez 2014
Non-significant increase from grade 9-10 (Hedge’s g=-.28).
Significant decrease from grade 10-11 (Hedge’s g=.-46).
Non-significant decrease from grade 11-12 (Hedge’s g=.-14).
Lu 2011 Significant increase from years: 1-2; 1-3; 1-4.
Paquot 2018 No significant differences across proficiency levels.
Shih & Ma 2012 Significant difference between grades 8-10 only.
Torras & Celaya 2001*
Increases over time.
Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Significant positive correlation with holistic essay score (r=.65).
Vidakovic & Barker 2010
Increases with proficiency level.
Yang, Lu & Weigle 2015
Significant correlation with writing quality for ‘appearance’ topic (r=.27)
and ‘future’ topic (r=.25).Yoon & Polio 2016 No significant increase over time for
both narrative and argumentative essays.
Yoon 2017 Significant differences between proficiency levels (η2=0.02).
t-units per sentence
# T-units per sentence Hou, Verspoor & Loerts 2016
No significant change over time reported for any of the groups.
Homburg 1984 No significant differences across proficiency levels.
Ishikawa 1995 No significant change over time for class 1. Significant increase for class 2.
Lu 2011 No significant difference across year groups.
Paquot 2018 No significant differences across proficiency levels.
Shih & Ma 2012 No significant differences between any of the grade levels.
Struc & Wood 2011
Non-significance increase over time for narrative and argumentative essays.
Yang, Lu & Weigle 2015
Non-significant positive correlation with writing quality for ‘appearance’ topic (r=.15) and ‘future’ topic (r=.11).
Yoon 2017 Non-significant differences between proficiency levels (η2=0.00).
Yoon & Polio 2016 No significant increases over time for narrative and argumentative essays.
clauses per sentence
# Clauses per orthographic sentence Struc & Wood 2011
Non-significant increase over time for narrative and argumentative essays.
SCA index Hou, Verspoor & Loerts 2016
No significant change over time for the senior high
school or lower proficiency university group.
Significant decrease over time for higher proficiency group.
Ishikawa 1995 No significant change over time for class 1. Significant increase for class 2.
Kim 2014 Significant increase with proficiency level.
Lorenzo & Rodriguez 2014
Non-significant increase from grade 9-10 (Hedge’s g=.19).
Non-significant decrease from grade 10-11 (Hedge’s g= -.26).
Non-significant increase from grade 11-12 (Hedge’s g=.28).
Lu 2011 Significant decrease from years: 1-4.Paquot 2018 No significant differences across
proficiency levels.Length of orthographic sentence
Mean length of orthographic sentence Struc & Wood 2011
Significant increase over time for narrative and argumentative essays.
Length of restricted sentence
Mean length of restricted sentence Struc & Wood 2011
Significant increase over time for narrative essays.
Non-significant increase over time for argumentative essays.
Simple sentences
# Simple sentences/sentences Struc & Wood 2011
Increase over time for narrative essays.Decrease over time for argumentative essays.
Bulté & Housen 2014
Significant decrease over a semester (Cohen’s d=.43).
Significant negative
correlation with overall essay score (r=-.43) and language use score (r=-.45).
Lahuerta-Martinez 2018
Non-significant negative correlation with holistic essay score (r=-.03).
Year 3 students produce non-significantly more simple sentences than year 4 students (p=0.93).
Complex sentences
# Complex sentences/sentences Struc & Wood 2011
Increase over time for narrative and argumentative essays
Bulté & Housen 2014
Non-significant increase over a semester.
Significant positive correlation with overall essay score (r=.21) and language use score (r=.21).
Lahuerta-Martinez 2018
Significant positive correlation with holistic essay scores (r=.42).
Significantly more complex sentences produced by year 4 students than year 3 students (p=<0.001).
# Complex sentences Cahyono, Mukminatien &
Amrina 2016
Significant positive correlation with writing quality scores (r=.32).
Compound sentences
# Compound sentences/sentences Struc & Wood 2011
Increase over time for narrative and argumentative essays
Bulté & Housen 2014
Significant increase over time (Cohen's d=.39).
No significant correlation with
overall essay score (r=.11) or language use score (r=.05).
Lahuerta-Martinez 2018
Significant positive correlation with holistic essay scores (r=.29).
Significantly more compound sentence produced by year 4 students than year 3 students (p=<0.001).
Compound-complex sentences
# Compound-complex sentences/sentences Struc & Wood 2011
Increase over time for narrative essays.
Decrease over time for argumentative essays.
Bulté & Housen 2014
Non-significant increase over a semester.
Non-significant correlation with overall essay score (r=.18).
Significant positive correlation with language use score (r=.24).
Lahuerta-Martinez 2018
Significant positive correlation with holistic essay scores (r=.22).
Year 4 students produce non-significantly more compound-complex sentences than year 3 students (p=0.16).
Sentence variety index
Variety index is a measure of sentence variety which ranges from 0-100 with 0 meaning sentences are all one type.
Struc & Wood 2011
Increase over time for narrative and argumentative essays.
Simple sentence
% Simple sentences over all sentence types. A simple sentence is taken to be a sentence with one main clause (including non-finite dependent
Verspoor, Lowie, Chan & Vahtrick
Non-significant positive correlation with holistic essay
ratio clauses) with subject and finite verb (2017, study 1) score (r=.35).Compound sentence ratio
% Compound sentences over all sentence types. A compound sentence is taken to be two main clauses each with its own subject and finite verbs
Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Significant negative correlation with holistic essay score (r=-.04).
Complex sentence ratio
% Complex sentences over all sentence types. A complex sentence is taken to be one main clause and one or more finite dependent clauses
Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Non-significant negative correlation with holistic essay score (r=-.20).
Compound complex ratio
% Compound complex sentences over all sentence types. A compound complex sentence is taken to be a sentence containing both subordinate and coordinate elements (each with a finite verb)
Verspoor, Lowie, Chan & Vahtrick (2017, study 1)
Non-significant positive correlation with holistic essay score (r=.09).
Sentence complexity measure from Botel, Darwins and Granowsky
Sentence complexity is rated with simple sentences scored as a 0; SVIODO, SVOC, prepositional modifiers added to 0-count structures, noun modifiers, adverbial modifiers and modals are scored as 1; passive sentences, paired constructions, comparatives, dependent clauses and conjunctive adverbs are scored as 2; clauses used as subjects and absolute constructions are scored as 3. The final sentence index is the averaged score of all sentences.
Perkins 1980 Non-significant differences between proficiency levels.
Significant positive correlation with writing score (r=.44).
Sentences containing dependent clauses
# Sentences containing a dependent clause Verspoor, Schmid & Xu 2012a
Significant positive correlation with essay score (r=.51).
Number of nodes per sentence
# nodes per sentence (manual calculation) Torras & Celaya 2001*
Increases over time.
*denotes a study that has no inferential statistics a = denotes a study where features are not explicitly normalised for text length