Variations in L2 spoken and written English: investigating patterns of grammatical errors across proficiency levels

Mariko Abe, Yukio TonoTakasaki City University of Economics, Meikai University

mabe@tcue.ac.jp, y.tono@meikai.ac.jp

1. Purpose

The purpose of this research is to investigate the variability of interlanguage by means of a corpus-based quantitative analysis. Since the previous studies on language acquisition have focused on a relatively limited amount of data, studies using a large amount of learner data may be able to make a significant contribution to the field of L2 acquisition (Biber, Conrad and Reppen 1998: 180). The spoken and written data of Japanese learners of English was especially created to investigate the difference between spontaneous production in speaking and written production with less time pressure. There is a substantial body of research showing that the processing modes of learners affect their performance in L2. Therefore, it would be essential to show the acquisition sequence of certain grammatical features on both spoken and written data. This paper also aims to describe the developmental patterns of grammatical features in learner English by analysing learners’ use and misuse across different proficiency levels.

2. Method2.1. Learner corpora data

Two types of learner data were compared in this study; the NICT JLE Corpus, a corpus of more than 1,200 Japanese EFL learners’ oral interview transcripts and the JEFLL (Japanese EFL Learner) Corpus, a corpus of written essays of more than 10,000 Japanese secondary school students.

The spoken data were extracted from the NICT JLE Corpus. The data consists of 1,200 examinees who have taken Standard Speaking Test (SST), and the test has 9 different levels to assess speaking proficiency. The spoken data used in this research came from 100 examinees whose proficiency level was assessed at SST level 2 to level 9. Since the size of this sub-corpus is rather small for detailed study (as shown in the following table), we hope to increase the size in future studies. There are five different tasks in this speaking test, but only one of them, single picture description stage, was used in the current study. A picture is chosen from five different pictures, and the examinees were asked to describe it in two or three minutes. Therefore the spoken data of this research does not consist of conversation or speech, but of story retelling production.

SSTLEVEL SST2/3 SST4 SST5 SST6 SST7 SST8/9 total

File 22 17 16 19 16 10 100Token 1222 1418 1755 1891 2004 1370 9660

Table 1 Corpus size of spoken data

1

The written data were extracted from the JEFLL Corpus. The data of this learner corpus consist of five different composition topics written by learners of six different academic years at several junior and senior high school in Japan. In this paper, the written composition produced by one particular junior and senior high school was chosen, and only the composition entitled “My school festival” was selected.

ACADEMIC YEAR

Junior1(J1)

Junior2(J2)

Junior3(J3)

Senior1(S1)

Senior2(S2)

Senior3(S3) TOTAL

File 104 77 87 46 53 55 422Token 4994 5004 5000 4997 5000 5005 30000

Average token 48.02 64.99 57.47 108.63 94.34 91 71.09

TTR 10.79 15.91 14.62 18.41 17.02 19.02 -

Table 2 The corpus size of the written data

We will also demonstrate the software for processing these two learner corpora later in the presentation. The Shogakukan Corpus Network (SCN) provides the free service for JEFLL, while the NICT JLE Corpus has its own specialized retrieval software.

2.2. Data processing

An equal amount of data were sampled and manually error-tagged, and also POS-tagged by CLAWS7. Major error categories related to (1) parts of speech, and (2) tense and aspect were identified and their normalised frequencies were compared in terms of error types and modes of production, speech and writing, together with the subjects’ proficiency levels in L2.

All of the spoken data were already error-tagged and published as the NICT JLE Corpus, therefore written data of JEFLL corpus were error-tagged manually by referring to the same Error Tagging guide (Isahara, Saiga, and Izumi 2002) as used for the NICT JLE Corpus. The operational error tags such as redundancy, omission, and misordering were excluded in this research but by adding the correction within the error tag, we can still retrieve the operational errors from the concordance lines. There are 6 types of noun-related tagset and 11 types of verb-related tag set provided in the NICT JLE Corpus, but only the errors shown in the following table are considered in this paper. This is because some of the errors did not have a high enough frequency to be worth analysing. Part-of-speech (POS) tags were also provided by the CLAWS tagger by using C7 tagset of Garside, Leech & McEnery (1997) in order to calculate the accuracy rate in each category.

In addition to this, vocabulary or phrases that examinees were not able to produce in English were indicated by a special “<jp>” tag. By analysing this tag, we can determine the words and phrases that learners avoided or failed to produce in English. As the data of written composition was all hand-written by the learners, numerous spelling mistakes were found in written production. The spelling errors were not considered in this paper, because a tag for misspelling was not included in the tag set of NICT JLE Corpus. Therefore, it was crucial to make a correction to the output to POS tagging to avoid skewing the data on accuracy rates.

2

CATEGORY TAG EXAMPLESNoun Inflection <n_inf> *childerens / *peoples / *girls’s

Agreement <n_agr> many *book / one *thingsCountability <n_cnt> *a music / *an informationCase <n_cs> my *friend houseLexical errors <n_lxc> a *type→a typewriter

Verb Inflection <v_inf> *sleeped / *makedAgreement <v_agr> There *are a lady / a cat *sleep in her bedForm (only spoken data) <v_fm> to *drinks / is *sleepTense <v_tns> He *has jogged now. 　Aspect (only written data) <v_asp> They *are knowingLexical errors <v_lxc> She *is black and short hair.

others Japanese words <jp> anmitsu / origami / yukata

Table 3 Error tag set of noun and verb

As the size of spoken sub-corpus was different for different levels, the frequency of erroneously and correctly used nouns and verbs were normalised considering the sub-corpus size of each proficiency level and the total frequency of nouns and verbs. The size of written sub-corpus was almost same, since the composition files were randomly extracted from each academic year to equalise the corpus size; therefore, normalisation was only conducted by the total number of nouns and verbs.

3. Results and Discussion 3.1. Error of parts of speech: noun and verb

In this section on the results of the data analysis, we tackle such questions as whether particular groups of errors are prone to occur at certain developmental stages, and if so, why that happened. Error categories related to part of speech, especially noun and verb, were identified and their normalised frequencies were compared in terms of error types and modes of production, speech and writing, together with the subjects’ proficiency levels in L2.

The following figure in the research of Abe (2004) indicates the proficiency levels of learners and noun and verb-related errors of spoken data extracted from NICT JLE Corpus. The learners of SST level 2 and 3 seem to have a strong correlation with verb-related error categories “v_agr” and “v_tns”, the SST4 to 5 level learners are connected with various types of error categories, and the SST6 to 9 level learners can be linked with noun-related error categories such as “n_agr”, “n_cnt”, “n_lxc”, and “n_num”. The research, which has been conducted using the same spoken data as in this paper, shows that lower proficiency level learners have firm connection with verb-related errors and advanced level learners with noun-related errors.

3

行ﾎﾟｲﾝﾄと列ﾎﾟｲﾝﾄ対称的正規化

1次元

1.0.50.0-.5-1.0-1.5

2次

元

1.0

.5

0.0

-.5

-1.0

-1.5

VAR00001

VAR00002

sst8/9sst7

sst6

sst5sst4

sst2/3v_lxcv_finv_mov_ngv_vo

v_infv_cmp v_tnsv_fm v_agr

n_lxc

n_dprpn_gen

n_inf

n_num

n_cnt

n_agr

Figure 1 The proficiency levels and errors (Abe, 2004)

The question then is: can this result of previous research be applied to written production in this paper to describe the language acquisition stages? In the following table, the distinctions of academic years and SST proficiency levels are not identical, but they are simply allocated in the same column for the sake of convenience.

J1 J2 J3 S1 S2 S3SST2/3 SST4 SST5 SST6 SST7 SST8/9

wr_noun 5.38 4.47 3.59 4.55 3.35 3.85wr_verb 32.07 10.03 8.52 7.50 5.70 5.22sp_noun 0.92 3.96 2.66 3.43 2.37 1.37sp_verb 12.11 5.36 5.12 1.68 2.00 1.31

Table 4 Error rate of nouns and verbs in each corpus (%)

The above table shows that the verb-related errors in spoken data are strongly connected with novice learners, and these errors gradually decrease as the proficiency levels increase. This tendency of verb-related errors in spoken mode is identical with writing production. The error rate of verb is strongly linked with the 1st grade of junior high school (J1) learners, and it gradually decreases as the developmental stages progress. However, when we focus on the error rate of nouns, it shows a dissimilar tendency. The error rate of nouns in written data is relatively low compared with that of verb-related errors, which suggests the possibility that the acquisition of verbs is much more difficult than that of nouns.

However, the error rate of nouns in written production is almost identical from the academic year of J1 to S3, and this also implies that the noun-related errors do not easily vanish during the developmental stages of written English. Concerning the spoken language, the error rate of nouns shows unpredictable changes, which are difficult to explain. But one hypothesis can be put forward. It is that the novice learners of English do not consciously avoid using nouns to prevent themselves from

4

Proficiency level

making errors, but they only have a minute stock of noun and repeatedly use them. This can be gathered from the data where the total number of nouns is similar (by per thousand words: J1=169.39, J2=157.26, J3=157.26, S1=160.76, S2=163.17, S3=167.15). It is essential to examine more detailed data to confirm that noun-related errors have a strong relation with upper level learners, but we may conclude that the verb-related errors have the same firm connection with novice learners’ written production as in their spoken data.

The next step is to examine which particular grammatical category of noun and verb has a strong connection with spoken and written modes. The following pie charts show the accuracy rate and error rate for each mode. Japanese words, preceding noun of title (e.g. Mr. Ms.), and all of the proper nouns were excluded from the total number of nouns.

5

6

Figure 2 Accuracy and error rate of noun in different modes

As can be seen from the pie charts, the accuracy rate of “n_cnt” (countability), “n_lxc” (lexical error), and “n_agr” (agreement) is similar in the two different modes (countability: WR=32%, SP=33%, lexical error: WR=31%, SP=33%, and agreement: WR=29%, SP=32%). However, nouns associated with “n_inf” (inflection) and “n_cs” (case) are more frequently used in written mode than in spoken mode (inflection: WR=6%, SP=2%, case: WR=2%, SP=0%). One reason for this is the difference in the tasks between the two different corpora. Since the spoken test task was to describe a simple picture so that its vocabulary size might be automatically limited to a small range, there was not so much need to use the plural and possessive form of nouns and the different case forms. As a result, the use of singular nouns is enormously high in spoken data, and makes up 81.73% of the total number of nouns.

Another interesting result is that the rate of lexical error in written and spoken corpus is surprisingly almost identical as the accuracy rate was similar in both sets of data (WR=17%, SP=18%). Therefore, we can suppose that erroneous and accurate use of lexical nouns cannot be an indicator of the difference in production modes. In addition, on the one hand, inflection is used much more in the written mode than in spoken mode (WR=6%, SP=2%) as mentioned before, but, on the other hand, it is misused more frequently in spoken data (WR=2%, SP=5%). Similarly, another noun-related grammatical category “n_cnt” (countability) is misused more regularly in spoken corpus when we compare it with the written corpus (WR=3%, SP=21%). These two categories, inflection and countability, will require a mechanical change of form for the speakers of English, as well as knowledge of vocabulary. Accordingly, we can suppose that speakers will be more likely to make errors in more time-pressured spoken production, and the error rate of these grammatical categories are strongly connected with spoken production.

3.2. Morphological error: agreement

When we compared the accuracy and error rate of the two modes, the accuracy rate of “n_agr” (agreement) was almost identical in both modes (WR=29%, SP=32%), but its error rate in written data was higher than that of spoken data (WR=69%, SP=54%). Since there is no agreement rule in Japanese grammar, it can be easily assumed that agreement must be a troublesome grammar point for Japanese learners of English, as it is indicated in the high error rate in both modes. However, the error rate of noun agreement changes slightly in the developmental stage of written production. It is mostly misused in J1 and its error rate mostly decreases in J3, but for the most part there is not a large difference. For that reason, we can suppose that noun-related agreement errors cannot be a strong indicator of the developmental sequence in written production.

Noun agreement errors may increase as modifiers, an ordinal number or a quantifier, are added and noun phrases become complicated, but as can be seen from the following bar graph they seem to be especially connected with SST level 6 in spoken production in some way. Agreement error increases mostly at SST level 6 and continues decreasing by SST level 8/9. As mentioned previously, noun-related lexical error and accuracy rate in each mode was almost identical, but in the next bar graph of spoken data it shows an irregular change in the developmental sequence as in agreement error. These irregular

sequences cannot be clearly explained, but they expose the variability in development of learner English. We cannot observe the same sequence in written mode, so that it might be caused by the difference of production modes, or by the lack of competency of learners in written data, because the error rate of agreement does not decrease below the lowest error rate of J3.

Figure 3 Noun- related errors in both production modes (%)

In the next stage, the focus will be shifted to the accuracy and error rate of verbs. First of all, it must be said that the verb-related error tags used for written and for spoken corpora in this research paper are slightly different, and they must be modified in further studies. In considering errors of inflection, past and past participle form of be-verbs were excluded, because an inflection error of be-verbs rarely occurred, even in J1, the first year of studying English in Japanese language education.

The following pie charts show the accuracy and error rate of verbs in each different production mode. The first thing to point out is that the accuracy rate of “v_lxc” (lexical error), “v_tns” (tense), and “v_agr” (agreement) is similar in the two different modes (lexical error: WR=37%, SP=33%, tense: WR=28%, SP=30%, and agreement: WR=21%, SP=26%). In spite of this similarity, the error rate of these grammatical categories shows dissimilarities. In spoken mode, verb agreement is much more misused than in written production (WR=7%, SP=60%), but tense and lexical errors are more frequently observed in written corpus data (tense: WR=65%, SP=18%, lexical error: WR=22%, SP=9%). As a result, it is clear that a verb agreement error has a strong connection with spoken production, and a tense and lexical error have a firm relation with written production. Another interesting point is that both accuracy and error rate of verb-related inflection error is fairly low in speaking mode, but its accuracy rate is relatively high in

writing mode. Therefore, in this study, it appear that the verb-related inflection error is peculiar to written production.

Figure 4 Accuracy and error rate of verb in different modes (%)

As mentioned above, verb agreement error can be regarded as an effective indicator of novice learners, because it decreases as the proficiency level increases. However, the error rate does not decrease at SST level 6 and SST level 8/9, and interestingly this shows the same pattern as in the noun agreement error of spoken production. As can be seen from the following bar graph, the verb agreement error shows an irregular change at the

level of SST6 and SST8/9, while the error in verbal form increases and the verbal lexical error decreases. Both noun and verb-related agreement errors in spoken data show a complicated pattern, which is difficult to explain, but it is interesting to discover that level 6 and level 8/9 can be some kind of turning point for errors especially related with agreement.

3.4. Errors of tense and aspect

What can we find out when we focus on the accuracy and error rate of verbs in written corpus data? Firstly, as it can be observed from the pie charts in the previous section, written production is strongly connected with tense errors. Although the error rate does not decrease drastically, except from J1 to J2, it keeps to a fairly high level throughout the academic year. This implies that tense is a problematical grammar point for Japanese learners of English, and it does not easily disappear in writing, even for the upper level of learners. Subsequently, we can conclude that it cannot be an index to identify the proficiency level of written production, but can be an indicator to distinguish the mode.

As mentioned above, tense errors seem to be peculiar to novice learners in written data. What, then, can we discover from the spoken data? The error frequency starts to increase at level 4 and level 6, but then decreases at higher proficiency levels. The errors of simple present and simple past were good indicators of progress in simple tense usage, however, a picture description task used in this study turned out to be unsuitable for observing various complicated tense and aspect forms that might be used by learners at much more advanced levels.

Figure 5: Verb-related errors in both production modes (%)

Regarding aspect errors, learners in academic year of J1 tend to use a progressive form inappropriately. In J2 level, the error rate increases, but what is more noteworthy here is

that same four learners in written corpus have misused aspect in their composition more than 3 times. This phenomenon might occur because learners have difficulty in using the same tense and appropriate aspect while they are writing a sequence of English paragraphs. A significant educational implication that must be concluded from this fact is that learners need to practice using tense and aspect by writing longer English compositions.

Another thing that must be pointed out here is that an error rate of aspect suddenly drops at the J3 level, but once again it is successively misused as the developmental stage increases. This trend cannot be clearly explained here, but it may be strongly connected with an introduction of perfect tense in English grammar textbook in the upper academic year. An error of perfect aspect was especially observed in the upper grade when it is introduced in English classes at school. Finally, learners seem to have a tendency to use the progressive and perfect tense when there is no necessity, but they rarely use the simple tense when they need to use the progressive and perfect tense.

4. Conclusion

Through the detailed examination on error categories, we were able to observe that some errors have common developmental patterns and a firm correlation with certain proficiency levels and production mode. The study of learners’ errors showed how the errors varied according to their stages of language acquisition and the production mode. And at the same time, there were other categories that can be described not by the pattern of error rate, but by the accuracy rate. By analysing the errors and proficiency level of two different production modes, this study has arrived to identify the features of interlanguage in a more objective way, and also was able to show the value of utilising learner corpus for SLA research.

It is necessary for future study to enlarge the size of learner corpus and to increase the variety of tasks used to elicit the data. Also, as a statistical analysis, the multivariate statistical analysis, called Correspondence Analysis, can be performed on this data to identify how learner errors change as levels of proficiency increase. Such an analysis may be able to clarify the correlation of proficiency level and the error frequency of each production mode. Additionally, larger corpus can represent several features of interlanguage, but still we need further consideration on error tags. The focus of this study was agreement and tense/aspect errors, but there is obviously scope for similar investigations that examine the relationship between proficiency level and other features of learner language. Also, it would be useful to compare learners’ production with a native speaker corpus to examine the difference of modes and the points that learners are avoiding on the process of language acquisition.

ReferencesAbe, M. (2004) A Corpus-based Analysis of Interlanguage: Errors and English

proficiency Level of Japanese Learners of English. Handbook of an International Symposium on Learner Corpora in Asia (ISLCA), 28-32.

Bardovi-Harlig, K. and Bofman, T. (1989) Attainment of syntactic and morphological accuracy by advanced language learners. Studies in Second Language Acquisition 11, 17-34.

Biber, D., Conrad, S. and Reppen, R. (1998) Corpus Linguistics: Investigating language Structure and Use (Cambridge: Cambridge University Press).

Chamot, A. (1978) Grammatical problems in learning English as a third language, in E. Hatch (ed.) Second Language Acquisition (Rowley, Mass.: Newbury House), 175-189.

Chamot, A. (1979) Strategies in the acquisition of English structures by a child bilingual in Spanish and French, in R. Andersen (ed.) The acquisition and Use of Spanish and English as First and Second Languages (Washington D.C.: TESOL), 90-106.

Corder, S. P. (1967) The significance of learners’ errors. International Review of Applied Linguistics 5, 161-169.

Dagneaux, E., Denness, S. and Granger, S. (1998) Computer-aided Error Analysis. System An International Journal of Educational Technology and Applied Linguistics 26 (2), 163-174.

Ellis, R. (1994) The Study of Second Language Acquisition (Oxford: Oxford University Press).

Ellis, R. and Barkhuizen, G. (2005) Analysing Learner Language (Oxford: Oxford University Press).

Garside, R., Leech, G and McEnery, T (eds.) (1997) Corpus Annotation: Linguistic Information from Computer Text Corpora (Harlow: Longman).

Granger, S. (1999) Use of Tenses by Advanced EFL learners: Evidence form an Error-tagged Computer Corpus, in Hasselgard, H and Oksefjell, S (eds.) Out of Corpora: Studies in Honour of Stig Johansson (Amsterdam: Rodopi), 191-202.

Granger, S. and Rayson, P. (1998) Automatic profiling of learner texts, in Granger, S (ed.) Learner English on computer (London: Longman), 191-202.

Isahara, H., Saiga, T and Izumi, E. (2002) The TAO Speech Corpus of Japanese Learner English Error Tagging Manual Ver.1.0.

Izumi, E., Uchimoto, K. and Isahara, H. (2004) A Speaking Corpus of 1200 Japanese Learners of English (Tokyo: ALC Press Inc.)

Housen, A. (2002) A corpus-based study of the L2-acquisition of the English verb system, in Granger, S., Hung, J. and Petch-Tyson, S (eds.) Computer Learner Corpora, Second Language Acquisition and Foreign Language Teaching (Amsterdam, Benjamins), 77-116.

James, C. (1998) Errors in Language Learning and Use: Exploring Error Analysis (Harlow: Longman).

Kathleen Bardovi-harlig. (2000) Tense and Aspect in Second Language Acquisition: Form, Meaning, and Use (Malden: Blackwell).

Lennon, P. (1991) Error: Some problem of definition, identification and distinction. Applied Linguistics 12, 180-95.

Tabata, T. (2002) Investigating Stylistic Variation in Dickens through Correspondence Analysis of Word-Class Distribution, in Saito, T., Nakamura, J and Yamazaki, S (eds.) English Corpus Linguistics in Japan (Amsterdam: Rodopi), 165-182.

Tono, Y. (1999) A Corpus-based analysis of interlanguage development: Analysing part-of-speech tag sequences of EFL learner corpora, in Lewandowska-Tomaszczyk, B. and

Melia, P.J (eds.) PALC’99: Practical Applications in Language Corpora (Frankfurt: Peterlang), 323-340.