The CBAL Writing Assessment Project Paul Deane Nora ...
Transcript of The CBAL Writing Assessment Project Paul Deane Nora ...
The CBAL Writing Assessment Project
Paul Deane
Nora Odendahl
Thomas Quinlan
Mary Fowles
Doug Baldwin
ETS, Princeton, NJ
Paper presented at the annual meeting of the
American Educational Research Association (AERA) and the
National Council on Measurement in Education (NCME)
held between March 23 to 28, 2008, in New York.
Unpublished Work Copyright © 2008 by Educational Testing Service. All Rights Reserved. These materials are an
unpublished, proprietary work of ETS. Any limited distribution shall not constitute publication. This work may not
be reproduced or distributed to third parties without ETS's prior written consent. Submit all requests through
www.ets.org/legal/index.html.
Educational Testing Service, ETS, the ETS logo, and Listening. Learning. Leading. are registered trademarks of
Educational Testing Service (ETS).
i
Abstract
The overarching goal of the CBAL Writing project is to design assessment that enhances
instruction, while being grounded in a cognitive theory of writing competency. There is a
rich cognitive literature on the nature of writing expertise and the difference between
novice and skilled writers. Bereiter and Scardamalia (1987) observed that novice writers
typically adopt a ‘knowledge-telling’ approach, while more skillful writers sometimes
deploy a ‘knowledge-transforming’ approach. This ability to adopt a knowledge-
transforming approach depends upon an array of writing subskills. We developed a
‘competency model’ to explicitly identify these subskills, which fall into three broad
categories: (a) basic language and literacy skills; (b) strategically manage writing
processes; and (c) think critically for writing. In designing an assessment to measure
these areas of writing competence, we wanted to capture something of the complex
coordination of writing subskills, while also taking into account the role of other relevant
factors, such as background knowledge. To emphasize critical thinking, we opted for a
project-based approach, in which smaller writing tasks (e.g., notes, a summary) serve to
scaffold a larger task (e.g., a letter to the editor). Preliminary responses from teachers
have been quite favorable, and the preliminary analysis of data from our first pilot
suggests that this approach has promise.
1
Introduction
There is a longstanding tension between writing assessment and writing
instruction. Writing assessment is constrained by the need to establish score reliability
and predictive power (Elliott, 2005) within limits determined by time availability and
cost of administration, and is constrained by concerns that writing tests should be fair and
not unduly influenced by outside knowledge. Writing instruction, on the other hand, is
motivated by a concern with the complexities of producing texts in a variety of contexts,
for a variety of purposes, and under conditions that require strategic management of a
variety of intellectual and linguistic skills (Bazerman, 2008; MacArthur, Graham &
Fitzgerald, 2006). The constraints of the one do not always match the concerns of the
other, which can have unfortunate and negative consequences for both (Hillocks, 2002).
The goal of the work reported here is to approach the problem of writing assessment in a
manner that conforms to the recommendations of the NRC Committee on the
Foundations of Assessment (Pellegrino et al., 2001: 292-293), who argue that a sustained
effort must be made to coordinate instruction and assessment, and to ground both in a
cognitive theory of domain area expertise. This paper presents initial results from two
years of designing and piloting such an assessment as part of CBAL (‘Cognitively-Based
Assessments of, for and as Learning’), an internal ETS research initiative.
This initiative is designed to conduct research to support the creation of a future
system of assessment that
• Documents what students have achieved (“of learning”),
• Helps identify what instruction should occur next (“for learning”),
2
• Is considered by students and teachers to be a worthwhile educational
experience in and of itself (“as learning”).
It seeks to build a unified approach to accountability assessment, formative
assessment, and professional support, in line with the following principles:
• Accountability tests, formative assessments, and professional support should (i)
be based on a single integrated framework motivated by cognitive research, (ii)
be responsive to state instructional and curricular standards, and (iii) be
designed to support best practices in writing instruction while maintaining
rigorous psychometric standards.
• Assessments should consist largely of engaging, extended, constructed-response
tasks, delivered by computer and automatically scored where appropriate.
• Individual tasks should be viewed by teachers and students as worthwhile
learning experiences in their own right, resulting in positive washback, in which
test preparation becomes an appropriate learning experience.
• Accountability assessment should be distributed over multiple administrations
(i) to reduce the importance of any one assessment and testing occasion; (ii) to
provide time for complex integrated tasks that better assess the construct; and
(iii) to provide prompt interim information in support of instruction.
• Assessments should help students participate actively in their own learning.
This paper describes work done to date on the writing portion of the CBAL
initiative, focusing on initial work developing the conceptual design, but with brief
discussion of initial pilot results, and focusing almost exclusively on summative
3
assessment design. We will touch briefly on formative issues, but these will primarily be
addressed by another presentation in this symposium.
4
What the Cognitive Literature Teaches Us about Learning to Write
There is a rich cognitive literature on the nature of writing expertise and the
difference between novice and skilled writers. In particular, skilled writers spend more
time planning and revising their work than novice writers; they focus more of their effort
and attention on managing the development of content, and concern themselves less with
its formal, surface characteristics; and they employ a variety of self-regulatory strategies.
(Bereiter & Scardamalia, 1987; Galbraith, 1999; Graham, 1997; Graham & Harris, 2000;
Kellogg, 1988; McCutchen, Francis, and Kerr, 1997; McCutchen, 2000). Moreover,
novice writers benefit from instruction on planning and revision strategies. They also
benefit when provided with instruction that enables them to think critically about topic-
relevant content (De La Paz, 2005; De La Paz & Graham, 1997a, 1997b, 2002; Graham
& Perin, 2006; Hillocks, 1987; Kellogg, 1988; Quinlan, 2004). Bereiter and Scardamalia
(1987) characterize the difference between novice and skilled authors as the difference
between a ‘knowledge-telling’ approach and a ‘knowledge-transforming’ approach to
writing. In a knowledge-telling approach, the focus of the writer’s effort is on the
process of putting words on the page. In a knowledge-transforming approach, writing is
a recursive process of knowledge-development and knowledge-expression.
Knowledge-transforming is by its nature a much more effortful and sophisticated
process than knowledge-telling, and develops only as writers gain significant expertise.
The literature suggests five major reasons why a student may fail to deploy a knowledge-
transforming approach to writing: (i) undeveloped or inefficient literacy skills; (ii) lack of
strategic writing skills; (iii) insufficient topic-specific knowledge; (iv) weak content
reasoning and research skills, and (v) unformed or rudimentary rhetorical goals
5
Undeveloped or Inefficient Literacy Skills
The high-level, strategic skills required for a knowledge-transforming approach to
writing place heavy demands on memory and attention. Inefficient oral fluency,
transcription, and text decoding may render it impossible to free up the working memory
capacity needed for strategic thought (Bourdin & Fayol, 1994, 2000; Kellogg, 2001;
Olive & Kellogg, 2002; Pearl, 1979; Piolat, Roussey, Olive, & Farioli, 1996; Shanahan,
2006; Torrance & Galbraith, 2005). Similarly, the ability to monitor and reflect upon
one’s own writing, which is critical to planning and revision, depends in large part upon
aspects of reading skill, both decoding and higher verbal comprehension, and thus
reading difficulties can obstruct revision and planning (Hayes, 1996, 2004; Kaufer,
Hayes, & Flower, 1986; McCutchen et al., 1997).
Lack of Strategic Writing Skills
Even skilled writers cannot handle all aspects of complex writing tasks
simultaneously; they succeed by applying strategies that reduce each task into
manageable pieces. A significant element in writing skill is thus the ability to
strategically and efficiently intersperse planning, text production, and evaluation,
sometimes switching back and forth rapidly among tasks, and other times, devoting
significant blocks of time to a single activity (Matsuhashi, 1981; Schilperoord, 2002).
Controlling writing processes so that the choice of activities is strategically appropriate
and maximally efficient is itself a skill to be learned (see Coirier, Anderson & Chanquoy,
1999 with respect specifically to persuasive writing expertise).
6
Insufficient Topic-Specific Knowledge
All models of writing expertise presuppose a critical role for long-term memory in
which the subject matter of writing must be retrieved, either top-down (Bereiter &
Scardamalia, 1987; Hayes & Flower, 1980) or bottom-up (Galbraith, 1999; Galbraith &
Torrance, 1999). Prior topical knowledge gives writers a major advantage, not only in
generating content, but in pursuing critical thinking tasks where topic knowledge is
required to support judgments of relevance and plausibility. Thus it is not surprising that
topic knowledge is a major predictor of writing quality (DeGroff, 1987; Langer, 1985;
McCutchen, 1986).
Weak Content Reasoning and Research Skills
Academic writing should be viewed not merely as an expressive act, but also as
intrinsically involving critical thought. While many of the problems the writer faces are
rhetorical, involving audience and purpose, these goals typically require the author to
develop ideas, to identify information needs, and to obtain that information, whether by
observation, inference, argument, or research (see Hillocks, 1987 meta-analysis, which
indicated the critical importance of inquiry strategies to improve student writing, and the
related arguments in Hillocks, 1995).
The reasoning required successfully to complete a writing task varies with
purpose, audience, and genre. Overall the literature suggests that one cannot assume that
novice writers, or even all adults, possess the full range of content reasoning and research
skills needed to support a knowledge-transforming approach to writing (Felton & Kuhn,
2001; Kuhn, 1991; Kuhn, Katz & Dean, 2004; Means & Voss, 1996; Perkins, 1985;
Perkins, Allen, & Hafner, 1983).
7
Unformed or Rudimentary Rhetorical Goals
A sophisticated writer is aware that all writing is communication within a social
context, in which the author must take the audience into account, collaborate with others,
and more generally, act within one or more communities of practice with well-defined
social expectations about writing. Students benefit from instructional activity that
clarifies the intended audience and the writer’s obligations to that audience (Cohen &
Riel, 1989; Daiute, 1986; Daiute & Dalton, 1993; Yarrow & Topping, 2001). In
particular, social, interactive activities such as peer review have a strong beneficial
impact (Graham & Perin, 2006), particularly when writing instruction is structured to
enable students to internalize social norms about academic writing (Beaufort, 2000;
Flower, 1989; Kent, 2003, Kostouli, 2005).
The Competency Model
We seek to implement the principles of Evidence-Centered Design (Mislevy et
al., 2003) in which assessment design is driven by the construction of explicit evidentiary
arguments. As part of the design process entailed by this approach, we developed a
‘competency model’ which explicitly identifies what skills a writing assessment should
measure. According to this model, there are three basic strands of writing competence:
Strand I: Language and literacy skills for writing.
Strand II: Writing process management skills
Strand III: Critical thinking skills for writing
Strand I is concerned with being able to use Standard English, being able to use
basic literacy skills such as reading in support of writing, and most centrally, being able
to draft and edit text. Strand II is concerned with being able to manage writing processes
8
strategically to produce an effective document, and thus is concerned with document
planning, selection and organization of materials, and text and content evaluation.
Each strand can be broken down further. The following diagram presents the
major skills we isolate as potentially relevant to writing. Note that though we present a
hierarchical diagram, with each node having only a single parent, the model does not
presuppose a strict hierarchy of componential skills; almost every part of writing
intrinsically interacts, and our primary goal in laying forth the hierarchy shown in Figure
1 is to present a reasonably complete outline of the kinds of skills required to succeed as
a writer, and to set as the goal of assessment measurement of these skills.
The nodes in Figure 1 are what Evidence-Centered Design refers to as “student
model variables” or “competency model variables.” Such variables have a dual nature by
design: on the one hand, they represent aspects of student writing competency; on the
other, they are intended to be operationalized in a measurement model, and must
therefore be explicitly connected to measurable features.
It is critical to interpret these three strands as jointly describing a model of skills
that need to be assessed to measure writing expertise viewed as including knowledge-
transforming, not just knowledge-telling strategies. The various competencies described
above form part of an interacting and interwoven complex of skills that cannot easily be
separated from one another or tested in isolation, and yet measuring writing skill is in
large part measuring progress in the development and integration of these intellectual and
social abilities.
9
Figure 1. The CBAL Writing Competency Model
WRITE
Use Language and Literacy
Skills
Use Strategies to Manage the
Writing Process
Use Critical Thinking Skills for Writing
Reason Critically
about Content
Reason about
Social Context
(Purpose, Audience)
Plan/Model
Control/Focus
Document Structure
Evaluate/Reflect
Activate/Retrieve
Select/Organize
Assess/Critique
Edit/Revise
Collaborate/Review
Accommodate/Engage
Narrate/Describe
Explain/Hypothesize
Gather/Synthesize
Support/Refute
Draft
Proofread/
Correct
Compose
(Express/Clarify)
Transpose
(Spelling, Mechanics)
Inscribe
(Handwriting,
Use Written
Vocabulary
Control
Sentence
Structure
Use Written
Style
Read/
Decode
Speak/
Understand
10
Strategies for Test Design
The literature review and competency model have led us to emphasize the
following key ideas:
• Writing proficiency is a complex array of skills
• These skills notably involve critical thinking about content, audience, and
purpose.
• Background knowledge plays a central role.
Each of these considerations has had a major impact on CBAL test design:
Complexity. The test design follows the CBAL principle of giving periodic
assessments throughout the year, but each prototype test focuses on a different mix of
writing proficiencies. The richness and complexity of the writing construct entails that
we can only effectively measure writing skill if we provide multiple occasions for
writing, using a wide range of writing tasks and situations.
Critical thinking. The prototype design addresses the importance and variety of
critical-thinking skills by devising smaller tasks to measure these skills separately. Many
critical thinking tasks occur naturally as activities people have to do as part of preparing
for, writing, and revising a long piece of writing, so that they can naturally be presented
as subsidiary activities within a larger writing project.
Background knowledge. Rather than develop generic prompts that minimize the
role of background knowledge, we adopted an approach in which relevant resource
materials are provided, so that all students have available a rich array of useful
information to support effective writing. While such an approach is not unprecedented,
11
we chose to make it the centerpiece of our approach because it enables tasks to be
designed whose instructional relevance is immediate and obvious.
Scaffolding. Rather than relying on a single prompt, we seek to provide materials
and activities that provide initial guidance to students and enable them to display their
writing skills to best advantage. Structuring tests to provide such scaffolding natural led
to a project-based approach in which all tasks hang together conceptually as parts of a
larger writing project.
In the resulting test design,
• Each test presents a multi-part “project” and is structured around a
scenario or situation that provides a context and purpose for a series of
related tasks.
• Tasks focus explicitly on writing strategies and critical thinking and are
explicitly scored for quality of thinking, not just for generic writing quality
• Each test typically focuses on one genre or mode of discourse and the
critical thinking/writing strategies associated with that mode of discourse,
but not rigidly; any of the activities naturally associated with a coherent
project may appear together in a single assessment.
• Language and literacy skills (vocabulary, grammar, usage, and mechanics)
are not measured by separate items, but are scored ‘in the background’ as
a feature of each written response.
• Usually two or three short tasks precede a longer, more integrated writing
task; sometimes, a short follow-up task comes after the long task, with
12
shorter tasks designed to model appropriate steps in the research and
thinking processes for a larger writing project.
• Source materials, rubrics, guidelines, and other supporting documents are
systematically built into the test design, reflecting the reality that academic
writing does not take place in a vacuum. This approach also permits
assessment of research skills crucial to academic writing.
Essentially, these strategies create “thematic” assessments, in which all the tasks
in a single assessment draw from the same topic and take place in the same context, so
that they hang together as different (but separable and psychometrically distinct) tasks
within a single larger-scale writing project.
Initial Pilots
In order to examine how well these concepts worked out in practice, nearly a
dozen possible writing assessments were developed, ranging over a variety of genres and
modes of writing. Two of these tests were selected for administration in collaboration
with middle school teachers from an urban/suburban school district. A battery of
formative tasks was also developed in cooperation with teachers, who tried them out in
the classroom. These tasks mirrored the summative tasks and enabled students to gain
some familiarity with test formats beforehand; teacher reaction was strongly favorable,
supporting the conclusion that we had designed tasks with clear instructional value. We
are still analyzing the results of these pilots, and thus the information presented in this
paper should be considered tentative. We will be administering additional pilot tests in
an expanded testing program in 2008.
13
The pilot materials focused primarily on persuasive writing, though we intend to
cover a variety of genres as the work proceeds. Some tasks focused on evaluating
sources and choosing appropriate material to support or refute an argument; others, on
building and structuring an argument and embodying it in an essay; still others on
methods for critiquing other people’s arguments and revising one’s own. We expect to
pilot additional tests to create a sequence of tests that cover all the major modes of
writing, and which provide a variety of project models that require instructionally useful
combinations of specific writing competencies.
The tests were administered via computer on custom-designed software in a
format that supported integration of reading and research with project-focused writing
tasks. Although the design is still exploratory, results from the first pilot test are
encouraging. The pilot was administered to a group of 120 students, with individuals
being randomly assigned to one or the other of the two tests. The distribution of scores
across the two tests was roughly comparable, with high reliability across tasks.
For the pilot study briefly described above, we developed an analytic scoring
rubric focused on the three strands of writing proficiency proposed in the competency
model.
Strand I (use language and literacy skills). Instead of using multiple-choice items
to measure these skills, the current approach is to apply a generic Strand I rubric to all
written responses of sufficient length. This rubric focuses on sentence-level features of
the students’ writing, such as grammar, usage, mechanics, spelling, vocabulary, and
variety of sentence structures.
14
Strand II (use strategies to manage the writing process). Although some tasks may
be designed to assess this category of skills, the primary approach is to apply a generic
Strand II rubric to all written responses of sufficient length in order to measure
document-level skills, including organization, structure, focus, and development.
Strand III: (use critical-thinking skills). In the test design, the prewriting and
inquiry tasks (or, occasionally, post-essay tasks) tend to focus primarily on critical-
thinking skills; the extended task, of course, draws on these skills as well. However, in
contrast to the generic rubrics for sentence-level and document-level skills, the rubrics for
critical-thinking skills are specific to each task. Evaluating the quality of ideas and
reasoning in a response requires a tailored rubric that reflects the content-related
requirements of the task.
In addition, lower-half scores were given diagnostic labels indicating how they
failed to meet the rubric, so that (for instance) an essay might be identified as having too
many grammatical errors on Strand I, being disorganized on Strand II, or failing to take a
clear position on Strand III. The most common error patterns in the pilot data involved
problems in sentence structure, failure to develop content adequately, and weak reasoning
(not giving strong supporting evidence.)
We are still experimenting with the scoring approach, and hope to develop novel
methods that will allow us to combine reliable measurement of strands I and II (where
automated scoring methods are most defensible) while giving the tests a clear focus on
content and rhetorical purpose.
15
Potential for Automated Scoring
One issue that should be mentioned here is the relevance of automated scoring
techniques. For many purposes – such as scoring the quality of critical thinking in an
essay – there is no substitute for human scoring. However, there are several strong
reasons to consider automated scoring as part of the CBAL test design: it can make more
rapid and detailed feedback feasible, thereby making tests more instructionally useful,
and can help to contain costs and compensate for the difficulty of obtaining sufficient
teacher time to score multiple assessments per year.
Most of the features relevant to Strand I of the writing competency model (and
some of those relevant to Strand II) can be measured using the kinds of features needed in
any case for automated essay scoring. Recently several approaches to automated essay
scoring have come into use for scoring large-scale assessments, including the PEG
system, a descendant of Page’s original system (Page 1966a, b; 1994, 1995, 2003),
methods based on Latent Semantic Analysis, or LSA (Foltz, Laham & Landauer, 1999;
Landauer, Laham & Foltz, 2003), and the e-rater™ system (Burstein et al., 1998,
Burstein, 2003).
At ETS, the primary automated essay scoring system is e-rater, which has
changed significantly over the years. While the original e-rater system used more than 50
individual features and selected whichever features best predicted human scores for a
particular prompt, e-rater 2.0 uses a much smaller set of features specifically selected to
measure particular aspects of writing performance: e.g., content, organization,
development, vocabulary, grammar, usage, mechanics, and style (Attali & Burstein,
2005; Ben-Simon & Bennett, 2007; Burstein, 2003; Chodorow & Burstein, 2004). It is
16
important to note, however, that the definition of features in e-rater 2.0 and their
alignment with a writing model is not based upon an explicit model of writing
competency, nor is the evidence used in e-rater’s feature set based upon direct evidence
of how writers mature and develop in their writing skills. It represents, rather, a selection
of features whose relation to known writing constructs is clear and defensible and which
can be shown to provide good predictions of human essay grades. Given that e-rater was
originally built specifically for a particular type of essay (persuasive GMAT essays) and
that a subset of the CBAL writing tasks are intended to be in essay format, we do not
intend to use e-rater unmodified for scoring, but it provides a crucial foundation upon
which automated scoring for CBAL writing can be built. Moreover, there is evidence
that e-rater scores can be used to provide a developmental scale covering student writers
from 4th to 12th grades (Attali & Powers, 2007).
We expect in future research to use many of the features used in automated essay
scoring systems to provide measurement particularly of strand I of the CBAL
competency model (language and literacy skills), and thus to provide automated scoring
for some (but not all) of the constructs relevant to writing. We do not, however, expect to
use automated scoring as the sole scoring technology, since one of the main purposes of
the CBAL Writing Assessment is to focus instruction and test preparation on the skills
needed to support effective, rhetorically appropriate engagement with content. However,
it is possible that automated scoring could play a confirmatory (check scoring) role even
for Strand III scores, since early pilot results indicate strong correlations across strand
scores.
17
Conclusions
The strategies and prototypes presented here are in an early stage. The efficacy of
the CBAL approach is unproven either as an assessment design or as a method of
aligning assessment with instruction, and much research remains to be done. Preliminary
response from teachers and preliminary analysis of data from our first pilot suggests,
however, that the approach has promise, and at the very least it represents a major
attempt to develop a new kind of writing assessment from first principles.
18
References
Attali, Y. & Burstein, J. (2005). Automated essay scoring with e-rater v. 2.0 (ETS
Research Report RR-04-45). Princeton, NJ: Educational Testing Service.
Attali, Y. & Powers, D. (2007). [A developmental writing scale]. Unpublished raw data.
Beaufort, A. (2000). Learning the trade: A social apprenticeship model for gaining
writing expertise. Written Communication: 17(2), 185-223.
Ben-Simon, A. and Bennett, R.E. (2007). Toward more substantively meaningful
automated essay scoring. The Journal of Technology, Learning and Assessment:
6(1).
Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition.
Hillsdale, NJ: Lawrence Erlbaum Associates.
Bourdin, B., & Fayol, M. (1994). Is written language production more difficult than oral
language production? A working memory approach. International Journal of
Psychology: 29(5), 591-620.
Bourdin, B., & Fayol, M. (2000). Is graphic activity cognitively costly? A developmental
approach. Reading and Writing: 13(3-4), 183-196.
Burstein, J. (2003). The E-rater Scoring Engine: Automated Essay Scoring with Natural
Language Processing. In M.D. Shermis and J.C. Burstein (Eds.), Automated essay
scoring: A cross-disciplinary perspective (pp. 113-122). Mahwah, NJ: Lawrence
Earlbaum Associates.
Burstein, J., Kukich, K., Wolff, S., Lu, C., Chodorow, M., Braden-Harder, L. and Harris,
M.D. (1998). Automated scoring using a hybrid feature identification technique.
19
Proceedings of the Annual Meeting of the Association of Computational
Linguistics, Montreal, 1, 206-210.
Chodorow, M. & Burstein, J. (2004). Beyond essay length: Evaluating e-rater’s
performance on TOEFL essays (TOEFL Research Report 73). Princeton, NJ:
Educational Testing Service.
Cohen, M. and M. Riel. (1989). The effect of distant audiences on students' writing.
American Educational Research Journal, 26(2): 143-159.
Coirier, P., Andriessen, J. E. B., & Chanquoy, L. (1999). From planning to translating:
The specificity of argumentative writing. In P. Coirier & J. Andriessen (Eds.),
Foundations of argumentative text processing (pp. 1–28). Amsterdam:
Amsterdam University Press.
Daiute, C. (1986). Do 1 and 1 make 2? Patterns of influence by collaborative authors.
Written Communication, 3(3), 382-408.
Daiute, C., & Dalton, B. (1993). Collaboration between children learning to write: Can
novices be masters? Cognition and Instruction, 10(4), 281-333.
Deane, P., Quinlan, T., Odendahl, N., Welsh, C. and Bivens-Tatum, J. Forthcoming.
Cognitive models of writing: Writing proficiency as a complex integrated skill.
CBAL Literature Review—Writing. ETS Research Report, under review.
DeGroff, L. J. C. (1987). The influence of prior knowledge on writing, conferencing, and
revising. Elementary School Journal: 88(2), 105-118.
De La Paz, S. (2005). Effects of historical reasoning instruction and writing strategy
mastery in culturally and academically diverse middle school classrooms. Journal
of Educational Psychology: 97(2), 139-156.
20
De La Paz, S., & Graham, S. (1997a). Effects of dictation and advanced planning
instruction on the composing of students with writing and learning problems.
Journal of Educational Psychology: 89(2), 203-222.
De La Paz, S., & Graham, S. (1997b). Strategy instruction in planning: Effects on the
writing performance and behavior of students with learning difficulties.
Exceptional Children: 63(2), 167-181.
De La Paz, S. & Graham, S. (2002). Explicitly teaching strategies, skills and knowledge:
Writing instruction in middle school classrooms. Journal of Educational
Psychology: 94(4), 687-698.
Elliot, N. (2005). On a scale: A social history of writing assessment in America. New
York: Peter Land.
Felton, M. and D. Kuhn (2001). The development of argumentative discourse skill.
Discourse Processes: 32(2&3), 152-153.
Flower, L. (1989). Cognition, context, and theory building. College Composition and
Communication: 40(3) 282-311.
Foltz, P. W., Laham, D., & Landauer, T. K. (1999a). The Intelligent Essay Assessor:
Applications to educational technology. Interactive Multimedia Electronic
Journal of Computer-Enhanced Learning: 1(2). Retrieved March 11, 2008, from
http://imej.wfu.edu/articles/1999/2/04/index.asp
Galbraith, D. (1999). Writing as a knowledge constituting process. In M. Torrance & D.
Galbraith (Eds.), Knowing what to write: Conceptual processes in text production
(pp. 139-160). Amsterdam: Amsterdam University Press.
21
Graham, S. (1997). Executive control in the revising of students with learning and writing
difficulties. Journal of Educational Psychology, 89(2), 223-234.
Graham, S. and Harris, K.R. (2000). The role of self-regulation and transcription skills in
writing development. Educational Psychologist: 35(1), 3-12.
Graham, S. and Perin, D. (2007). Writing next: Effective strategies to improve writing of
adolescents in middle and high schools – A report to Carnegie Corporation of
New York. Washington, DC: Alliance for Excellent Education.
Hayes, J. R. (1996). A new framework for understanding cognition and affect in writing.
In C. M. Levy & S. Ransdell (Eds.), The science of writing: Theories, methods,
individual differences, and applications (pp. 1-27). Mahwah, NJ: Lawrence
Erlbaum Associate.
Hayes, J.R. (2004). What triggers revision? In G. Rijlaarsdam (Series Ed.) & L. Allal, &
L. Chanquoy (Vol. Eds.), Studies in writing: Vol. 13. Revision: Cognitive and
instructional processes (pp. 189–207). Boston: Kluwer Academic Publishers.
Hayes, J. R., & Flower, L. S. (1980). Identifying the organization of writing processes. In
L. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 3-30).
Hillsdale, NJ: Lawrence Erlbaum Associates.
Hillocks, G, Jr. (1987). Synthesis of research on teaching writing. Educational
Leadership: 44(8), 71.
Hillocks, G., Jr. (1995). Teaching writing as reflective practice. New York: Teachers
College Press.
Hirschman, J. (2007, November 10). Letters: A school is more than an A, B, or C. New
York Times. Retrieved March 11, 2008, from http://www.nytimes.com
22
Kaufer, D. S., Hayes, J. R., & Flower, L. S. (1986). Composing written sentences.
Research in the Teaching of English: 20(2), 121-140.
Kellogg, W. H. 1970. Syntactic maturity in schoolchildren and adults. Monographs of the
Society for Research in Child Development, 35(1, Serial No. 134).
Kellogg, R. (1988). Attentional overload and writing performance: Effects of rough draft
and outline strategies. Journal of Experimental Psychology: Learning, Memory
and Cognition: 14(2), 355-365.
Kellogg, R.T. (2001). Competition for working memory among writing processes. The
American Journal of Psychology: 114(2), 175-191.
Kent, T. (2003). Post-process theory: beyond the writing-process paradigm. In L. Z.
Bloom, D. A. Daiker & E. M. White (Eds.), Composition studies in the new
millennium: Rereading the past, rewriting the future. Carbondale, IL: Southern
Illinois University Press.
Kostouli, T. (2005). Writing in context(s): Textual practices and learning processes in
sociocultural settings. Heidelberg: Springer Verlag.
Kuhn, D. (1991). The skills of argument. Cambridge: Cambridge University Press.
Kuhn, D., Katz, J. B., & Dean, D., Jr. (2004). Developing reason. Thinking & Reasoning:
10(2), 197-219.
Kuhn, D., Shaw, V., & Felton, M. (1997). Effects of dyadic interaction on argumentative
reasoning. Cognition and Instruction: 15(3), 287-315.
Landauer, T.K., Laham, D. and Foltz, P.W. (2003). Automated scoring and annotation of
essays with the Intelligence Essay Assessor. In M.D. Shermis and J.C. Burstein
23
(Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 87-112).
Mahwah, NJ: Lawrence Earlbaum Associates.
Langer, J. A. (1985). Children's sense of genre: A study of performance on parallel
reading and writing. Written Communication: 2(2), 157-187.
Matsuhashi, A. (1981). Pausing and planning: The tempo of written discourse production.
Research in the Teaching of English: 15(2), 113-134.
McCutchen, D. (1986). Domain knowledge and linguistic knowledge in the development
of writing ability. Journal of Memory and Language: 25(4), 431-444.
McCutchen, D. (1996). A capacity theory of writing: Working memory in composition.
Educational Psychology Review: 8(3), 299-325.
McCutchen, D. (2000). Knowledge, processing, and working memory: Implications for a
theory of writing. Educational Psychologist: 35(1), 13-23.
McCutchen, D., Francis, M., & Kerr, S. (1997). Revising for meaning: Effects of
knowledge and strategy. Journal of Educational Psychology: 89(4), 667-676.
Means, M. L., & Voss, J. F. (1996). Who reasons well? Two studies of informal
reasoning among children of different grade, ability, and knowledge levels.
Cognition & Instruction: 14(2), 139.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational
assessments. Measurement: Interdisciplinary Research and Perspectives: 1(1), 3-
62.
Olive, T. & Kellogg, R.T. (2002). Concurrent Activation of high- and low-level processes
in written composition. Memory and Cognition: 30(4), 594-600.
24
Page, E.B. (1966a). The imminence of grading essays by computer. Phi Delta Kappan:
47, 238-243.
Page, E.B. (1966b). Grading essays by computer: Progress report. Proceedings of the
Invitational Conference on Testing Problems (pp. 86-100). Princeton, NJ:
Educational Testing Service.
Page, E.B. (1994). New computer grading of student prose, using modern concepts and
software. Journal of Experimental Education: 62(2), 127-142.
Page, E.B. (1995). Computer grading of essays: A different kind of testing? Address for
APA Annual Meeting, Sunday, Aug. 13, 1995. Session 3167, Sheraton N.Y.
Hotel, Princess Ballroom, 1:00-1:50 P.M. Invited address sponsored by Divs. 16,
5, 7, 15, Prof. Timothy Z. Keith, Chair.
Page, E.B. (2003). Project essay grade: PEG. In M.D. Shermis and J.C. Burstein (Eds.),
Automated essay scoring: A cross-disciplinaryperspective (pp. 43-54). Mahwah,
NJ: Lawrence Erlbaum
Pearl, S. (1979). The composing processes of unskilled college writers. Research in the
Teaching of English, 13(4), 317-336.
Pellegrino, J.W., Chudowsky, N., and Glaser, E. (Eds.). (2001). Knowing what students
know: The science and design of educational assessment. Washington, D.C.:
National Academy Press.
Perkins, D. N. (1985). Postprimary education has little impact on informal reasoning.
Journal of Educational Psychology: 77(5), 562-571
25
Perkins, D. N., Allen, R., & Hafner, J. (1983). Difficulties in everyday reasoning. In W.
Maxwell & J. Bruner (Eds.), Thinking: The expanding frontier (pp. 177-189).
Philadelphia, PA: The Franklin Institute Press.
Piolat, A., Roussey, J.-Y., Olive, T., & Farioli, F. (1996). Charge mentale et mobilisation
des processus rédactionnels : Examen de la procédure de Kellogg. Psychologie
Française: 41(4), 339-354.
Quinlan, T. (2004). Speech recognition technology and students with writing difficulties:
improving fluency. Journal of Educational Psychology: 96(2), 337-346.
Schilperoord, J. (2002). On the cognitive status of pauses in discourse production. In T.
Olive & C. M. Levy (Eds.), Contemporary tools and techniques for studying writing
(pp. 61-90). Dordrecht/Boston/London: Kluwer Academic Publishers.
Shanahan, T. (2006). Relations among oral language, reading and writing development.
In C.A. MacArthur, S. Graham, and J. Fitzgerald (Eds.), Handbook of writing
research (pp. 171-183). New York & London: Guilford Press.
Torrance, M., & Galbraith, D. (2005). The processing demands of writing. In C.
MacArthur, S. Graham & J. Fitzgerald (Eds.), Handbook of writing research.
New York: Guilford Publishers.
Yarrow, F.; Topping, K.J. (2001). Collaborative writing: The effects of metacognitive
prompting and structured peer interaction. British Journal of Educational
Psychology: 71(2): 261-282.