Maestro: A Computer Tutor for Writers

Kurt Rowley, Ph.D.Maestro Principal Investigator 1995-1998

www.KurtRowley.com

'They [students] seem to be less distressed about pre-writing, drafting & editing...'

IntroductionA Computer Tutor for Writers (CTW) titled

'Maestro' was developed by Air Force Research Laboratory (AFRL) 1995-8 as a technology transfer project

Advanced cognitive training technologies Best available information about writing composition Best available research methods

Goal of improving confidence in education

"..we can never have meaningful progress in educational programs until we can have the same level of confidence in them that physicians can have in procedures or medications passed by the FDA." (Robert Slavin)

Background

Effectiveness of computer-based tools in improving student writing

Zellermayer and colleagues tested a computerized 'writing partner' (1991): They determined that students can internalize guidance and produce improved writing products with use of facilitative writing software.

Vygotsky (1978) and Salomon (1988) suggested the notion of 'procedural facilitation' in which visualizations of procedural skills are used as support structures for students.

Background

Effectiveness of computer-based tools in improving student writing (continued)

AFRL background studying 'intelligent tutoring systems' using supportive learning environments

Patricia Carlson, English Professor from Rose Hulman Institute of Technology (Indiana) with colleagues at AFRL designed a supportive writing environment 'R-WISE' following the 'procedural facilitation' strategy (1994, 1996, 1998)

R-WISE provided a rich background for the development of the Maestro CTW

BackgroundReading and Writing in a Supportive Environment (R-WISE) Research 1992-93 1993-94 1994-95 1995-96

Software R-WISE 1.0

(Cubing, Idea Board)

R-WISE 2.0 (Revision)

R-WISE 3.0 (Guided vs. Open mode)

R-WISE 4.4 (Better Networking)

Study Design Control-Treatment (Efficacy)

Comparison of Traditional and WP Controls

Learner control variance & teaching styles

Replication and prompt equality study

N 852 1,151 1,277 & 1,286 617

Results 23% net gain over Classroom control (Sig.)

4% net gain over WP control (Sig.)

8% gain, 4% for Open over Guided (Sig.)

4% net gain (Significant)

Teacher Inputs

Many lessons learned from R-WISE, inputs collected from 25 R-WISE English teachers

Address student learning styles

Motivational strategies (R-WISE included multiple individual writing exercises, teachers wanted students to write entire papers in the tutor)

Incorporate teacher-generated assignments and State-required writing standards

Address entire writing process as a single skill

Use a single, consistent user interface

Cognitive Model

Design of a CTW required a writing model Overall instructional goal selected for CTW

was to address teacher concerns and build an instructional system that could teach the writing process as a singular skill

Flower & Hayes cognitive model of writing process selected as primary guide

Added Scardemalia & Bereiter notion of 'knowledge transforming' practiced by expert writers vs. 'knowledge telling' by novices

Cognitive Model

WRITING TASK ENVIRONMENTWriting assignments (prompts, topics, audience)

The writer uses workspaces to help with text production

WRITER’S KNOWLEDGE

Goals Ideas

Writing Plan (organize ideas)

Drafting (translate ideas into paragraphs)

Revision Publish (review and edit)

Monitor Writing Goal Achievement

Adapted from: Hayes & Flower, 1980

Prewrite Compose Edit

WRITING PROCESS

The design of the Maestro CTW wasbased on cognitivestudies of the Expert Writing Process

TIMs & Advice (memory, topic research, audience, writing plans)

CTW DesignStudent Interface Workspace Descriptions

Workspace Category

Description

Goals Goals workspaces cover the spectrum of goal setting, from identifying topic, reader, and style, to reflecting on writer biases and experience, and analyzing thesis ideas.

Ideas Ideas workspaces include a sequence of visualizations of the idea-generation process of an expert writer. This includes a questioning technique to explore a topic (cube), note-taking (notes), organizing ideas into categories (cluster), and organizing categories into an outline format (outline).

Plan The Plan workspaces assist with the development of a formal writing plan based on information generated in the Ideas screens. This includes several split-screens in which the student copies ideas from the left half of the screen to a structured writing plan diagram on the right half.

Draft Drafting workspaces assist in the generation of sentences and paragraphs, based on the writing plan diagram. The student refers to the writing plan diagram on the left side of the screen, and then translates those ideas into text by generating sentences and paragraphs on the right-hand side of the screen.

Revise Revision workspaces help the student select appropriate editing questions for their draft based on their writing goals, and then apply those questions in the final editing and review of their work.

Publish This is an open workspace for formatting, highlighting, proof-reading, spell-checking, and printing a final writing product. Students may also save an electronic copy for email to the teacher or peers for review, send to an acquaintance, or to post to a web page.

CTW Design

Student Interface

Teacher Module

Student Records

Cognitive Tutoring Engine

Teaching Resources

Designed to Address Individual Student Needs

Individualized guidance for progressive development of cognitive writing process skills Tailored Instructional Modules accommodate student

learning styles and interests, based on automated survey Advice guides student through the writing process making

suggestions and summarizing writing process skills (solicited and unsolicited)

Workspaces provide procedural facilitation, helping students to systematically utilize an expert-level writing process

Students motivated to learn writing process Teachers select realistic, multimedia prompts Publish to real audience (print-outs, used for assignments)

75 Instructional Modules

400+ Advice Statements

22 Workspaces(note-taking example)

22 Workspaces(Revision Example)

CTW Study

Full school-year study 1996-1997 with students and teachers using CTW to learn writing process and to complete writing assignments.

Pre-test / post-test, contrasted groups design

Combined N=471, control group N=174

Treatment groups:

1 (2+ sessions, 2-6 hours using CTW), N=99

2 (4+ sessions, 6-11 hours using CTW), N=163

3 (6+ sessions, 11+ hours using CTW), N=36

CTW Study

ANOVA showed differences between the groups were significant

F [(3, 471) = 3.595, p<.05)]

Posthoc Tukey's Honestly Significant Difference Test showed that the mean score differences were significant between group 3 (6+ sessions) and the control group, and groups 3 and 1, but not between groups 2 and 3

The Pre/Posttest result of primary interest was a 11% score gain for group 3

CTW Study

1996-1997 CTW Study Descriptive Statistics*

Group Pretest Mean

Pretest SD

Posttest Mean

Posttest SD

Gain %

N

0 (Control) 2.28 1.00 2.25 1.02 -1% 174

1 (2+ Sessions)** 2.19 0.94 1.98 0.94 -4% 99

2 (4+ Sessions) 2.29 0.94 2.36 0.89 1% 163

3 (6+ Sessions) *** 1.74 0.84 2.31 1.00 11% 36

* All tests scored on a 5 point scale ** The negative gain for group 2 is curious and may reflect negative transfer *** Pre/posttest mean differences between group 3 and all other groups except the group 2 are statistically significant.

CTW Study

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

Control Group 1 Group 2 Group 3

Gain Scores

Feedback from Teachers

Students more motivated to write, little lab management trouble, even at low SES schools

Students produce more coherent essays

Teachers like the individualized pacing

Teachers like the fact that the software is consistent with the writing processes they are teaching

Conclusions

The Maestro CTW appeared to produce positive effects, although statistical power is weak due to low N for the full treatment (group 3)

The study suggests that writing process skills can be taught / reinforced through procedural facilitation using computer software

Further research with specific populations is merited, particularly with groups that need help managing the writing process.

Publications

Rowley, K. & Meyer, N.  (2003).  The effect of a computer tutor for writers on student writing achievement.  Journal of Educational Computing Research, 29(2), 169-187.

Steuck, K., Rowley, K. & Kretschmer, M.  (2000).   Partnering to implement computer-based tutoring systems in secondary schools.   Journal of Interactive Instruction Development, 12(1), 16-22.

Rowley, K., Carlson, P., & Miller, T. (1998). A cognitive technology to teach composition skills: Four studies with the R-WISE writing tutor. Journal of Educational Computing Research, 18(3), 259-296.