A National Online Essay A National Online Essay Evaluation Service:Evaluation Service:Valid, Reliable, & Cost Valid, Reliable, & Cost Effective Writing AssessmentEffective Writing Assessment
Leslie C. Perelman <Leslie C. Perelman <[email protected]@mit.edu>>
Program in Writing and Humanistic StudiesProgram in Writing and Humanistic Studies
Massachusetts Institute of TechnologyMassachusetts Institute of Technology
iMOATiMOATEnhancing the best way of assessing Enhancing the best way of assessing writingwritingEnthusiastic reception from both Enthusiastic reception from both students & facultystudents & facultyCost effectiveCost effectiveReliable scoring and deliveryReliable scoring and delivery
MotivationsMotivations
Existing modes of placement Existing modes of placement inadequateinadequate
Standardized testsStandardized testsTimed impromptuTimed impromptuPortfolioPortfolio
Students write on computers not Students write on computers not paperpaper
The Common PracticeThe Common Practice
Student walks into large Student walks into large room and writes with room and writes with pen and paper on small pen and paper on small deskdeskStudent reads, at most, a Student reads, at most, a two or three paragraph two or three paragraph text for each essaytext for each essay
Student writes one or Student writes one or two essays on specific two essays on specific questionsquestions
Handwritten essays are Handwritten essays are scored (usually by two scored (usually by two readers)readers)
Score is sent on paper to Score is sent on paper to student and / or advisorstudent and / or advisor
Student works at homeStudent works at home
Student views readings of Student views readings of any length onlineany length onlineStudent accesses Student accesses questions onlinequestions onlineStudent writes one or Student writes one or two essays at home two essays at home within a specified period within a specified period using a computer.using a computer.
Essays are scored on Essays are scored on either printed copy or on either printed copy or on screen by readers in real screen by readers in real or virtual grading or virtual grading sessionssessionsStudent and advisor Student and advisor access both score and access both score and text of essays.text of essays.
iMOATiMOAT
Developed by Experts in Writing Developed by Experts in Writing Assessment at Five Diverse Assessment at Five Diverse UniversitiesUniversities
More Time (and savings) More Time (and savings) Allowing for Two EssaysAllowing for Two Essays
““if you want to give real weight to if you want to give real weight to [student] essays, you must secure two test [student] essays, you must secure two test essays from each student with some essays from each student with some separation in time as well as in topic, and separation in time as well as in topic, and the least possible separation in time as the least possible separation in time as well as in topic, and the least possible well as in topic, and the least possible separation in time is to have one test essay separation in time is to have one test essay written in the morning and the other in written in the morning and the other in afternoon of the same day.afternoon of the same day.””
---- Paul Paul DiederichDiederich of ETSof ETSThe founder of modern writing The founder of modern writing assessmentassessment
iMOATiMOAT FeaturesFeaturesAutomates all components of Automates all components of best practices in writing best practices in writing assessmentassessment
Focusing on writing processFocusing on writing processAllowing students modern toolsAllowing students modern toolsOnline holistic scoring sessions with Online holistic scoring sessions with real time quality control measuresreal time quality control measures
Digital Archives of EssaysDigital Archives of Essays
For program assessmentFor program assessment
For portfoliosFor portfolios
For institutional researchFor institutional research
Two Perpetual QuestionsTwo Perpetual Questions
What about cheating?What about cheating?
Will all students have access?Will all students have access?
Feedback from Feedback from StudentsStudents
““I am from a small town in Missouri. My I am from a small town in Missouri. My writing essay was easy for me because I writing essay was easy for me because I enjoy thinking about these things, such enjoy thinking about these things, such as philosophy and the future. I went as philosophy and the future. I went out to my cabin and I just sat around out to my cabin and I just sat around the campfire the whole time, just the campfire the whole time, just enjoying the silence and writing enjoying the silence and writing whatever came into my head. The next whatever came into my head. The next day I typed it up.day I typed it up.”” (DePaul)(DePaul)
Feedback from StudentsFeedback from Students
““Taking the online writing tests, I Taking the online writing tests, I found, gave me the opportunity to found, gave me the opportunity to leave the information when I needed leave the information when I needed time to think and come back to it time to think and come back to it when I was ready. The freedom also when I was ready. The freedom also allowed me to revise my work and allowed me to revise my work and not feel though my work was rushed not feel though my work was rushed or inhibited in any way.or inhibited in any way.”” (DePaul) (DePaul)
Benefits for Writing FacultyBenefits for Writing FacultyAllows essays and questions to reflect the Allows essays and questions to reflect the universityuniversity’’s and programs and program’’s specific s specific educational needs, priorities, and goalseducational needs, priorities, and goalsMaintains control of all aspects of Maintains control of all aspects of evaluation process within the university evaluation process within the university and writing programand writing programOnline grading sessions maintain Online grading sessions maintain ““staff staff consensus building consensus building ”” benefit while benefit while potentially reducing costspotentially reducing costsReflects and measures the writing process Reflects and measures the writing process used and valued in collegeused and valued in collegeEncourages sharing of Encourages sharing of ““best practicesbest practices””among schools using the systemamong schools using the system
Reliability of AssessmentReliability of Assessment
““Faculty involved in English Faculty involved in English placement strongly feel that the placement strongly feel that the Online represents much better the Online represents much better the kind of writing students will do in kind of writing students will do in our courses, and, so far, it is our courses, and, so far, it is working equally well for a wide working equally well for a wide range of students from honors to range of students from honors to very low developmental.very low developmental.”” (U. of (U. of Cincinnati)Cincinnati)
Reliability of Assessment at MIT:Reliability of Assessment at MIT:Fall 2002 Survey of Instructors Fall 2002 Survey of Instructors Teaching Introductory Writing ClassesTeaching Introductory Writing Classes
1st Semester Students in Introductory Writing
Classes N = 147
1% 12%
34%53%
1 2 3 4
1=Should not 1=Should not be in classbe in class
4=Definitely should4=Definitely shouldBe in classBe in class
Reliability of Assessment at MIT:Reliability of Assessment at MIT:Fall 2002 Survey of Faculty TeachingFall 2002 Survey of Faculty TeachingWritingWriting--Intensive ClassesIntensive Classes
1st Semester Students in Writing-Intensive Humanities Classes N=125
0%10%20%30%40%50%60%70%80%90%
100%
1 2 3 4Faculty Responses 1=Should not have been in class; 4 = appropriate writing
skills for class
Pass
Subject Req
Reliability of Assessment: Reduction in Reliability of Assessment: Reduction in Complaints and Appeals at MIT 1992Complaints and Appeals at MIT 1992--20032003
Appeals & Complaints by Students about Scores on MIT Placement Test
0
10
20
30
40
50
60
70
80
90
1992
1993
1995
1996
1997
1998
1999
2000
2001
2002
2003
Year
# of
Stu
dent
s
Architectural OverviewArchitectural Overview
Web Server MS Windows 2000 / IIS
MS ASP.NETC#
DatabaseMS SQL/2000
InternetInternet
Administrator
Student
Reader
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
[Write Comments]
Assign Readers
Finalize Scores
Publish Results
Reporting
Register for Exam
View Results [& Comments]
Read & Score Essays
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
TestWrite HighTestWrite High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students Register for Exam
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Register for Exam
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
Register for Exam
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
Assign Readers
Register for Exam
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
Assign Readers
Register for Exam
Read & Score Essays
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
Assign Readers
Finalize Scores
Register for Exam
Read & Score Essays
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
Assign Readers
Finalize Scores
Publish Results
Register for Exam
Read & Score Essays
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
Assign Readers
Finalize Scores
Publish Results
Register for Exam
View Results
Read & Score Essays
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
Assign Readers
Finalize Scores
Publish Results
Reporting
Register for Exam
View Results
Read & Score Essays
iMOAT HighiMOAT High--Level WorkflowLevel Workflow
Create Class
Administrator Student Reader
Create Test
Upload Students
View Readings & Questions
Write & Submit Essay
Assign Readers
Finalize Scores
Publish Results
Reporting
Register for Exam
View Results
Read & Score Essays
Greater Efficiency in Scheduling Greater Efficiency in Scheduling and Staffing Classesand Staffing Classes
Saving Money: Reliability & Timeliness Saving Money: Reliability & Timeliness of Assessment Reduce Number of of Assessment Reduce Number of Sections NeededSections Needed
Accuracy of Online Evaluation = Accuracy of Online Evaluation = Fewer SectionsFewer Sections
LSU (4,000 1LSU (4,000 1stst Year Students) is Year Students) is scheduling 4 fewer sections per year scheduling 4 fewer sections per year because of fewer students being because of fewer students being misplacedmisplaced
Save Money: Typed texts (even if they Save Money: Typed texts (even if they are longer) cost less to gradeare longer) cost less to grade
At MIT At MIT (@$25 / hr.)(@$25 / hr.)
the average the average handwritten text handwritten text costs $7.15 to be costs $7.15 to be scored by two scored by two readersreadersthe average typed the average typed text costs $5.40 text costs $5.40 to be scored by to be scored by two readerstwo readers
$0.00
$1.00
$2.00
$3.00
$4.00
$5.00
$6.00
$7.00
$8.00
Handwritten Typed
$7.15
$5.40
Save Money: Automatically Save Money: Automatically AccommodateAccommodateStudents with DisabilitiesStudents with Disabilities
iMOATiMOAT
A more valid and reliable A more valid and reliable assessmentassessmentGreater faculty and student Greater faculty and student satisfactionsatisfactionSignificant reduction in costsSignificant reduction in costsProven and reliable platformProven and reliable platform
Top Related