Running head: TEACHER MADE ASSESSMENT ACTIVITY…student.fgcu.edu/mlenglan/eportfolio/images/Teacher...
Transcript of Running head: TEACHER MADE ASSESSMENT ACTIVITY…student.fgcu.edu/mlenglan/eportfolio/images/Teacher...
Teacher Made Assessment Activity 1
Running head: TEACHER MADE ASSESSMENT ACTIVITY: EDF 6432
Teacher Made Assessment Activity: Accelerated Reader Teacher Made Tests
Margaret England
Florida Gulf Coast University
Teacher Made Assessment Activity 2
Abstract
This paper investigates the Validity and Reliability of one teacher made
Accelerated Reader quiz. During Item Analysis it was determined that the book
Two Crows Counting was not a valid selection for a 1.0 Accelerated Reader quiz.
The book contained sentence fragments, lacked verbs, and included incorrect
punctuation. The book did not support the reading skills being taught
in the classroom.
Teacher Made Assessment Activity 3
Teacher Made Assessment Activity: Accelerated Reader Teacher Made Tests
Statement of Purpose:
This paper investigates decisions about managing classroom instruction by evaluating a
teacher made Accelerated Reader quiz in an elementary school’s Accelerated Reader
(AR) technology-based literacy program database. AR Criterion Reference Tests (CRT)
are low-stakes formative assessments “intended to determine whether or not a student has
read a book” (Renaissance Learning1999 p.2) within their tested Zone of Proximal
Development (ZPD) reading level. “These quizzes are encouraging rather than
intimidating, chiefly because a student who has read a book should be able to pass the
quiz” (Renaissance Learning 1999, April). The AR program provides students with
immediate feedback after they take a 5-20 question quiz. The quiz score, points,
student’s average, number of quizzes, and goal are included on the TOPS (Three
Opportunities to Praise Students) report. A computer-generated comment is included with
space at the bottom for comments from the monitor and teacher.
The school’s database includes over 8,000 quizzes purchased from Renaissance Learning
(AR). Up to 500 teacher made quizzes may be added to the database. The teacher made
quizzes have not been evaluated for validity or reliability during the eight years AR has
been implemented in the school. The purpose of this paper is to improve the validity and
reliability of one teacher made quiz in level 0.2-1.9 through item analysis.
Table of Specifications
The Table of Specifications (Appendix A) for Two Crows Counting could be used with
any Accelerated Reader Quiz or group of quizzes with adjustments being made for 10 or
20 item quizzes. The percentages of the evaluation are 50% Literal Comprehension, 15%
Skill in Book Selection, 15% skill in Paired Reading, 15% Reading Independently, and
Teacher Made Assessment Activity 4
5% Recording in Reading Log. The main purpose of an AR quiz is to check the student’s
literal comprehension of text as a way to motivate students and determine their
understanding (knowledge) of the content of a book (Renaissance 1999 p.2). The table
could be modified for use as a 9-week unit with percentage from quizzes scores used for
the content score and the totals percentages from the book selection, reading log, dualog
(paired) reading, and independent reading as the skill scores. When the student reads
book independently or with a partner the text itself can provide instruction “that
effectively promotes both comprehension development and the exercise of critical
thinking” (Institute p.6). The teacher provides instruction through frequent monitoring of
students during independent and paired reading. During Status of the Class (monitoring)
the teacher has the opportunity to “assess ongoing work, monitor individual progress,
provide informative feedback to students, adapt instruction as needed, and ultimate
improve overall student performance” (Renaissance, 2002 p.1). The importance of the
teacher guiding reading and intervening when needed are key elements to a successful
AR program. (Institute 1999 p.6-7). Another skill in this table is the use of the Reading
Log, “student-involved record keeping” (Willis), which is used to motivate students and
to help them record and reflect on their improvement. Students are also assessed for
selecting a book within their Zone of Proximal development (ZPD).
Choices of Assessment and Item Format
In order to choose a teacher made AR quiz to evaluate a Quiz Takers Reading Practice
Report was run on all teacher made quizzes in the 0.2 - 1.9 ATOS (AR readability
formula) range taken by students during the current school year (2003). The percentage
scores were reviewed for these quizzes. The quiz, Two Crows Counting, taken by 21
different students, had the lowest percentage correct with 63.8%. The goal of AR is for
students to earn an average score of 85 –92% on AR quizzes within their ZPD range
(Institute 1997). Students in the school’s database who had taken the quiz during the
Teacher Made Assessment Activity 5
previous 2002 school year scored an average of 65% and 74% in the 2001 school year.
The AR database includes records for students currently enrolled in the school. The
averages of students who have withdrawn are not included in the yearly averages.
In order to develop the teacher made quiz the teacher writes 5 - 20 multiple-choice
questions, an answer and 3 distracters taken from a book. Five questions are usually
written for shorter books and 10 or 20 questions for longer books. The directions for AR
quizzes are the same for all quizzes and quiz takers. After reading a book, the student
logs into the AR database using their name and password, they locate the quiz for the
book they read, and then they take the selected-response AR Quiz. The questions are
presented in the order that matches the chronology of the book and reinforces the story
grammar of the book (Renaissance 1999, April). The program randomly changes the
order in which the 4 responses to questions are presented. Only one answer may be
selected to each question. Once a student selects an answer and goes to the next question
the program does not allow them to go back to change an answer. The actual quiz taken
by a student can’t be printed. Appendix I. shows the performance of students after the 2nd
revision of the quiz.
In order to earn points on a quiz students “need to score at least 60 percent on a 5- and
10-quesiotn quiz and 70 percent on 20-question quizzes” (Renaissance1999, April). The
number of questions is determined by the difficulty and content of the book. Lower level
picture books usually have 5 questions. “Questions typically focus on significant events,
character’s and literacy features of a book” (Renaissance 1999, April). If a student scores
60% or more the program will show them the correct answers to questions they answer
incorrectly. Information concerning the questions is not retained in the database.
Teacher Made Assessment Activity 6
Individual interviews or students writing the number of missed the questions are two
ways to record the questions answered incorrectly.
Instrument and Performance
In order to determine the ATOS readability of Two Crows Counting, a text file
(Appendix C) was sent twice to the Renaissance Learning ATOS (Readability For Books)
Analyzer. The ATOS Analyzer results (Appendix D) included a word count of 113
words, book level of 3.7, and 0.5-point value if a student answers all questions correctly.
When the teacher made quiz was developed the word count entered into the database the
database required that a minimum of 1,000 words or more be entered in the word count
field. Three years ago AR went from Fry Readability to ATOS Reading Levels. The
book level using the Fry Readability Index (Advantage Learning, 1996) was 1.0.
Analysis of Instrument: Validity, Reliability, and Item Analysis
As stated earlier the purpose of taking an AR quiz is to determine if a student has read a
book and if they have read a book then they should be able to pass the quiz. Is it valid to
interpret the scores from Two Crows Counting as measuring reading comprehension
(Nitko p.37)? Upon examination of Two Crows Counting it was noted that the book is a
counting book written in rhyming sentence fragments (Appendix C). The book contains
no verbs. Periods and commas were used in some of the sentence fragments. The text in
Two Crows Counting was then compared with text in 1.0 and 3.7 level books in the
school library collection. The 1.0 and 3.7 books contained complete sentences and
punctuation. The vocabulary and word count in the 1.0 book more closely matched the
Two Crows Counting vocabulary. Complete sentences, verbs, and punctuation are skills
that are taught to beginning readers. In order for validity to be attained in assessments,
including AR assessments, they should emphasize what you taught and contain content
Teacher Made Assessment Activity 7
worth learning (Nitko, p.39). A quiz on Two Crows Counting is not a valid assessment
of what is being taught in the classroom.
The scores for students taking the Two Crows Counting quiz were evaluated using
Livingston’s Formulae for Reliability of a Criterion Referenced test (RCR). In 2001 ten
students took the quiz with a Livingston’s RCR of .84 (Appendix E). In 2002 twelve
students had a RCR of .83 (Appendix F) and in 2003 twenty-one students had a RCR
of .85 (Appendix G). These scores support the stability of the performance of students’
with a 1.0 ZPD reading level that took the quiz during the past three years. The Formula
was used to determine the RCR of .80 for 16 students (Appendix H) who to the first
revision of the quiz and 11 students who took the 2nd revision of the quiz with the RCR of
.81. The two revisions were administered one day only to a limited number of students.
Appendix J. includes the student’s scores after the 2nd revision. Appendix B includes the
original questions and the two revisions. .
Analysis of Instrument
Interviews were conducted after students took the original and first revision quiz. The
changes in the instrument, observations, and comments follow.
Question 1:
One _____ rising
Answer: sun Distracters: crow bird balloon
Students who missed this question were looking at the picture clue of the crows flying
towards the sun (p.6) and chose crow or bird for an answer. The question will be
rewritten with a blank at the end of the stem and distracters that are not on the same page
as the text for the stem.
Question 1 Revision 1:
One sun _______
Answer: rising Distracters: smiling shining hiding
Teacher Made Assessment Activity 8
Students missing this question were using prior knowledge and chose shining as the
answer.
Question 1 Revision 2:
What was rising?
Answer: sun Distracters: boats cats people
After collaboration with a first grade teacher and reading the suggestions for improving
the quality of stems of multiple-choice items (Nitko, p. 150) the question and stem were
rewritten.
Question 2
Three people ______ and four trees blowing
Answer: rowing Distracters: running sitting eating
Students incorrectly chose running. When rewritten the blank was put at the end of the
stem.
Question 2 Revison 1
Three people rowing and four trees ___________
Answer: blowing Distracters: growing crowing glowing
Using prior knowledge students incorrectly chose growing. After collaboration the stem
was shortened.
Question: 2 Revison 2
Four trees __________
Answer: blowing Distracters: growing crowing glowing
Question 3
Seven ____wading and eight guess parading.
Answer: herons Distracters: crows birds ducks
Students missing this question chose ducks. The stem was shortened with the blank
added at the end of the stem.
Question 3 Revision 1
Teacher Made Assessment Activity 9
Seven herons ___________
Answer: wading Distracters: fading parading swimming
The stem was shortened with the blank added at the end of the stem. Students missing
this question chose swimming. The herons in the picture were walking in the water.
Question 3 Revison 2
Seven herons ________
Answer: wading Distracters: fading eating flying
The distracters were rewritten.
Question 4
Four trees standing and _______ boats landing
Answer: three Distracters: two five six
Students are not allowed to open their AR book during testing and did not remember how
many boats were landing.
Question 4 Revison 1
Four trees standing and three boats ________-
Answer: landing Distracters: sailing rowing sinking
The question was rewritten with the blank at the end of the stem
Students chose rowing as an incorrect answer. After collaboration with another teacher
the stem was shortened and another question selected.
Question 4 Revision 2
The socks are ________
Answer: flapping Distracters: sailing rowing resting
Question 5
Big crow and small crow are __________
Answer: fast asleep Distracters: counting eating flying
This question was rewritten and different distracters were selected.
Question 5 Revision 1
Teacher Made Assessment Activity 10
At the end of the story big crow and little crow are ________
Answer: fast asleep Distracters; counting sheep eating corn flying alone
One student chose flying alone.
Question 5 Revision 2
At the end of the story big crow and little crow are ________
Answer: fast asleep Distracters; counting sheep eating corn flying alone
No changes were made to the stem or distracter.
After two revisions of the quiz, review by a colleague, and review of the ATOS Analyzer
there was evidence that the Two Crows Counting was not the appropriate book for a 1.0
teacher made AR quiz. The content did not assess the reading comprehension skills
being taught to beginning readers. The incomplete sentences, lack of verbs, and incorrect
punctuation were not an appropriated text for assessment. According to Popham reading
vocabulary and/or sentence structure that is too difficult is a factor affecting the validity
of an assessment.
Performance of Group and Individual Students
With each revision of the quiz there was an increase in the total percentage answers
correct. The average for this school year was 63.0 %, after the 1st revision it was 66.2%,
and 72.7 % after the 2nd revision. The target percentage was 80%. Only six students took
the quiz for both revision 1 and 2 after they reread the book. Each student’s score
showed an increased.
Student: Revision 1 # Correct Revision 2 # Correct
AA2 2 4
AA3 3 4
AA4 3 5
AA6 2 5
Teacher Made Assessment Activity 11
AA7 3 4
AA11 3 5
Five students took the quiz for the 2nd revision, but not revision 1.
AA1 5
AA5 1
AA8 1
AA8 3
AA10 3
The reading records for AA5 and AA8 were reviewed in the AR database. These
students consistently score 60% or more on AR tests in the 1.0 - 1.2 ZPD range.
Conclusion
AR quizzes are intended to determine whether or not a student has read a book within
their ZPD. Students who have read a book in their range should be able to pass a quiz.
(Renaissance Learning, 1999, April). Teachers need to be skilled in developing high
quality assessments (Nitko p.81). After careful analysis it was determined that a quiz
from the book Two Crows Counting is not a valid assessment of what is being taught in
the 1.0 reading level curriculum. The quiz and quiz records will be removed from the AR
database. In order to improve the validity of quizzes in the database the teacher made
quizzes will be evaluated using the Validity, Reliability, and Item Analysis approaches in
order to improve provides high quality assessments. It is our professional responsibility
to provide students with valid assessments (Nitko p.455-462).
Teacher Made Assessment Activity 12
References
Advantage Learning Systems. (1996). Accelerated Reader Installation User’s
Manuel, 82-84
Hutchcraft, G. Table of Specification for a Weather Unit in Middle School
Science. Message posted to http://courses.fgcu.edu~141
Hutchcraft, G. Livingston's Formulae for Reliability of a Criterion referenced test
Relval.xls. Message posted to http://courses.fgcu.edu
Hutchcraft, G. Steps for Building a Teacher-made Classroom Assessment.
Message posted to http://courses.fgcu.edu~144
Teacher Made Assessment Activity 13
Hutchcraft, G. Livingston's Reliability. Message posted to
http://courses.fgcu.edu~144
Institute for Academic Excellence, Inc. (1997, November). Critical Thinking and
Literature-Based Reading. Report from the Institute for Academic
Excellence, 1-8.
Kehoe, F. (1995, October). Basic Item Analysis for Multiple-choice Tests. Eric/
AE Digest, ED398 237.
Nitko, A. J. (2001). Educational Assessment of Students. Upper Saddle River,
New Jersey: Merrill Prentice Hall. (Original work published 1996)
Orgel, D. (1996). Two Crows Counting. Milwaukee, WI: Garth Stevens
Publishing.
Paul, T. J., M.B.A. (1996, October 4). Learning Information Systems: Theoretical
Foundations. Paper presented at National Research Center Conference.
Popham, W. J. What Do Classroom Teachers Really Need to Know about
Validity?. Retrieved from Dr. Gilbert Hutchcraft:
http://courses.fgcu.edu~144
Popham, W. J. What Do Classroom Teachers Really Need to Know about
Reliability? Retrieved from Dr. Gilbert Hutchcraft:
http://courses.fgcu.edu~144
Renaissance Learning. ATOS Analyzer,
http://www.renlearn.com/ar/atosanalyzer.htm
Renaissance Learning. (1999, April). The Design of Accelerated Reader
Assessments. Report from Renaissance Learning, 1-8.
Renaissance Learning. (2002). National Reading Studies Validate Accelerated
Reader, Reading Renaissance. Scientifically Based Research Abstract
Accelerated Reader and Reading Renaissance, L1542, 1-4.
Teacher Made Assessment Activity 14
Renaissance Learning. (2002, July). How Scientific Research Support the School
Renaissance School Improvement Process. Report from Renaissance
Learning.
Spencer, S. J. (1995, November). Modifying tests for diverse learners.
Educational tests & measurements, 31(2), p.84. Retrieved April, 2003,
from Academic Search Elite: http://207.203.214.140:2211
Stiggins, R. (2001, Fall). Educational Measurement: Issues and Practices.
Retrieved March, 2003, from Dr. Gilbert Hutchcraft:
http://courses.fgcu.edu~144
Willis, S. (1997, December). Using Assessment to Motivate Students.
Educational Update, 39(8). Retrieved March, 2003, from Dr. Gilbert
Hutchcraft: http://www.ascd.org/readingroom/edupdate/1997/1dec.html
Teacher Made Assessment Activity 15
Appendix A
Specifications:Table of Specification for book: Two Crows Counting
ObjectivesKnows Skill In Skill In Skill In Skill In Total
# items
Content Literal Comprehension
Selecting Use status of the
class or ratting scale to access.
Use status of the class or rating scale
to
Use Reading Log To
Evaluate Recording
of
VocabularyPractice and
Reinforcement
5 BookIn ZPD range
Dualog reading, students reading with a partner
Access students reading
independently
Title, LevelAnd
ScoreOn Quiz
by students
5
% Of Evaluation
50% 15% 15% 15% 5% 100%
The above table of specifications is the book Two Crows Counting by Doris Orgel. Knowledge, literal comprehension of the vocabulary in the book would be assessed through the Accelerated Reader Computer Program. Performances would be accessed through frequent monitoring and status of the class taken by
the classroom teacher.
The numbers in the cells represent the number of items measuring the content area and the type of target objective. There are 5 items on the whole test so the percentages show what percent of the test measures each type of objective (you
could show the same thing for each content area. The percentage of emphasis on the test should roughly match the percentage of emphasis given in instruction.
Teacher Made Assessment Activity 17
Appendix C
Big crow, small crow, on the wing, counting, counting everything...
1 ONE sun rising
1 Two shadows gliding
3 Three people rowing
4 Four trees blowing
5 Five cats lapping
6 Six socks flapping
7 Seven herons wading
8 Eight geese parading
9 Nine farmers haying
10 Ten children playing
Small crow, big crow, home the go, counting things they see below...
10 Ten children snoozing
9 Nine farmers snoring
8 Eight geese resting
7 Seven herons nesting
6 Six socks hardly stirring
5 Five cats purring
4 Four trees standing
3 Three boats landing
2 Two shadows slowing
1 One sun glowing
Big crow, small crow, dreaming deep, counting nothing fast asleep.
Teacher Made Assessment Activity 18
Appendix D
Renaissance Learning, Inc.
Wednesday, April 09, 2003 4:39:45 PM
MessageFrom: ATOS <[email protected]>
Subject: ATOS for book results (Confirmation # 8891)To: Margaret L. England
Attachments:
Thank you for your inquiry. The file you submitted was analyzed, and these are the results:
Confirmation Number: 8891Title of entry: Two Crows Counting
Author First Name: DorisAuthor Last Name: Orgel
ISBN: 083681617xATOS book level: 3.7AR Points: 0.5Total Word Count: 113
Renaissance Learning, Inc.
Appendix E