Vocabulary*Assessment*Study*in*Educa5on* … · 2019-06-10 ·...
36
Vocabulary Assessment Study in Educa5on (VASE): Using Mul5dimensional Testlets to Provide Diagnos5c and Summa5ve Assessments of Vocabulary Growth in Grades 4 and 5 Judith A. Sco- and Susan Leigh Flinspach University of California, Santa Cruz Jack L. Vevea University of California, Merced
Transcript of Vocabulary*Assessment*Study*in*Educa5on* … · 2019-06-10 ·...
Vocabulary Assessment Study in Educa5on (VASE): Using Mul5dimensional Testlets to Provide Diagnos5c and Summa5ve
Assessments of Vocabulary Growth in Grades 4 and 5
Judith A. Sco- and Susan Leigh Flinspach University of California, Santa Cruz
Jack L. Vevea University of California, Merced
http://vineconsortium.org/vase/
US Dept of Education /IES/NCER
Reading & Writing Research Grant
Goal 5: Measurement
#R305A090550
h-p://vineproject.ucsc.edu US Dept of EducaGon /IES/NCER Reading & WriGng Research Grant Goal 2: Development
http://vineproject.ucsc.edu
US Dept of Education /IES/NCER
Reading & Writing Research Grant
Goal 2: Development
Goal of the VINE project
To co-construct, with teachers, an intervention that helped elementary (grade 4 & 5) students, particularly ELs and those traditionally underserved by schools, develop word consciousness and an understanding of how to use academic language as a tool of communication.
Word Consciousness
• an interest in and awareness of words as building blocks of communicaGon
• metacogniGve, metalinguisGc and affecGve aspects of word learning
• teaches generaGve vocabulary skills across the curriculum with no predetermined word set
Vocabulary Assessment Needed
We wanted to measure growth in vocabulary as a result of the VINE intervenGon.
Vocabulary Assessment Needed
Commonly used measures of vocabulary fail to:
• adequately assess words associated with the curriculum that students study
• take into account levels of knowledge that students might know about the words being tested.
Pearson, Hiebert & Kamil, 2007
Vocabulary Assessment Needed
This was an unmapped and ill-‐defined domain for assessment
The VINE Tests
• We selected novels and textbooks used in local 4th-‐ and 5th-‐grade classrooms
• Had VINE researchers and local teachers form a word corpus by idenGfying words that were potenGally difficult for students
• Reduced that word corpus to words with word families represented in more than one source text
The VINE Tests
• We balanced our word selecGon from the reduced corpus so that the words tested reflected variability in part of speech, morphological family, and word family frequency.
• We developed two tests—one with words from ficGon sources and the other with words from non-‐ficGon sources.
• 36 mulGple choice items on each.
The VINE Tests
• To measure some of the effects of the VINE intervenGon, we gave the tests in fall & spring.
• The tests were given in the classrooms in a 15-‐minute Gmed slot.
The VINE Tests
Each item contained 5 ?s that asked: – If students can idenGfy the word – How well they know the word – If they can idenGfy its semanGc field – If they can idenGfy its definiGon – If they can idenGfy its part (s) of speech
Analyzed items as “testlets” with grouped responses for each word using an IRT approach (Thissen,1989) and Samejima’s (1996) graded IRT model
The VINE Tests
• Marginal reliabiliGes very high:
.90 – ficGon .92 -‐ nonficGon • Difference in growth rate is robust and highly significant (VINE vs. Control classroom).
2007-‐08 (t611 =3.34, p= .001)
2008-‐09 (t918=2.675, p = .008)
• VINE intervenGon was parGcularly effecGve for English Learners: Effect size: .41 & .3
The VINE Tests
VINE developed a vocabulary assessment that:
• Reflected local grade-‐level materials
• Paid a-enGon to levels of word knowledge • Was staGsGcally reliable
Sco>, Hoover, Flinspach, & Vevea 2008
• Was effecGve at showing growth
Sco>, Miller, Flinspach, & Vevea, 2010 & 2011
h-p://vaseproject.ucsc.edu US Dept of EducaGon /IES/NCER
Reading and WriGng Research Grant Goal 5: Measurement
Goals of the VASE Assessment
To develop a test that is valid, reliable and useful on a broad scale (across California) with a great number of students.
To create linked grade 4 and 5 tests to show growth in mulGple a-ributes of vocabulary knowledge over Gme.
To provide both summaGve and diagnosGc informaGon about types of vocabulary knowledge and growth.
Developed a corpus of 4th & 5th grade words
• Sources of vocabulary words: – Selected 19 CA-‐adopted textbooks (math, science, socials, ELA), 21 Newbery Medal books, and the CA ELA standards for 4th and 5th grades
• Hired 8 elementary teachers who idenGfied over 39,000 words that are “unfamiliar” or “conceptually new” for fourth and fioh graders from these sources.
Developing VASE Word Bank
word corpus
word bank
• Word corpi were narrowed by subject
• Word bank:
– Words found in more than one textbook
– Inflected, derived, and related variants of words already in the word bank were added
Word Bank
• 1,910 science words, • 1,022 math words,
• 10,567 social studies words, • 9,855 English words • Narrowed to 625 words for item wriGng by an algorithm that considered: grade, number of sources, the subject(s), word frequency, morphological family size, a ceiling indicator of difficulty
Test development
• Pilot tested items for 324 words (9 tests of 36 items) with about 4000 students from schools across the state.
• Created 5 versions of the 24 item test.
Types of knowledge tested in VASE:
Level of Confidence/Metacogni=on:
How well do you know the word official?
O I’m sure I know what this word means.
O I think I know what this word means.
O I am not sure what this word means.
O I don’t know what this word means.
Types of knowledge tested in VASE:
Related Words (Seman=c Field):
I think that these words are related to official:
O Slowly, inefficient, dragging on
O SuggesGng, guidance, informaGve
O Jewelry, ornamental, wearing
O Righqul, endorsed, permission
Types of knowledge tested in VASE:
Morphology
Which word do you think is a correct word connected in meaning to official:
O Reofficial
O Officialen
O Unofficial
O Officialest
Words in Context (Syntax)
Choose the sentence that you think makes the most sense.
O There is an official website for the new movie.
O Any ouqit looks fancy when you put on a fine official.
O The teacher wanted to official the student about going to college.
O Jessica is so official that it takes her hours to do a simple job.
Types of knowledge tested in VASE:
Defini=on I think the word official is closest in meaning to:
O Giving good advice to someone
O Unable to finish something in a reasonable amount of Gme
O Approved by someone in authority
O A large decoraGve pin or clasp
SOME WORDS HAVE ONE PART OF SPEECH, BUT SOME HAVE MORE THAN ONE. FILL IN THE CIRCLE BESIDE THE CORRECT ANSWER OR ANSWERS.
I think the word official may be:
O A noun
O A verb
O An adjecGve
O An adverb
Parts of Speech
VASE Assessment
• 24-‐word mulGple-‐choice tests • 6 sub-‐items per word (a testlet)
• 4 linked tests, with 2 equated forms per grade
• 15 minutes
• Teacher administered; some accommodaGons
• Scores reported on a common scale that spans the two grades, allowing for longitudinal tracking of vocabulary growth
VASE Assessment
• Testlet: grouped responses to the 6 sub-‐items for each word
• With the excepGon of part of speech, each sub-‐item is scored as 0 or 1
• The range of scores for part of speech is 0-‐3 • Analysis using Samejima’s (1996) graded IRT model with a range of scores from 0-‐8 for each testlet
VASE Assessment
• Reliability – Final scaling sGll pending, but iniGal results have marginal reliability in the .91 to .93 range
• Validity – Convergent and divergent validity will be established by analysis with California Standards Tests
VASE Assessment
• Validity – Concept validity derives from: 1. the word-‐selecGon process, drawing words
from commonly used classroom materials 2. the on-‐going involvement of experienced
fourth-‐ and fioh-‐grade teachers in all stages of the assessment’s development: test design, word selecGon, item wriGng and selecGon, test administraGon pracGces, setng performance levels, and preparaGon of final test materials
VASE Assessment
• Validity of the testlet as an indicator of depth of word knowledge – Confirmatory factor analyses validate the concept of a second-‐order word knowledge factor consisGng of metacogniGon, semanGcs, morphology, syntax, and definiGon
– Part of speech fits more poorly but is included because it enhances informaGon about the most proficient students
VASE Assessment
• What’s leo to do? – Doing the final scaling this month
– Analyzing growth on the 2012-‐13 tests – Developing the final reporGng formats – ConducGng an analysis of DIF for gender – ConducGng a validity study using results from the California Standards Tests
– Preparing the final test materials
Summary
The VASE assessment is breaking new ground in vocabulary tesGng.
We believe that the VASE assessment has the potenGal to drive vocabulary instrucGon across the curriculum be-er than other current assessments of vocabulary knowledge in these grades.
Ques=ons?
VASE Website: hOp://vineconsor=um.org/vase/ VASE office: 831-‐459-‐2239
Judy Sco-, [email protected];
Susan Flinspach, [email protected];
Jack Vevea, [email protected]
