Issues in Using Assessments in Accountability Systems for ...ectacenter.org/~pdfs/eco/OSEP...
Transcript of Issues in Using Assessments in Accountability Systems for ...ectacenter.org/~pdfs/eco/OSEP...
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issues in Using Assessments in Accountability Systems for
Young Children with Disabilities
Issues in Using Assessments Issues in Using Assessments in Accountability Systems for in Accountability Systems for
Young Children with Young Children with DisabilitiesDisabilities
Kathy HebbelerECO at SRI International
OPEP Think Tank on AssessmentWashington, DC February 2007
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
ObjectiveObjective
Provide background information relevant to the discussion, including what the ECO Center has learned about the current state of assessment practices through our work with states
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
AssumptionsAssumptions
Children 0-5 participating in IDEA programs also will be participating in the required OSEP reporting (approx. 1 million children)Some of these children also may be participating in other assessment systems
OSEP Reporting
Child Care
Head Start
State Preschool
Participation in Multiple
Accountability Systems??
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Possible uses of assessment in Possible uses of assessment in EI/ECSEEI/ECSE
Eligibility determination Norm-referenced test % delay
Individual program planningCurriculum-based tools – e.g., LAP, Carolina
Ongoing progress monitoringCurriculum-based tools?
Accountability assessmentE.g., Head Start reporting system
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Variation in how assessments Issue: Variation in how assessments are usedare used
Do not know how many EI/ECSE programs routinely use assessment tools for anything other than eligibilityIn some programs, the only formal assessment is for eligibilityAppears to be much state-to-state variation
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Thinking on assessment has Issue: Thinking on assessment has changed in general early childhood changed in general early childhood communitycommunity
Major changes in last 15 years in how assessment of young children is viewedOld position: Do not test little kidsNew position: Ongoing assessment is part of a high quality early childhood program
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
What changedWhat changed
New and different tools became available for general EC
Curriculum-based assessments were developed, e.g., Creative Curriculum, Work Sampling, etc. Tools for 3-5 came first; 0-3 tools are coming now
Interesting sidebar: Curriculum-based assessments for programs serving children 0-5 with disabilities have been around for years
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
What changedWhat changed
The purpose of assessment was redefinedNot about: sorting, labeling, using to deny accessNow about: Getting a rich picture of what children can do and can’t do and using that information to help them acquire new skills
“progress monitoring”
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
What changedWhat changed
Assessment had always been seen as a process with multiple purposesDistinctions have been made between good and bad uses of assessment with young childrenGood uses are now promotedFor more information: NAEYC web site (Position statement on Curriculum, Assessment and Evaluation)
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Interesting IronyInteresting Irony
Even though the disability community had developed many curriculum-based assessment tools, currently [many? some?] programs do not practice ongoing assessmentThe push for ongoing assessment to monitor how a child is doing and plan for instruction/intervention is coming from the general education community
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Use of existing Issue: Use of existing assessments/multiple purposesassessments/multiple purposes
Can assessments already being used by programs for other purposes (whatever they are) be used for accountability purposes?Does this apply to all tools or only some categories of tools?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Limitations of existing Issue: Limitations of existing assessment toolsassessment tools
“Assessment of young children poses greater challenges than people generally realize….assessment results—in particular, standardized tests, that reflect a given point in time—can easily misrepresent children’s learning…There is widespread dissatisfaction with traditional norm-referenced standardized tests which are based on early 20th century psychological theory.”
National Research Council, 2001
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Problem: Nature of the young Problem: Nature of the young childchild
Not well suited to a standardized testing situationPerformance varies from day to day, place to place, person to person Don’t perform well for strangers or on demand Growth is sporadic and uneven
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Problem: Response capabilities Problem: Response capabilities of children with disabilitiesof children with disabilities
Same issue as with school-age children: assessment assumes child who can see, hear and understand spoken language, point, etc.Few assessments include accommodationsVery little data on validity of accommodations with young children
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Problem: Impact of Problem: Impact of disability/delay on development disability/delay on development
Typically developing children tend to develop in multiple areas simultaneously
Language, cognition, motor skills march forward more or less togetherEven though development has been divided into domains for assessment and research, much of development is intertwined These interconnections present challenges for obtaining a “pure” domain score
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Problem: Impact of Problem: Impact of disability/delay on development disability/delay on development
More difficult to accurately portray the development of children developing atypically with available assessments, esp. children with language delays
Do they understand the directions?Is the assessment tapping cognition or language?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Problem: Psychometric properties of Problem: Psychometric properties of existing instrumentsexisting instruments
Some of the most common instruments are being used with limited or no reliability and validity data None have validity or reliability data reporting when used for outcomes and accountability
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Response: New forms of Response: New forms of assessmentassessment
Growing recognition that the only way to get a valid picture of what a child can do/does to is look at performance over a variety of settings and people including what the child does spontaneously with familiar adults Can’t base conclusions about child’s capabilities on elicited responses alone “Authentic assessment”
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Response: Use multiple Response: Use multiple sources of information (best sources of information (best practice)practice)
“A single test, person, or occasion is not a sufficient source of information. This means that we must gather information from several sources, instruments, settings and occasions to produce the most valid description of the child’s status or progress”---DEC Recommended Practices
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Multiple sources of Issue: Multiple sources of informationinformation
And just how is that information supposed to be put together?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Validity vs. credibility Issue: Validity vs. credibility dilemma for accountabilitydilemma for accountability
Strangers can’t elicit valid data on young children’s performance capabilities in a testing situation
BUTcan data produced by those who know
the child and whose programs are being evaluated, be credible in an accountability system?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Validity and reliabilityIssue: Validity and reliability
Are not characteristics of an assessment per se Validity – context dependent on the use of the results
Individual vs. group decisions
“Validity , the degree that an assessment measures what it purports to measure, relates to the use of the test, rather than the test itself.”
Score Reliability, Pg. 113
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Reliability and validityIssue: Reliability and validity
Reliability is a characteristic of a set of scores, not of a test“…reliability refers to the degree of consistency of the information obtained from an information gathering process”“…reliability of the scores provided by an instrument or procedure may fluctuate depending on how, when, and to whom the instrument or procedure is administered..”Joint Committee on Standards for Educational
Evaluation, 1994Quoted in Score Reliability, pg.95
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Validity in an Issue: Validity in an accountability systemaccountability system
Do the assessment results lead to the “right” decisions?
CCSSO framework
How does one measure validity in an accountability system?What should be the measures of the validity of the OSEP data?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Existing assessments are Issue: Existing assessments are domainsdomains--basedbased
OSEP outcomes are functional and cut across domainsExisting assessments provide scores for domains, not 3 outcomesExisting assessments vary in the extent to which they assess functioning vs. isolated skills
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Outcomes Are FunctionalOutcomes Are Functional
Functional outcomes: Refer to things that are meaningful to the child in the context of everyday livingRefer to an integrated series of behaviors or skills that allow the child to achieve the important everyday goals
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Functional Outcomes are Functional Outcomes are NOTNOT
A single behavior
The sum of a series of discrete behaviors or splinter skills such as…..
*Knows 10 words *Pincer grasp (picks up a raisin)
*Smiles at mom
*Stacks 3 blocks
*Goes up and down stairs with one foot on each stair
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Variation in provider Issue: Variation in provider knowledge of assessmentknowledge of assessment
(Based on ECO work with states)Some providers are skilled in administering and interpreting multiple assessments, some in one, some rarely use any.Many children served in programs for typically developing children where knowledge and use of assessment is limited. How will providers be trained and supervised?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issues related to OSEP Issues related to OSEP accountability accountability
States must report data on all children with IFSP/IEPs in the 3 outcomes areas, even if the child has no problems in an area and is not receiving services
E.g., many preschoolers are receiving services only for a communication problem, possibly only an articulation problem, so they are not routinely assessed in other areas (even for eligibility at program entry)
Can a screening tool be used in an accountability system to verify that children are maintaining functioning comparable to same aged peers in those areas?
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Role of families in the Issue: Role of families in the assessment processassessment process
Families provide a unique perspective on the child’s functioningFew assessment tools have procedures for incorporating that perspectiveNeed good tools/procedures for learning about child from the family
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Issue: Role of families in the Issue: Role of families in the assessment processassessment process
Programs vary in how much and how they share assessment data with families, especially with regard to communicating developmental ages or extent of a child’s delay.Some providers are “soft pedaling” the assessment resultsProviders need training in sharing results with families
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Bottom LineBottom Line
States will be reporting data on outcomesThese data could be critical to federal level (and, in many states, state level) policy decisions related to Part C and Part B Preschool in the future
Early Childhood Outcomes CenterEarly Childhood Outcomes Center
Bottom LineBottom Line
What needs to happen to make sure assessments make a meaningful contribution to improved outcomes and program improvement?What can be done to insure assessment data used for outcomes measurement are:
MeaningfulValidReliableCredible?