Successfor Allfor All - Educational Assessment · 2020-03-26 · Overview of commentsOverview of...

SuccessSuccess for Allfor All::SuccessSuccess for Allfor All::A personal view of educational A personal view of educational

challenges and research possibilitieschallenges and research possibilitieschallenges and research possibilities challenges and research possibilities from an assessment perspectivefrom an assessment perspective

Brian GongBrian GongggCenter for AssessmentCenter for Assessment

Presentation at the Annual Meeting of the NorthernPresentation at the Annual Meeting of the Northern Rocky Rocky gg yyMountain Educational Research Association Mountain Educational Research Association (NRMERA) (NRMERA)

October 4, 2012 Park City, UTOctober 4, 2012 Park City, UT

Overview of commentsOverview of commentsOverview of commentsOverview of comments

“Success for all students”“Success for all students”Success for all studentsSuccess for all students ResearchResearch

An example of research: conceptual clarity, evidence, andAn example of research: conceptual clarity, evidence, andAn example of research: conceptual clarity, evidence, and An example of research: conceptual clarity, evidence, and argument, done iteratively and collectivelyargument, done iteratively and collectively

How our research might be more useful and usedHow our research might be more useful and used

2Gong – NRMERA - "Success for All" - 10/4/12

OverviewOverviewOverviewOverview

How has “Success for All” been defined and pursued,How has “Success for All” been defined and pursued,How has Success for All been defined and pursued, How has Success for All been defined and pursued, from a national, measurement perspective?from a national, measurement perspective? Three major approaches, from 1960 to nowThree major approaches, from 1960 to now

Inclusive opportunityInclusive opportunity Attainment Attainment –– more and more and moremore and more and more More than attainmentMore than attainment More than attainmentMore than attainment

Role of measurementRole of measurement

How can research help “All” achieve “Success”?How can research help “All” achieve “Success”?pp ExemplifiedExemplified by NRMERA conferenceby NRMERA conference


A personal A personal story: Success as story: Success as Inclusive OpportunityInclusive Opportunity

Walter Gong (my father)Walter Gong (my father)g ( y )g ( y )


Walter GongWalter Gong -- 11Walter Gong Walter Gong 11

Born: 1923 MercedBorn: 1923 Merced CaliforniaCalifornia Born: 1923, Merced,Born: 1923, Merced, CaliforniaCalifornia Parents:Parents: Chee Gong and Shee Wong Chee Gong and Shee Wong

ELL i i ( l d k E li h)ELL i i ( l d k E li h) ELL immigrants (never learned to speak English)ELL immigrants (never learned to speak English) Low education in native language (3Low education in native language (3rdrd grade)grade)

BilingualBilingual Low SESLow SES

Parents operated a handParents operated a hand--laundry; later owned laundry; later owned butcher store,butcher store, then grocerythen grocery

Some racial discriminationSome racial discrimination5Gong – NRMERA - "Success for All" - 10/4/12

EducationaEducational “Success”l “Success”EducationaEducational Successl Success

High school graduation (1941)High school graduation (1941) High school graduation (1941)High school graduation (1941) PostPost--secondary training (Navy radar, 1942)secondary training (Navy radar, 1942) C ll d r STEMC ll d r STEM d ti n r rd ti n r r College degrees, STEMCollege degrees, STEM--education careereducation career

Taught high school scienceTaught high school science

B SB S M A Ed DM A Ed D B.S.,B.S., M.A., Ed.D., M.A., Ed.D., StanfordStanford

Natural Science faculty,Natural Science faculty, San José (CA) State UniversitySan José (CA) State University

C l f l d l d blC l f l d l d bl l il i Consultant on faculty development and problemConsultant on faculty development and problem--solving solving (e.g., BYU, IBM)(e.g., BYU, IBM)

3 children advanced degrees working in education3 children advanced degrees working in education 3 children advanced degrees, working in education3 children advanced degrees, working in education


Current national policyCurrent national policyCurrent national policyCurrent national policy

KK--12 education focused on helping almost all12 education focused on helping almost all KK 12 education focused on helping almost all 12 education focused on helping almost all students be “collegestudents be “college-- and careerand career--ready” by the ready” by the timetime the students graduate from high schoolthe students graduate from high schooltimetime the students graduate from high schoolthe students graduate from high school

N d fi i i f “ ” “ llN d fi i i f “ ” “ ll dd New definition of “success” = “collegeNew definition of “success” = “college-- and and careercareer--ready”ready”


Expanding goals forExpanding goals for public educationpublic educationExpanding goals for Expanding goals for public educationpublic education

“S ” Wh T d“Success” Who Targeted

3 R’s – Reading, ‘Riting, ‘Rithmetic Some3 R s Reading, Riting, Rithmetic Some

Equitable access & opportunity Specific subgroups

Basic job knowledge & skills Some

Minimum competency – high school Some

“Proficiency” All

Now: “College- and career-readiness” All


Now: College and career readiness All

How didHow did we get here?we get here?How didHow did we get here?we get here? Brown v. Board of Education (1954) Brown v. Board of Education (1954) –– inclusion/opportunityinclusion/opportunity

ESEA/TitleESEA/Title 1 (1965); 1 (1965); NormNorm--referenced tests; Minimum competency (1970referenced tests; Minimum competency (1970--80’s)80’s)

N i Ri k (1984) & “L k W b ” (1987)N i Ri k (1984) & “L k W b ” (1987) Nation at Risk (1984) & “Lake Woebegone” (1987)Nation at Risk (1984) & “Lake Woebegone” (1987) “All students can learn” and Standards“All students can learn” and Standards--basedbased

Fiscal equity/opportunity lawsuits (1990’s+)Fiscal equity/opportunity lawsuits (1990’s+) Fiscal equity/opportunity lawsuits (1990 s+)Fiscal equity/opportunity lawsuits (1990 s+) StateState--based, taxbased, tax increases, accountabilityincreases, accountability

IASA (1995) & IASA (1995) & No ChildNo Child Left BehindLeft Behind (2001) (2001) with their with their ( )( ) N dN d f df d ( )( ) wwfeatures and shortcomings features and shortcomings –– all students included; all students proficient by all students included; all students proficient by 2014;2014; equal attainment criteria for all subgroups; equal attainment criteria for all subgroups; individual state proficiency standardsindividual state proficiency standards

C C St t St d dC C St t St d d & t& t Common Core State StandardsCommon Core State Standards & common assessment & common assessment consortia; consortia; Race to the Top Race to the Top & & ESEA WaiversESEA Waivers (2010 to present)(2010 to present)


Success as AttainmentSuccess as AttainmentSuccess as AttainmentSuccess as Attainment

“Success” is reaching a defined level or “Success” is reaching a defined level or ggcondition,condition, typically marked by a valued typically marked by a valued educational indicator (e g test score gradeeducational indicator (e g test score gradeeducational indicator (e.g., test score, grade, educational indicator (e.g., test score, grade, admittance, completion, degree)admittance, completion, degree)

F KF K 12 f b hi h h l12 f b hi h h l Focus on KFocus on K--12 from concerns about high school 12 from concerns about high school graduation and college preparationgraduation and college preparation

MM i i di d b li i di d b l More successMore success is indicated by more people is indicated by more people reaching the “successful” levelreaching the “successful” level “Success for all”“Success for all”


U.S. High School GraduationU.S. High School GraduationU.S. High School GraduationU.S. High School Graduation

90% 85% 80% 75%

Figure XIII. Educational Attainment Decompositions, Males and Females 1900-1980 Birth Cohorts Graduate HS Attend College Attend Given HS

tage

75% 70% 65% 60% 55% 50%

Attend Given HS Graduate College Graduate Given Attend

Per

cent 45%

40% 35% 30% 25% 20%20% 15% 10% 5% 0%


Year of Birth

High School Graduation RatesHigh School Graduation Rates –– 22High School Graduation RatesHigh School Graduation Rates 22

9090

80

70

60

50 AFGR

Goldin‐Katz 40 Heckman‐Lafontaine

30 NLSY

20

10

1950

1953

1956

1959

1962

1965

1968

1971

197 4

1977

1980

1983

1986

1989

1992

1995

1998

2001

2004

2007

0

Figure 1: High School Graduation Rate for U.S., 1950-2008, as estimated by different methods


g g , , yusing different data sets.

HighHigh School Graduation RatesSchool Graduation Rates -- 33HighHigh School Graduation Rates School Graduation Rates 33


High School Graduation RatesHigh School Graduation Rates -- 44High School Graduation Rates High School Graduation Rates 44


Comparative High School Comparative High School Graduation RateGraduation Rate


Comparative College Graduation RateComparative College Graduation RateComparative College Graduation RateComparative College Graduation Rate

70Australia

50

60

AustraliaAustriaCanadaCzech RepublicDenmarkFinlandGermany

Percent first-time college (tertiary) graduates, by country, for 1995, 2000, 2003, 2008

40

50GreeceHungaryIcelandIrelandItalyJapanLuxembourg

20

30LuxembourgMexicoNetherlandsNew ZealandNorwayPolandPortugal

0

10

Slovak RepublicSpainSwedenSwitzerlandTurkeyUnited KingdomUnited States


1994 1996 1998 2000 2002 2004 2006 2008 2010 United States

OECD average

SuccessSuccess in the 1970’s and 80’sin the 1970’s and 80’sSuccessSuccess in the 1970 s and 80 sin the 1970 s and 80 s

NormNorm--referenced test performancereferenced test performance –– anan NormNorm referenced test performance referenced test performance an an evaluation indicator of successful inputsevaluation indicator of successful inputs “Lake Woebegone”“Lake Woebegone” every state and districtevery state and district Lake Woebegone Lake Woebegone –– every state and district every state and district

reported as performing above the national averagereported as performing above the national average Minimum competencyMinimum competency test performancetest performance Minimum competencyMinimum competency test performance test performance ––

attainment requirement for most students, e.g., attainment requirement for most students, e.g., high school graduation requirementhigh school graduation requirementhigh school graduation requirementhigh school graduation requirement Often established around grade 8 content and skills; Often established around grade 8 content and skills;

recognized as a low barrecognized as a low barrecognized as a low barrecognized as a low bar


StandardsStandards--based Proficiencybased ProficiencyStandardsStandards based Proficiencybased Proficiency

Learning standards (ends) same for all students;Learning standards (ends) same for all students; Learning standards (ends) same for all students; Learning standards (ends) same for all students; different supports to achieve goals (means)different supports to achieve goals (means)

AllAll students included appropriatelystudents included appropriately (( AllAll students included appropriately students included appropriately (e.g., (e.g., accommodations)accommodations)

Use of technologyUse of technology Use of technologyUse of technology Transform system constraintsTransform system constraints

F db kF db k h d/Diff i d i ih d/Diff i d i i FeedbackFeedback--enhanced/Differentiated instruction, enhanced/Differentiated instruction, (e.g., (e.g., RtI to achieve general curriculum; early warning dropout risk systems)RtI to achieve general curriculum; early warning dropout risk systems)

Disruptive/reform approaches,Disruptive/reform approaches, (e.g. technology(e.g. technology--enhancedenhanced Disruptive/reform approaches, Disruptive/reform approaches, (e.g., technology(e.g., technology enhanced enhanced delivery systems; marketdelivery systems; market--driven schools; “bolder” comprehensive early driven schools; “bolder” comprehensive early childhood supports)childhood supports)


States’ “Proficiency” drawbacksStates’ “Proficiency” drawbacksStates Proficiency drawbacksStates Proficiency drawbacks

Notably different from each other and NAEPNotably different from each other and NAEP

80

90

100

50

60

70

10

20

30

40State

NAEP

0

Mas

sach

uset

tsCo

nnec

ticut

ew H

amps

hire

Ver

mon

tCo

lora

doN

ew Je

rsey

Virg

inia

Mar

yland

Min

neso

taPe

nnsy

lvan

iaO

hio

New

Yor

kFl

orid

aK

entu

cky

Rhod

e Is

land

Miss

ouri

Neb

rask

aK

ansa

sM

onta

naN

orth

Dak

ota

Del

awar

eM

aine

Iow

aW

iscon

sinSo

uth

Dak

ota

Was

hing

ton

Wyo

min

gId

aho

Illin

ois

Indi

ana

Nor

th C

arol

ina

Ore

gon

Uta

hM

ichig

anG

eorg

iaA

rkan

sas

Tenn

esse

eA

labam

aSo

uth

Caro

lina

Texa

sO

klah

oma

Alas

kaW

est V

irgin

iaH

awaii

Ariz

ona

Nev

ada

Calif

orni

aM

ississ

ippi

New

Mex

ico

Loui

siana

ct o

f Col

umbi

a


N N S

Dist

ric

Percent Proficient or above, Grade 4 Reading, by state, 2009

States’ “Proficiency” drawbacksStates’ “Proficiency” drawbacks -- 22States Proficiency drawbacks States Proficiency drawbacks 22

High rate of high school graduates judged notHigh rate of high school graduates judged not High rate of high school graduates judged not High rate of high school graduates judged not ready for collegeready for college Placed into nonPlaced into non credit bearing initial course (e gcredit bearing initial course (e g Placed into nonPlaced into non--credit bearing initial course (e.g., credit bearing initial course (e.g.,

remedial mathremedial math, writing, English, writing, English): About 55% ): About 55% nationally of students in 2nationally of students in 2--year institutions, about year institutions, about yy y ,y ,25% of students in non25% of students in non--selective public 4selective public 4--year year institutionsinstitutions

LowerLower than desired college completion ratethan desired college completion rate


MultiMulti--statestate CollaborationCollaborationMultiMulti statestate CollaborationCollaboration

Common contentCommon content standardsstandards –– Common Core StateCommon Core State Common contentCommon content standards standards Common Core State Common Core State StandardsStandards Sponsored by NGA and CCSSOSponsored by NGA and CCSSO Sponsored by NGA and CCSSOSponsored by NGA and CCSSO ELA and mathematics, KELA and mathematics, K--1212 Adopted b 46 states to dateAdopted b 46 states to date Adopted by 46 states to dateAdopted by 46 states to date

Common assessments of CCSS Common assessments of CCSS –– common state common state iiassessment consortia assessment consortia (SBAC, PARCC for general population; (SBAC, PARCC for general population;

NCSC, DLM for students with severe cognitive disabilities; WIDA, and newly NCSC, DLM for students with severe cognitive disabilities; WIDA, and newly announced Oregon for English language proficiencyannounced Oregon for English language proficiency))g g g g p yg g g g p y))


College and Career Ready = College and Career Ready = Success?Success?

CollegeCollege-- and Careerand Career--ReadyReady--anchoredanchored CollegeCollege and Careerand Career ReadyReady anchored anchored StandardsStandards--basedbased Empirical (normative)Empirical (normative) Empirical (normative)Empirical (normative)


Recap: SuccessRecap: Success as attainmentas attainmentRecap: SuccessRecap: Success as attainmentas attainment

High school graduation and postHigh school graduation and post--secondary indicatorssecondary indicatorsHigh school graduation and postHigh school graduation and post secondary indicators secondary indicators led to interest in Kled to interest in K--12 achievement12 achievement

1970’s and 1980’s gave1970’s and 1980’s gave rise to normrise to norm--referenced (not referenced (not gg ((success for all) and minimum skills criterionsuccess for all) and minimum skills criterion--referenced referenced testing (low bar of success for all)testing (low bar of success for all)

1990’s gave rise to standards1990’s gave rise to standards--based definitions of based definitions of proficiency for all proficiency for all (including subgroups)(including subgroups), established by states, established by states

2010’s gave rise to common content standards and 2010’s gave rise to common content standards and attempts to develop common assessments linked to attempts to develop common assessments linked to “ ll“ ll dd di ” ( fi i ) f lldi ” ( fi i ) f ll“college“college-- and careerand career--readiness” (proficiency) for allreadiness” (proficiency) for all


Success as more than attainmentSuccess as more than attainmentSuccess as more than attainmentSuccess as more than attainment


Four Views of School Four Views of School PerformancePerformance

(Carlson, 2001; Gong, 2002)(Carlson, 2001; Gong, 2002) StatusStatus ChangeChangeStatusStatus ChangeChange

AchievementAchievement “Status”:“Status”: How highHow high “Improvement”:“Improvement”: IsIsAchievementAchievement Status :Status : How high How high do students in this school do students in this school score on state score on state assessments?assessments?

Improvement :Improvement : Is Is the performance of the performance of successive groups successive groups increasing from one year increasing from one year g yg yto the next?to the next?

EffectivenessEffectiveness “Growth”:“Growth”: Are Are “Acceleration”:“Acceleration”: Is Is individual students individual students learning as they progress learning as they progress from one grade to the from one grade to the

the school becoming the school becoming moremoreeffective or improving effective or improving more rapidly?more rapidly?


next?next?

Status, Growth, ImprovementStatus, Growth, ImprovementStatus, Growth, ImprovementStatus, Growth, Improvement

Year Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

2008 A B C D E F

2009 G A B C D E

2010 H G A B C D

Status

Imp2010 H G A B C D

2011 I H G A B C

2012 J I H G A B

provem

en

2013 K J I H G A

2014 L K J I H G

nt


Rank the schools, give reasonsRank the schools, give reasons, g, g

School 1

School 2

School Year Grade 2000 2001 2002

3 55 56 54 4 60 61 62


3 92 90 88 4 93 89 884 60 61 62

5 66 67 65

4 93 89 885 88 88 89

School 3

School Year Grade

2000 2001 2002

School 4 School Year Grade

2000 2001 20022000 2001 20023 65 70 71 4 66 70 72 5 63 65 68

2000 2001 20023 80 76 78 4 82 75 76 5 81 79 77


Status, Improvement, GrowthStatus, Improvement, Growth, p ,, p ,

School 1

School 2


3 55 56 54 4 60 61 62


3 92 90 88 4 93 89 884 60 61 62

5 66 67 65

4 93 89 885 88 88 89

Status 60.3 61.3 60.3Improvem. 1.0 (1.0)G h 6 5 5 0

Status 91.0 89.0 88.3Improvem. (3.0) (0.7)

School 3

School Year Grade

2000 2001 2002

School 4 School Year Grade

2000 2001 2002

Growth 6.5 5.0 Growth (4.0) (1.0)

2000 2001 20023 65 70 71 4 66 70 72 5 63 65 68

2000 2001 20023 80 76 78 4 82 75 76 5 81 79 77


Status 81.0 76.7 77.0Improvem. (4.3) 0.3Growth (4.0) 1.0

Status 64.7 68.3 70.3Improvem. 3.7 2.0Growth 2.0 0

Status and GrowthStatus and GrowthStatus and GrowthStatus and Growth

Types of Performance Status/Growth CombinationsTypes of Performance

High Status

Status Change

Status/Growth Combinations

ImprovementStatusAchieve-ment High/Low High/High

Growth AccelerationEffective-ness Low/Low Low/High

Low Growth High GrowthLow Status


StatusStatus and Growth and Growth -- 11

Stat

us

S

Growth

30Gong – NRMERA - "Success for All" - 10/4/12Betebenner, Jan. 2008, for RI project

NormNorm-- and criterionand criterion--referenced referenced interpretations over timeinterpretations over time


SGPSGP projections for different projections for different starting achievement pointsstarting achievement points


Applications of GrowthApplications of GrowthApplications of GrowthApplications of Growth

ValueValue--added Growthadded Growth ValueValue added Growthadded Growth Standardized,Standardized, (de)contextualized?(de)contextualized?

Ed t E l tiEd t E l ti Educator EvaluationEducator Evaluation Theory of action: Theory of action:

Eff i h h d ’ h/l iEff i h h d ’ h/l i Effective teachers change students’ growth/learning Effective teachers change students’ growth/learning trajectoriestrajectories

How to determine “effective” teachers?How to determine “effective” teachers?How to determine effective teachers?How to determine effective teachers? How to increase effectiveness of teachers?How to increase effectiveness of teachers? How to distribute effective teachers? (obtain, place, retain)How to distribute effective teachers? (obtain, place, retain)


Growth:Growth: Normative & toNormative & to--StandardsStandardsGrowth:Growth: Normative & toNormative & to StandardsStandards

Definition of “success” for “all”Definition of “success” for “all” Definition of success for allDefinition of success for all What is “good” performance? For whom?What is “good” performance? For whom?

Wh i “ d h” f ? FWh i “ d h” f ? F What is “good enough” performance? For What is “good enough” performance? For whom?whom?

Two general sources of referents:Two general sources of referents:gg Empirical (historical) performanceEmpirical (historical) performance Values and theoryValues and theoryValues and theoryValues and theory


Needed R&DNeeded R&D -- 11Needed R&D Needed R&D 11

Conceptual clarity and policy agreementConceptual clarity and policy agreement Conceptual clarity and policy agreementConceptual clarity and policy agreement More comprehensive conceptionsMore comprehensive conceptions of “success” and of “success” and

“all”; new models/values“all”; new models/values of what to attend toof what to attend toall ; new models/valuesall ; new models/values of what to attend toof what to attend to Growth;Growth; “learning progressions” for all“learning progressions” for all CollegeCollege--ready beyond academics; relationready beyond academics; relation to careerto career--readyreadygg y yy y yy Desired “spread” of performanceDesired “spread” of performance


Define desiredDefine desired growth (construct)growth (construct)Define desiredDefine desired growth (construct)growth (construct) Growth is increase in performance on the same thing, Growth is increase in performance on the same thing,

ddtowards mastery. towards mastery. Growth is learning one topic and then learning a more Growth is learning one topic and then learning a more

d d i i fd d i i fadvanced topic in a sequence of content.advanced topic in a sequence of content. Growth is increase in expertise on the same thingGrowth is increase in expertise on the same thing (e.g., (e.g.,

ability to apply or analyze due to more powerful mentalability to apply or analyze due to more powerful mentalability to apply or analyze due to more powerful mental ability to apply or analyze due to more powerful mental model, increased fluency, greater independence).model, increased fluency, greater independence).

Growth is increase in integration across content andGrowth is increase in integration across content and Growth is increase in integration across content and Growth is increase in integration across content and skills. skills.

Growth is increase of knowledge and skills outside theGrowth is increase of knowledge and skills outside the Growth is increase of knowledge and skills outside the Growth is increase of knowledge and skills outside the defined areas.defined areas.


CollegeCollege-- and Careerand Career--ReadyReadyCollegeCollege and Careerand Career ReadyReady

CollegeCollege--ready: More thanready: More than CollegeCollege ready: More than ready: More than academic knowledgeacademic knowledge and skillsand skillsCC d Hd H i il ?i il ? CareerCareer--ready: Howready: How similar?similar?

Domain College Career

N d iNon-academic knowledge and skills

Academic


Academic knowledge and skills

What is… What weWhat is… What we desiredesireWhat is… What weWhat is… What we desiredesire

What is (often normal distribution)What is (often normal distribution) What is (often normal distribution)What is (often normal distribution)


What is… What weWhat is… What we desiredesire -- 11What is… What weWhat is… What we desire desire 11

What is (often normal distribution)What is (often normal distribution) What is (often normal distribution)What is (often normal distribution)


What is… What we desireWhat is… What we desire –– 22What is… What we desire What is… What we desire 22

What do we desire variability of students to be inWhat do we desire variability of students to be inWhat do we desire variability of students to be in What do we desire variability of students to be in relation to each other and to arelation to each other and to a criterioncriterion--referenced referenced “good enough” standard?“good enough” standard?


Needed R&DNeeded R&D –– 22Needed R&D Needed R&D 22

New models and tools that provide for New models and tools that provide for structurally better ways to achieve our goalsstructurally better ways to achieve our goals More powerful and relevantMore powerful and relevant

Example: statistical models and studies that support Example: statistical models and studies that support dealing with dealing with greater greater contextualization and causal contextualization and causal attributionattributionattributionattribution

Disruptive Disruptive –– selfself--sustainingsustaining SystemicSystemic change the constraintschange the constraints Systemic Systemic –– change the constraintschange the constraints

Example: Early warning (and action) systemsExample: Early warning (and action) systems

Broader communication of research with practiceBroader communication of research with practice Broader communication of research with practice Broader communication of research with practice and policyand policy


Knowledge to actionKnowledge to actionKnowledge to actionKnowledge to action

CPRECPRE CPRECPRE Declarative vs. ProceduralDeclarative vs. Procedural

G li d C li dG li d C li d Generalized x ContextualizedGeneralized x Contextualized PurposePurpose--drivendriven Avenues of disseminationAvenues of dissemination (audience, access, (audience, access,

social)social)))


Needed R&DNeeded R&D –– 33Needed R&D Needed R&D 33

Accelerate learningAccelerate learning Accelerate learningAccelerate learning Move from descriptionMove from description to to diagnosis, from diagnosis diagnosis, from diagnosis

to prescription from prescription toto prescription from prescription toto prescription, from prescription to to prescription, from prescription to implementation, from local implementation to scaled implementation, from local implementation to scaled implementationimplementationpp BreakBreak predictions of low performancepredictions of low performance How can catch up work/keep up/move up work?How can catch up work/keep up/move up work? Feedback information for assessments to inform growthFeedback information for assessments to inform growth Great teachers,Great teachers, great leadersgreat leaders

I i l lI i l l Intentional learnersIntentional learners


Feedback FrameworksFeedback FrameworksFeedback Frameworks Feedback Frameworks Shute (2007)Shute (2007)

F db kF db k Feedback purposeFeedback purpose Cognitive Cognitive (& affective, social)(& affective, social) mechanisms & feedbackmechanisms & feedback Feedback specificityFeedback specificity Features of feedbackFeatures of feedback

Kulhavy & Stock (1989): verification, elaborativeKulhavy & Stock (1989): verification, elaborative Feedback complexity/lengthFeedback complexity/lengthp y/ gp y/ g Formative feedback as scaffoldingFormative feedback as scaffolding

Feeding back, feeding up, feeding forwardFeeding back, feeding up, feeding forward GoalGoal--directed feedback and motivationdirected feedback and motivation GoalGoal directed feedback and motivationdirected feedback and motivation TimingTiming Feedback and other variablesFeedback and other variables

Learner level response certitude goal orientation (Black & Wiliam:Learner level response certitude goal orientation (Black & Wiliam: Learner level, response certitude, goal orientation (Black & Wiliam: Learner level, response certitude, goal orientation (Black & Wiliam: directive, facilitative), normative feedbackdirective, facilitative), normative feedback


Feedback Types Arrayed Loosely by ComplexityFeedback Types Arrayed Loosely by Complexity(Shute, 2007 – computer-based assessment focus)

Feedback type Description

No feedback Refers to conditions where the learner is presented a question and is required to respond, but there is no indication as to the correctness of the learner’s response

V ifi ti Also called knowledge of results (KR), or knowledge of outcome, it informs the learner about the correctness of her

( p f )

Verification g f ( ), g f ,response(s), such as right/wrong or overall percentage correct.

Correct response Also known as knowledge of correct response (KCR), it informs the learner of the correct answer to a specific problem with no additional information.

Try-again Also known as repeat-until-correct feedback, it informs the learner about an incorrect response and allows the learner one or more attempts to answer the question. p q

Error-flagging Also known as location of mistakes (LM), error-flagging highlights errors in a solution, without giving correct answer.

Elaborated A general term, it refers to providing an explanation about why a specific response was correct, and it might allow the learner to review part of the instruction. It also might present the correct answer (see below for six types of elaborated feedback).

Attribute isolation Elaborated feedback that presents information addressing central attributes of the target concept or skill being studied.

Topic-contingent Elaborated feedback that provides the learner with information relating to the target topic currently being studied. This might entail simply re-teaching material.

Elaborated feedback that focuses on the learner’s specific response It may describe why the answer is wrong andResponse-contingent Elaborated feedback that focuses on the learner s specific response. It may describe why the answer is wrong and why the correct answer is correct. This does not use formal error analysis.

Hints/cues/prompts Elaborated feedback that guides the learner in the right direction (e.g., strategic hint on what to do next or a worked example or demonstration). It avoids explicitly presenting the correct answer.

Bugs/misconceptions Elaborated feedback that requires error analysis and diagnosis. It provides information about the learner’s specific errors or misconceptions (e g what is wrong and why)


g / p errors or misconceptions (e.g., what is wrong and why).

Informative tutoring The most elaborated feedback (from Narciss & Huth, 2004), this presents verification feedback, error-flagging, and strategic hints on how to proceed. The correct answer is not usually provided.

Educator EvaluationEducator Evaluation Systems & Systems & F db kF db kFeedbackFeedback

Current RTTT educator evaluationCurrent RTTT educator evaluation systemssystems Current RTTT educator evaluationCurrent RTTT educator evaluation systems systems focus on “sorting” teachers into a few categories focus on “sorting” teachers into a few categories A lot of information is aggregated into highly reliableA lot of information is aggregated into highly reliable A lot of information is aggregated into highly reliable A lot of information is aggregated into highly reliable

but not very useful for directing improvement, i.e., but not very useful for directing improvement, i.e., giving feedback at Shute’s “verification” levelgiving feedback at Shute’s “verification” levelg gg g AlsoAlso many other conditions not useful for improvement (e.g., timing, many other conditions not useful for improvement (e.g., timing,

motivation, agental/social responsibilities)motivation, agental/social responsibilities) Very much like student scores in annual summative stateVery much like student scores in annual summative state assessmentassessment Very much like student scores in annual summative stateVery much like student scores in annual summative state assessmentassessment

Educator systemsEducator systems to improve would include to improve would include evaluation designed to give feedback to helpevaluation designed to give feedback to helpevaluation designed to give feedback to help evaluation designed to give feedback to help improve improve (e.g., consider Shute’s other “higher complexity feedback types)(e.g., consider Shute’s other “higher complexity feedback types)


May we eachMay we each have success in our have success in our learning to researchlearning to research, ,

conducting of research andconducting of research andconducting of research, and conducting of research, and providing research to benefit providing research to benefit

others so that others so that more students may be successfulmore students may be successfulmore students may be successful.more students may be successful.


For more information:For more information:For more information:For more information:

Center for AssessmentCenter for Assessmentiiwww.nciea.org www.nciea.org

Brian GongBrian Gonga Go ga Go [email protected]@nciea.org


Acknowledgements and ReferencesAcknowledgements and ReferencesAcknowledgements and ReferencesAcknowledgements and References I I express my appreciation to the Northern Rocky Mountain Educational Research Association, Brett Foley, President, for the express my appreciation to the Northern Rocky Mountain Educational Research Association, Brett Foley, President, for the

invitation and theme that sparked preparation of this presentation. I acknowledge the input of my Center for Assessment invitation and theme that sparked preparation of this presentation. I acknowledge the input of my Center for Assessment colleagues with whom I’ve learned much over the past decade or more. Charlie DePascale especially provided constructive colleagues with whom I’ve learned much over the past decade or more. Charlie DePascale especially provided constructive g p p y pg p p y pcomments on this presentation. Opinions expressed and any shortcomings are my own.comments on this presentation. Opinions expressed and any shortcomings are my own.

Slide 11, High school graduation rate, birth cohort 1900Slide 11, High school graduation rate, birth cohort 1900--1980, graphic from: Heckman, J. J. & LaFontaine, P.A. (2007). The 1980, graphic from: Heckman, J. J. & LaFontaine, P.A. (2007). The American High School Graduation Rate: Trends and Levels. IZA DP No. 3216. Bonn, Germany: ForschungsinstitutAmerican High School Graduation Rate: Trends and Levels. IZA DP No. 3216. Bonn, Germany: Forschungsinstitut

zur Zukunft der Arbeit / Institute for the Study of Labor.zur Zukunft der Arbeit / Institute for the Study of Labor. Slide 12, High school graduation rates, four sources: Murname, R. J. (2011). U.S High School Graduation Rates: Patterns andSlide 12, High school graduation rates, four sources: Murname, R. J. (2011). U.S High School Graduation Rates: Patterns and Slide 12, High school graduation rates, four sources: Murname, R. J. (2011). U.S High School Graduation Rates: Patterns and Slide 12, High school graduation rates, four sources: Murname, R. J. (2011). U.S High School Graduation Rates: Patterns and

Explanations. Downloaded from Explanations. Downloaded from http://www.newyorkfed.org/research/education_seminar_series/Murnane_12_05_11.pdfhttp://www.newyorkfed.org/research/education_seminar_series/Murnane_12_05_11.pdf.. Slide s13 and 14, Declining high school graduation rate, and racial/ethnic subgroup differences: in “Closing the gap betweenSlide s13 and 14, Declining high school graduation rate, and racial/ethnic subgroup differences: in “Closing the gap between hihigh gh

school and college,” published by the Blackboard Institute (no date). Cited source: school and college,” published by the Blackboard Institute (no date). Cited source: Diplomas Count,Diplomas Count, 2010, p.23. Downloaded from 2010, p.23. Downloaded from http://www.blackboard.com/CMSPages/GetFile.aspx?guid=6c5e9639http://www.blackboard.com/CMSPages/GetFile.aspx?guid=6c5e9639--db0edb0e--4caf4caf--8f508f50--7d801c4969af7d801c4969af..

Slide 15 High school graduation rates 2008 OECD countries source: of graphic: Science and engineering indicators 2012 cSlide 15 High school graduation rates 2008 OECD countries source: of graphic: Science and engineering indicators 2012 citiitingng Slide 15, High school graduation rates, 2008, OECD countries, source: of graphic: Science and engineering indicators, 2012, cSlide 15, High school graduation rates, 2008, OECD countries, source: of graphic: Science and engineering indicators, 2012, citiiting ng data source: OECD data source: OECD Education at a GlanceEducation at a Glance, OECD Indicators (2010), OECD Indicators (2010)

Slide 16, Tertiary (college) completion rates, OECD countries, 1995, 2000, 2003, & 2005: table by author; data from Slide 16, Tertiary (college) completion rates, OECD countries, 1995, 2000, 2003, & 2005: table by author; data from http://nces.ed.gov/pubs2012/2012026/tables/table_23.asphttp://nces.ed.gov/pubs2012/2012026/tables/table_23.asp

Slide 19, States’ “Proficiency” drawbacks, Slide 19, States’ “Proficiency” drawbacks, , graph by author; data from , graph by author; data from http://nces ed gov/nationsreportcard/studies/statemapping/2009 naep state table asphttp://nces ed gov/nationsreportcard/studies/statemapping/2009 naep state table asp;; SOURCE: U S Department ofSOURCE: U S Department ofhttp://nces.ed.gov/nationsreportcard/studies/statemapping/2009_naep_state_table.asphttp://nces.ed.gov/nationsreportcard/studies/statemapping/2009_naep_state_table.asp; ; SOURCE: U.S. Department of SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2009 Reading and Mathematics Assessments. SY 2008Progress (NAEP) 2009 Reading and Mathematics Assessments. SY 2008--09 Consolidated State Performance Report Part I 09 Consolidated State Performance Report Part I (retrieved on 01/18/2011 from (retrieved on 01/18/2011 from http://www2.ed.gov/admins/lead/account/consolidated/sy08http://www2.ed.gov/admins/lead/account/consolidated/sy08--09part109part1))

Slide 20: college remediation rates: see, for example, Attewell, P. Lavin, D., Thurston, D. & Levey, T. (2006). New Evidence Slide 20: college remediation rates: see, for example, Attewell, P. Lavin, D., Thurston, D. & Levey, T. (2006). New Evidence on on College Remediation. College Remediation. The Journal of Higher Education, Vol. 77, No. 5 (September/October 2006). Downloaded from The Journal of Higher Education, Vol. 77, No. 5 (September/October 2006). Downloaded from Co ege e ed at o .Co ege e ed at o . The Journal of igher duca ion, Vol. 77, No. 5 (Sep ember/Oc ober 006). ownloaded fromThe Journal of igher duca ion, Vol. 77, No. 5 (Sep ember/Oc ober 006). ownloaded fromhttp://www.gse.uci.edu/person/domina_t/docs/JHE%20remediation%20final.pdfhttp://www.gse.uci.edu/person/domina_t/docs/JHE%20remediation%20final.pdf. .


Acknowledgements and ReferencesAcknowledgements and References -- 22Acknowledgements and References Acknowledgements and References 22

Slide 21:, for information on the Slide 21:, for information on the Common Core State Standards,Common Core State Standards, see see www.corestandards.org/www.corestandards.org/. For information on the federally . For information on the federally funded assessment development consortia see: U.S. Department of Educationfunded assessment development consortia see: U.S. Department of Education -- http://www2.ed.gov/programs/racetothetophttp://www2.ed.gov/programs/racetothetop--funded assessment development consortia, see: U.S. Department of Education funded assessment development consortia, see: U.S. Department of Education http://www2.ed.gov/programs/racetothetophttp://www2.ed.gov/programs/racetothetopassessment/awards.htmlassessment/awards.html; PARCC ; PARCC -- http://www.parcconline.org/http://www.parcconline.org/; SBAC ; SBAC -- http://www.smarterbalanced.org/http://www.smarterbalanced.org/; DLM ; DLM --http://dynamiclearningmaps.org/http://dynamiclearningmaps.org/; NCSC ; NCSC -- http://www.ncscpartners.org/http://www.ncscpartners.org/ ; WIDA ; WIDA --http://www.wida.us/assessment/assets.aspxhttp://www.wida.us/assessment/assets.aspx; ELP assessment announced 10/12, with Oregon as lead state, ; ELP assessment announced 10/12, with Oregon as lead state, http://www2.ed.gov/programs/eag/awards11.htmlhttp://www2.ed.gov/programs/eag/awards11.html..

Slide 25, Four views of school performance: Carlson, D. (2001). Focusing State Educational Accountability Systems: Four Slide 25, Four views of school performance: Carlson, D. (2001). Focusing State Educational Accountability Systems: Four , p , ( ) g y y, p , ( ) g y ymethods of judging school quality and progress. Downloaded from: methods of judging school quality and progress. Downloaded from: http://nciea.org/publications/Dale020402.pdfhttp://nciea.org/publications/Dale020402.pdf; Gong, B., et ; Gong, B., et al. (2002). Designing School Accountability Systems: Towards a framework and process. al. (2002). Designing School Accountability Systems: Towards a framework and process. http://programs.ccsso.org/content/pdfs/designing_school_acct_syst.pdfhttp://programs.ccsso.org/content/pdfs/designing_school_acct_syst.pdf. .

Slides 27, and 28 Rank the schools example: developed by Richard Hill, personal communication.Slides 27, and 28 Rank the schools example: developed by Richard Hill, personal communication. Slide 30, Status and Growth: from Betebenner, D.Slide 30, Status and Growth: from Betebenner, D., ,, , Slides 31Slides 31--32, Norm32, Norm-- and criterionand criterion--referenced interpretations of growth, graphic from Betebenner, D. For an excellent discussioreferenced interpretations of growth, graphic from Betebenner, D. For an excellent discussion, n,

see Betebenner, D.see Betebenner, D. Slide 37, CollegeSlide 37, College--ready ready –– more than academic knowledge and skills, from Conley, D. T. (2007). more than academic knowledge and skills, from Conley, D. T. (2007). Redefining College Readiness.Redefining College Readiness. Eugene, Eugene,

OR: Educational Policy Improvement Center. OR: Educational Policy Improvement Center. Slide 37 CollegeSlide 37 College and Careerand Career readyready How similar? graphic from Center for Assessment RILS 2013 development groupHow similar? graphic from Center for Assessment RILS 2013 development group Slide 37, CollegeSlide 37, College-- and Careerand Career--ready ready –– How similar?, graphic from Center for Assessment RILS 2013 development group , How similar?, graphic from Center for Assessment RILS 2013 development group ,

especially C. Domaleski.especially C. Domaleski. Slide 46, Knowledge to action: CPRE analysis personal communication.Slide 46, Knowledge to action: CPRE analysis personal communication. Slides 48Slides 48--49, Feedback Frameworks: Shute, V. J. (2007). Focus on formative feedback. ETS Research Report RR49, Feedback Frameworks: Shute, V. J. (2007). Focus on formative feedback. ETS Research Report RR--0707--11. 11.

Princeton, NJ: Educational Testing Service. Princeton, NJ: Educational Testing Service.


Successfor Allfor All - Educational Assessment · 2020-03-26 · Overview of commentsOverview of...

Documents

Transcript of Successfor Allfor All - Educational Assessment · 2020-03-26 · Overview of commentsOverview of...