The Road More Traveled: Next Generation Assessments and the State of All Six Multi-State Consortia

Wednesday, June 24, 2015: 1:00 PM-2:30 PMHarbor B (Manchester Grand Hyatt San Diego)

The contents of this presentation were developed under a grant(s) from the U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education and you should not assume endorsement by the Federal government.

Introductions & Agenda Robert Crawford, West Virginia

The Panel

•Smarter Balanced Assessment Consortium: Jamika Burge, Director of Assessment Technology Product and Research, jamika.burge@smarterbalanced.org •PARCC: Chloe Torres, Associate Director of Technology, ctorres@parcconline.org•DLM: Neal Kingston, Director, Achievement and Assessment Institute, University of Kansas, nkingsto@ku.edu•NCSC: Chris Domaleski, Senior Associate, National Center for the Improvement of Educational Assessment, cdomaleski@nciea.org•WIDA: Carsten Wilmes, Assessment Director, WIDA Consortium, wilmes@wisc.edu•ELPA21: Wes Bruce, Field Test and Technology Readiness Team Lead, wesbruce3@gmail.com

Moderator•Robert Crawford, WV

Agenda

•With online administration and new technologies being implemented in schools and districts across the country, we will explore what’s working and where we need to focus attention for the next school year. •Consortia Overviews• System Development Update•What’s Working• Remaining Challenges•Guiding Questions•Audience Discussion with Panelists

Smarter Balanced Assessment ConsortiumJamika Burge, Director of Assessment Technology Product and Research, jamika.burge@smarterbalanced.org

System Development Update•Reporting - Data Warehouse • California (Interim) • Measured Progress States

(Summative) 

•Digital Library • 15 States• 2700+ instructional and professional

learning resources

•All specifications and code for major components available• SmarterApp.org

•Item Authoring  

•Test Authoring 

•Test Delivery • NV, MT, ND (Measured Progress)

•Scoring • Machine Scoring Engine • Measured Progress states • Michigan (DRC) • Missouri (CTB and DRC) • Wisconsin (ETS) 

• Teacher Hand-Scoring (Interim) • AIR states (in conjunction with

proprietary test delivery) 

What’s Working:Innovative Features

•Constructed Responses: Graphing, Equations

•Accessibility Features: Text-To-Speech, Refreshable Braille, Tactile Graphics

•Computer Adaptive Tests• Open source flexibility• SmarterApp Assessment Item Format

published under open license (Creative Commons)

• Securing enhancement procurement through RFPs• Increasing the pool of those who would

support development and investment

• Data standards aligned to CEDS

Remaining Challenges

•Deploying open source across vendors•Packaging assessments for multiple vendors•Open documents that are shared among vendors; Different from IMS standards

•Efficient assessment production system•Sustaining pace of innovation•Focus on reliability, scalability, and deployment of systems•Assessment production system is highly sociotechnical (people, processes, and technology)

Partnership for Assessment of Readiness for College and CareersChloe Torres, Associate Director of Technology, PARCC Inc.ctorres@parcconl ine.org

System Development Update

•Summative Administration Platform: Complete•PearsonAccessnext /TestNav 8 through 2017/18•Ongoing enhancements each administration year

•Other components: In Progress•Data Management and Reporting System (DMRS)•Assessment Delivery System* (ADS)•Partnership Resource Center (PRC)

What’s Working

•Stable and reliable summative platform implementation•Over 15 million tests completed online (PBA and EOY combined)•Our busiest day: 1.2 MILLION students; 204K students in an hour•NO SYSTEM OUTAGES

•Higher than expected online participation•Collaboration with partners to improve accessibility

Remaining Challenges

•Dependency on vendor roadmaps and cross-vendor coordination• PARCC priorities vs. core product priorities• Release timelines not always in sync with implementation requirements

•Interoperability•QTI/APIP compliant – Yes! Interoperable – Maybe

•Balancing innovation and practical implementation•Customer Support•Twitter

Neal KingstonDirector, Dynamic Learning Maps Alternate Assessment Consort iumDirector, Achievement and Assessment Inst i tute, University of KansasProfessor, Educational Psychology, University of Kansasnkingsto@ku.edu

Dynamic Learning Maps

System Development Update

•As of 6/8/2015•749,664 test sessions•66,388 students•18,436 teachers•10,631 schools•3,494 districts

•Focus now is on scoring, standard setting, reporting, technical documentation, and final system enhancements.

What’s Working

•Administration in general•Fitting diagnostic classification model•Design of next generation score reports

Remaining Challenges

•‘Administration’ functions continue to be the most prevalent areas requiring user support (e.g. adding/ editing organizations, reassigning students, reopening closed tests). •Roll out plans•Variability in need for help desk support

Chris Domaleski ,  Senior Associate, National Center for the Improvement of Educational Assessment,   cdomaleski@nciea.org

National Center and State Collaborative

System Development Update

•Following pilots in spring and fall of 2014, NCSC successfully completed operational testing in May of 2015. •Approximately 70K tests were administered to approximately 35K students in 15 states/entities.

•Focus now is on scoring, standard setting, reporting, technical documentation, and final system enhancements.•System enhancements include:•Public ‘sandbox’•Data warehouse and reporting capabilities

What’s Working

•The system worked without disruption or malfunctions•Feedback on enhanced assessment features and user interface improvements was positive•Evidence supports NCSC’s conceptual framework for the assessment•NCSC realized the goal of developing an enhanced open system. • This summer NCSC will make the system publicly and freely available as open source software to include: • The assessment platform, based on the open-source TAO system• Test administrator portal based on the open-source Drupal CMS technology

Remaining Challenges

•‘Administration’ functions continue to be the most prevalent areas requiring user support (e.g. adding/ editing organizations, reassigning students, reopening closed tests). •Ongoing to support a broader range of devices•Looking ahead, NCSC seeks to identify efforts/ resources to support the open source community

WIDACarsten Wilmes, Director of Assessment, WIDA Consortium, wilmes@wisc.edu

System Development Update

• Ready for December 2015 launch of new online ELP assessments• Sample Items now available on the operational test platform (partner: DRC)• Ongoing work on standard setting, score report design, and training dissemination

Innovative item design

What’s Working

• Simple interface that is engaging and accessible for all ELLs• Integration with existing consortium customer support and training roll out resources• Handling of large volumes of speaking voice capture• Accessibility features

Remaining Challenges

• Supporting both online and paper assessments• Providing next-generation ELP assessments for all ELLs, including those with significant cognitive disabilities• Interoperability across multiple vendors using QTI/APIP

English Language Proficiency Assessment for the 21st CenturyWes Bruce, Field Test and Technology Readiness Team Lead, wesbruce3@gmail.com

System Development Update• Field Test was successfully delivered late winter/early spring• Currently scoring, calibrating items, preparing operational forms and documentation• States are identifying operational vendors for 2015-16• Test will be delivered on a variety of platforms• Deliverable timeline is driven by winter 2016 start of testing

What’s Working

•Flexible administration of Field Test• No waiver from testing (double testing)• Wider window • School-based participation

•Transfer of items from item development vendor to field test vendor (QTI/APIP)• A variety of innovative item types aligned to new standards and academic expectations•Online assessment of young (K-2) students in reading, speaking, writing, and listening•Accessibility and accommodations

Remaining Challenges

•Providing an intuitive online interface to English learners with limited/no technology experience is as difficult as it sounds – working on improvements•You cannot provide too many resources for access to item types and practice on the platform•Consider population in variety and grain size of tools

•Current QTI/APIP standards only specify so much• Item display across vendor platforms is still an issue

Guiding QuestionsRobert Crawford

Questions for the Panel

•What are your top 2 helpdesk questions?•What security features have you enhanced and why?•What accommodations and accessibility features have you incorporated for ELLs and students with disabilities?•What advice do you have to states who are in consortia that use multiple platform vendors?

Questions for the Audience

•What do you see as the pros and cons of states choosing their own operational test vendor?•Based on the article, “What are IT leaders worried about? Assessment readiness and money” posted by Stephen Noonoo, April 28, 2015, in COSN [http://www.eschoolnews.com/2015/04/28/it-leaders-survey-693/?ps=philip.olsen@dpi.wi.gov-001a000001XgMAf-003a00000258pWr]• Assessment readiness• Privacy• Budget

•What do we, as states, think we could have done better or differently in hindsight?•What did the consortia do well and what are you pleased with?•What can consortia do to make it easier for states in the future?