Race To Accessibility:The Past, Present, and Future of Accessible Assessments
The Council of Chief State School Officers
National Conference on Student Assessment
June 23, 2015
Panel of Presenters Trinell Bowman
Program Manager, Maryland State Department of Education PARCC –
Partnership for Assessment of Readiness for College and Careers
Jan McSorleyHead of Accessibility for School, Pearson
Wayne DickSenior Research Fellow, Knowbility, Inc.
Dave EdyburnProfessor, University of Wisconsin-Milwaukee
Linda ZimmermanSr. Assessment Solutions and Design Manager, Pearson Assessment Centre
Paul Grudnitski, in absentia TestNav Chief Architect, Pearson
The Evolution of Accessible Assessments
Drivers for Accessible AssessmentsThe reauthorization of IDEA in 1997 required that students with disabilities participate in state and district assessments. It also required that for students who could not participate in statewide assessments with accommodations that an alternate state assessment be developed.
By 2004-2005, more that half of the states requested test vendors to use universal design principles in developing their assessments.
Beginning in 2005, a few states begin developing online assessments which provided some additional opportunities and challenges for online testing.
Web Content Accessibility Guidelines 2.0 (WCAG)
Section 508 Amendments to the Rehabilitation Act; Usability Guidelines based on Section 508. http://guidelines.usability.gov/
Accessible Portable Item Protocol 1.0 (APIP)
1997 2001 2002-2004 2005 2006 2007 2008 2009 2010
IDEA and NCLB
A series of Enhanced Assessment Grants were awarded that focused on
accessibility and accommodations
Statewide Testing and the Shift from Paper to Online Testing
Universal Design
Applied to Assessment
Bias and Sensitivity Reviews
Test Available in Braille and Large Print,
etc. Race to the Top
Race to the Top Fund Assessment Program Selection Criteria
(A)(1) Governance - (up to 20 points) (A)(2) Theory of Action - (up to 5 points) (A)(3) Assessment Design - (up to 55 points) (A)(4) Assessment Development - (up to 35 points) (A)(5) Research and Evaluation - (up to 30 points) (A)(6) Professional Capacity and Outreach - (up to 15
points) (A)(7) Technology - (up to 10 points) (A)(8) Project Management - (up to 30 points)
PARCC Strategies for Increasing Student Access Provide item developers with clear guidelines for writing items that are free
of bias, are sensitive to diverse cultures, are stated clearly, and use consistent formats
Require item developers to use principles of Universal Design to allow participation of the widest possible range of students, and increase the likelihood that test questions measure only what they are intended to measure
Conduct bias and sensitivity reviews and statistical procedures that are designed to detect bias as part of the item development/field testing process
Develop common test accommodation and participation policies for SWDs and ELLs
Use technology to provide and increase access to testing accommodations
Conduct research to determine factors that promote or hinder accessibility
Accessibility Priorities for thePARCC Assessment System The Assessment System must be based on universal design principles and
other research-based principles such that all applications will support access for all students and staff.
The content (e.g., passages, items) must be associated with meta-data that describes any changes that will be made to the content, display, or input method necessary to provide appropriate accommodations support to the student.
The overall approach must leverage the use of computer-based accessibility tools, driven by an item tagging system that will control and ensure appropriate application of those tools at the item level.
PARCC included accessibility requirements as part of the PARCC Architectural Design and Specifications
Assessing Students With Disabilities
Goal: Assure that the examination measures what it is intended to measure, rather than reflecting the individual's impaired sensory, manual, or speaking skills. (American with Disabilities Act Title III, Part XII)
Challenges: Different technology used in class and on tests Lack of Braille access to tests Lack of AT and browser support for mathematics content New items types available – often relying on visual representations Providing access to instructional supports Security and compatibility with 3rd party technologies
Lessons Learned and Counting…. Incorporate the use of accessibility training during item
development and review trainings.
Engage students in the use of cognitive labs and usability studies
It is important that the item types are accessible and do not create further barriers The use of scrolling The number of items on a screen Use of technology enhanced items and how they interact with
various assistive technology software and devices Infrastructure Trials and Technical Assistance Bulletin
Student Needs and the Technical Landscape
The job we do in education is important. Students rely on us to be the gatekeepers of quality and high expectations. They rely on us to help them maintain their hope and achieve their dreams.
Our responsibility to students
Given the right tools …
People with disabilities …can acquire the same
information
…participate in the same activities
…be active producers as well as consumers
Building AssessmentsWhen we build assessments, we have to juggle a lot of competing priorities. It’s not an easy thing to do, but we have to learn from our experiences and make improvements in our process as we go. If we keep doing things the same way, then something bad is eventually going to happen....
Building Assessments
Building assessments requires us to work with diverse groups of people and it requires a lot of different types of expertise. This can be challenging and because of these challenges, it can be tempting to make assumptions about student needs. Pressure and severe time constraints are part of building assessments, so we need a plan to keep us on track and to ensure that we’re not making avoidable missteps.
Building AssessmentsHaving a plan is USUALLY better than juggling … but if we have a bad plan then we’re still not going to get where we need to go. We have to think differently about assessments now that we can leverage the power of technology. Building digital assessments in the same we’ve always built paper assessments is not an improvement. We have to take responsibility to understand our users and the technical landscape of possibilities.
Building Assessment
When a good plan comes together, it’s because it’s built on a solid foundation. The corner stone of a solid assessment foundation, is truly understanding the users.
Understanding the population
10% - 15%
According to statistics, the community of people with disabilities is the is the largest minority in the world. Currently, 10 – 15 percent of the world or over 600 million people live with a sensory, intellectual or physical disability.
Overview of Global Landscape
United States 9.9% aged 16-64 35% work compared
with 78% 2 times more likely drop
out of high school Approximately 13%
student population 62% spend at least 80%
of school day in general education classroom
Globally 98% don’t attend school 2/3 of people with
disabilities live in low income countries
85% under age 15 are from developing nations
2025 the number of people with disabilities will double
Specialized Tools
Some Examples Include: Refreshable Braille Displays, Screen Readers, Screen Magnifiers, Closed Captions, Sign Language Videos, Text-to-Speech with Synchronized Highlighting, Augmentative Communication Devices, and even something as simple as a pair of eye glasses
Simply put …
If you don’t know your audience: you can’t
effectively assess them and
you can’t effectively inform changes in instruction
Assessment Definitions Features for All / Universal Tools - access features of the
assessment that are available to all students on the basis of student preference and selection (e.g. highlighter, line reader, etc.)
Accessibility Features Identified in Advance / Dedicated Supports - features that a student regularly uses in the classroom for instruction and/or assessment for whom the need has been indicated by an educator or group of educators, including an IEP or Section 504 team. (text-to-speech for math, color contrast, etc.)
Accommodations - changes in procedures or materials that increase equitable access during the assessments. (Braille, Large Print, Assistive Technology, etc.)
UDLUniversal Design has 3 principles that work together to support access for student learning. The first principle is representation which is how the information is presented. The second is action and expression which gives students alternatives to pencil and paper to express the knowledge on a subject and the third is engagement which stimulates student interest by providing learning in a variety of ways. It is important to note that it might not be feasible to incorporate all three principles into every lesson every time. Rather, they are intended to guide instruction over time. Even when teachers apply the three principles, some students may need additional support. Consequently, teachers will sometimes have to make accommodations (e.g., allow the use of a spell checker) to meet an individual student's needs.The important thing about UDL is that is supposed to be done BEFORE designing the lesson. That is what is important for our discussion today. The current design review process is a start in the right direction but it will not be enough as we move forward. New accessibility requirements go beyond current UDR considerations.
It’s a principle It is NOT a
standard
Representation
EngagementAction and Expression
It is irresponsible to talk about Universal Design for Learning without emphasizing that you cannot have a universally designed product that does not adhere to accessibility standards.
With great powercomes great responsibility.
Standards Conundrum
APIPAccessible
Portable ItemProtocol
QTIQuestion and Test
Interoperability
WCAGWeb Content Accessibility
Guidelines
Accessible PortableItem Protocol (APIP)
APIP Specification – IMS Global XML based specification that allows computer-
based assessments to be portable and accessible
Accommodations are built into the test delivery system
Personal Needs and Preferences (PNP) profile
http://www.imsglobal.org/APIP/index.html
Question & Test Interoperability (QTI)QTI Specification – IMS Global
Provides standard XML language
QTI ensures interoperability between systems
QTI has been used internationally for over a decade.
Not QTI compliant = Not APIP compliant
http://www.imsglobal.org/question/
Web Content Accessibility Guidelines (WCAG)
Internationally recognized accessibility web standards
Guidelines allow content and applications to be
accessible to people who depend on the use of
assistive technologies
ISO Standard #: ISO / IEC 40500:2012
12 Guidelines with 61 measurable Success Criteria
In summary …
APIP = building the access features into the testing platform
QTI = ensuring that XML versions of test items can be stored in banks and shared with other QTI-compliant testing systems.
WCAG = ensuring that third-party assistive technology tools can work with the testing platform
Challenges APIP: ensuring that all of the accommodations
students need are present within the system, particularly when students have complex access needs
WCAG: ensuring that test security and construct validity are maintained when third-party access tools are introduced to the testing environment.
QTI: there is a limited set of HTML elements included in QTI
Web Accessibility Initiative – Accessible Rich Internet Applications (WAI-ARIA)
WAI-ARIA Specification (W3C) Defines a way to make Web content and Web
applications more accessible to people with disabilities.
It especially helps with dynamic content and advanced user interface controls developed with AJAX, HTML, JavaScript, and related technologies.
Our Accessibility Journey In May 2014, after going full
speed down the APIP road for the previous 12 months, we discovered that our accessibility “solution” – including our Testnav 8 Client technology and our content encoding model (derived exclusively from QTI/APIP) – was ineffective for people with certain disabilities.
Why Ineffective? Lack of Fidelity
Standard Text-to-Speech implementations do not offer sufficient Semantic and Perceptual “visibility.”
Lack of Authenticity Standard Text-to-Speech implementations (e.g., clicking links and
listening to time-coded audio files), are not “authentic.” Using Assistive Technology (AT) such as JAWS, NVDA, and others,
is the authentic method used by students every day. Denying access to these kinds of tools leads to further unfairness.
Assistive Technology (AT) functions and features are tightly coupled to HTML patterns that are dying or dead. Even if we wanted to support authentic AT, doing so using common
conventions with browser technology is not easily achievable with the design of modern, hi-performance web applications such as TestNav 8.
About APIP APIP
is not effective for providing content fidelity for people with certain disabilities who use certain AT
is useful and sufficient for describing certain kinds of accommodations
is comprised of two accessibility-related xml vocabularies:
1. Personal Needs Profile (PNP) <accessForAllUser>
2. Item/Test Accessibility Info <apipAccessibility>
Limitations of <apipAccessibility> Does not account for the way some users; i.e., visually
impaired users of assistive technology “see” content using the authentic tools that are commonly used; e.g., JAWS, NVDA, ZoomText, etc.
Ineffective for providing random access to content. In other words, no good way to encode “semantic markers.”
Ineffective for providing “perceptual visibility” within Item content and Item Interactions.
APIP PNP (Personal Needs & Preferences)
PNP – A LOT of “goodness.” Standardizes ways of describing Student-Assessment accommodations.
<accessForAllUser>
<control>
<apip:apipControl>
<apip:additionalTestingTime>
<apip:assignedSupport>true</apip:assignedSupport>
<apip:timeMultiplier>1.5</apip:timeMultiplier>
</apip:additionalTestingTime>
<apip:breaks>
<apip:assignedSupport>true</apip:assignedSupport>
</apip:breaks>
</apip:apipControl>
</control>
<display>
<screenEnhancement>
<magnification>3</magnification>
</screenEnhancement>
</display>
</accessForAllUser>
Accommodation Fidelity
APIP WCAG + ARIA
• So let’s talk for a minute about how students with disabilities overcome barriers to digital content in the context of assessment.
• On the screen you see a pair of crutches that for illustrative purposes are representing the APIP standard.
• You also see a photograph of a girl in a customized wheelchair. This photo is meant to illustrate WCAG + ARIA standards and best practices, which testing companies can comply with in order to allow students to use third-party assistive technology
• The crutches have been decorated with bows and decorative tape. It looks like they may have added some additional padding to the top of the crutches. You can also see that the crutches can be made taller or shorter, depending on the height of the user. In the same way that built-in accommodations through APIP are meant to help learners overcome minor barriers to accessing digital content, these crutches are designed to help people overcome the effects of a minor leg injury. Lots of people could use these crutches or similar crutches that perhaps are decorated in a different way. They could use the crutches with very little training and even though it might be slightly annoying to have to use crutches, you could at least still be mobile and accomplish a lot of tasks.
• However, a person who needs a power wheelchair, will never be able to use a pair of crutches. They need a much more robust set of supports in order to accomplish what other people might accomplish with the crutches or without any support at all.
Accommodation Authenticity• And … the person who needs a power wheelchair, needs that chair to be
customized to their unique needs. You will notice that the girl in this wheelchair, has a head rest, a bar across the middle of her chair to help her stabilize herself and she has a mechanism to hold her legs in place. In the same way, students who use complex assistive technologies will often customize how those tools work so that they can meet their unique needs. Students learn to use these complex tools to a level of muscle memory so that they don’t even have to think about how the tool works as they use it to complete their school work.
• If we try to build in a complex access system, like a power wheelchair, no matter what we do it won’t be the same tool that students use every day with their customized settings. It won’t be a tool that students can use without having to think about how to use it. Notice that the other power wheelchair on this screen does not have the head set or the mechanism to hold the users legs in place, nor does it have a bar across the middle to help with stabilization.
• This is meant to illustrate that if testing companies try to build in complicated access systems into an online assessment system, they may meet the needs of some students, but because they have taken away the students’ ability to choose their access system through third-party assistive technology, they have created a man-made barrier to success for students who’s access needs exceed or differ from what is built into the system.
• My point is that there is a limit to what we can do with APIP and we, as a company, are continuing to explore where APIP ends and WCAG and third-party Assistive Technology begin
The One-Page WebappImportant AT features are tied to obsolete HTML patterns
All existing WCAG / ARIA examples (that we found) use old-school html page / form loads to set the stage for AT behaviors.
Modern, hi-performance web applications, such as TestNav 8, load the entire application once, and abandon old-school html form, fieldset, and submit patterns.
Without forms and fieldsets, a lot of AT stops working. This requires a great deal of engineering to work around assumptions made by AT.
Standards Toolkit
APIPAccessible
Portable ItemProtocol
QTIQuestion and Test
Interoperability
WCAGWeb Content Accessibility
Guidelines
Plus …WAI-ARIA!!!
Example – Two-Column Item
A two-column assessment item. The left column contains text outlining the assessment problem for the student. The student is to read the math word problem and select the correct answer from the multiple choice answers given in the right column. The right column contains four multiple choice answers labeled A, B, C, and D. Each multiple choice answer is a graphic image of a wrapped box with colored wrapping paper. Each present has a different color paper.
The human eye naturally scans this from top to bottom, left to right, “chunking” the content.
Application/Content designers add/remove visual importance to screen elements so as to draw the eye to certain elements.
The human eye can - and does - jump back and forth to “randomly access” the left and right columns.
Users of AT can do none of this without the injection of semantic markers into content
HTML Content through the lens of AT<div class="row"><div class="span6"><p>Sarah has eight dollars. She wants to buy as many boxes of chocolates as she can.</p><p>The blue box of chocolates cost two dollars and the red box cost three dollars. The yellow box of chocolates is three dollars more than the blue box. The purple box costs twice as much as the blue box.</p><p>If Sarah spends all eight dollars and only buys two different boxes of chocolates, which chocolates can she buy? Select the two boxes of chocolates that Sarah buys.</p></div><div class="span6"><div class="well"><div class="whitebg pad8"><ul class="int-choice-list two-up at-enable" data-module="ChoiceInteraction“ data-identifier="RESPONSE"><li><label class="checkbox" for="choice_cdf06"><div class="int-choice-control"><input type="checkbox" id="choice_cdf06" class="checkbox" name="group_RESPONSE" value="B" /></div><div class="int-choice-label">A.</div><div class="int-choice-desc"><span><img src="public/assets/bluebox.png" alt="blue box" width="88" height="100" /></span></div></label></li><li><label class="checkbox" for="choice_3230c"><div class="int-choice-control"><input type="checkbox" id="choice_3230c" class="checkbox" name="group_RESPONSE" value="R" /></div><div class="int-choice-label">B.</div><div class="int-choice-desc"><span><img src="public/assets/redbox.png" alt="red box" width="72" height="100" /></span></div>
</label></li><li><label class="checkbox" for="choice_a8002"><div class="int-choice-control"><input type="checkbox" id="choice_a8002" class="checkbox" name="group_RESPONSE" value="Y" /></div><div class="int-choice-label">C.</div><div class="int-choice-desc"><span><img src="public/assets/yellowbox.png" alt="yellow box" width="102" height="100" /></span></div></label></li><li><label class="checkbox"
for="choice_c79b2"><div class="int-choice-control"><input type="checkbox" id="choice_c79b2" class="checkbox" name="group_RESPONSE" value="P" /></div><div class="int-choice-label">D.</div><div class="int-choice-desc"><span><img src="public/assets/purplebox.png" alt="purple box" width="105" height="100" /></span></div></label></li></ul></div></div></div></div></div>
Semantic Markers Semantic Markers are elements within the html
markup that content authors can add in order to describe structure and meaning to assistive technology.
Semantic Markers are akin to the Table of Contents of a book. They provide an outline view of content.
WCAG / ARIA to the rescue! Three common strategies for adding semantic markers: Use header tags Use anchor tags Use ARIA “landmark” roles
What Does the QTI Look Like?<div class="row"> <div class="span6"> <h2 class="semantic-marker visuallyhidden">Introduction</h2> <p>Sarah has eight dollars. She wants to buy as many boxes of chocolates as she can. </p> <p>The blue box of chocolates cost two dollars and the red box cost three dollars. The yellow box of chocolates is three dollars more than the blue box. The purple box costs twice as much as the blue box.</p> <h2 class="semantic-marker visuallyhidden">Question</h2> <p>If Sarah spends all eight dollars and only buys two different boxes of chocolates, which chocolates can she buy? Select the two boxes of chocolates that Sarah buys </p> </div> <div class="span6"> <div class="well"> <div class="whitebg pad8"> <h2 class="semantic-marker visuallyhidden">Interaction</h2> <choiceInteraction class="two-up" maxChoices="0" responseIdentifier="RESPONSE" shuffle=“false"> <simpleChoice identifier="B"><span><img alt="blue box" height="100" src="assets/bluebox.png" width="88"/></span> </simpleChoice> <simpleChoice identifier="R"><span><img alt="red box" height="100" src="assets/redbox.png" width="72"/></span> </simpleChoice> <simpleChoice identifier="Y"><span><img alt="yellow box" height="100" src="assets/yellowbox.png" width="102"/></span> </simpleChoice> <simpleChoice identifier="P"><span><img alt="purple box" height="100" src="assets/purplebox.png" width="105"/></span> </simpleChoice> </choiceInteraction> </div> </div> </div></div>
Random Access
Three heading H2 tags of “<h2>Introduction</h2>”, “<h2>Question</h2>”, and “<h2>Interactions</h2>” that feed into assistive technology and then to a computer.
Semantic Markers provide the outline view of the Content. All modern AT “sees” heading tags and anchor tags. Users can
access these with “quick key” shortcuts, enabling AT users to discover the structure and meaning of the Content.
This gets us closer to random access of content, thus providing a more equitable experience.
Perceptual Visibility Semantic Markers are usually not enough. We need to add “Perceptual Visibility” to the
content. How? With ARIA.
Example – Describe the Interaction:
1. The interaction type; e.g., multiplechoice, matching, text entry, hotspot, graphing, etc.
2. The number of options
3. Option descriptions
4. Interaction context
Note: this can get very complex and sophisticated!
Four multiple choice answers labeled A, B, C, and D. Each multiple choice answer has an image of a box with colored wrapping paper. Each box is covered in a different color of paper.
<div class="row"> <div class="span6"> <h2 class="semantic-marker visuallyhidden">Introduction</h2> <p>Sarah has eight dollars. She wants to buy as many boxes of chocolates as she can. </p> <p>The blue box of chocolates cost two dollars and the red box cost three dollars. The yellow box of chocolates is three dollars more than the blue box. The purple box costs twice as much as the blue box.</p> <h2 class="semantic-marker visuallyhidden">Question</h2> <p>If Sarah spends all eight dollars and only buys two different boxes of chocolates, which chocolates can she buy? <span aria-hidden=“true”> Select the two boxes of chocolates that Sarah buys </span></p> </div> <div class="span6"> <div class="well"> <div class="whitebg pad8"> <h2 class="semantic-marker visuallyhidden">Interaction</h2> <choiceInteraction class="two-up" maxChoices="0" responseIdentifier="RESPONSE" shuffle=“false“ aria-label=“Select the two boxes of chocolates that Sarah buys. Multiple choice checkbox group with 4 options” role=“group”> <simpleChoice identifier="B“ aria-setsize="4" aria-posinset="1"><span><img alt="blue box" height="100“ src="assets/bluebox.png" width="88"/></span></simpleChoice> <simpleChoice identifier="R” aria-setsize="4" aria-posinset=“2"><span><img alt="red box" height="100" src="assets/redbox.png" width="72"/></span></simpleChoice> <simpleChoice identifier="Y“ aria-setsize="4" aria-posinset=“3"><span><img alt="yellow box" height="100" src="assets/yellowbox.png" width="102"/></span></simpleChoice> <simpleChoice identifier="P“ aria-setsize="4" aria-posinset=“4"><span><img alt="purple box" height="100" src="assets/purplebox.png" width="105"/></span></simpleChoice> </choiceInteraction> </div> </div> </div></div>
What Might the QTI Look Like?
Semantic and Perceptual VisibilitySemantic Markers and ARIA, when used together, get us closer to the best of both worlds.
1. Provide big picture meaning of content.
2. Random access to chunks of content, saving time, reducing cognitive load.
3. Fine-grained, perceptual detail, where needed.
…all of the above using the authentic AT that students use every day in classrooms and at home. APIP, alone, accounts for very little of this.
Progressive Enhancement
Drag and Drop
This emulates Matching with paper.
On the screen you see a picture of a sample drag and drop item. There are four “draggers” labeled “Capulet, Demetrius, Lysander, and Prospero.” There are three drop bays, labeled, “A Midsummer-Night’s Dream, Romeo and Juliet, and The Tempest.” The purpose of the item is to match the character with the play in which they appear.
The same information can be presented to screen readers as a table with 4 columns and 5 rows. In this example, the names of the plays are listed in the first row as: A Mid-Summer Night’s Dream; Romeo and Juliet; The Tempest
The names of the characters are listed in the first column as: Capulet; Demetrius; Lysander; Prospero
The data in the top row and the first column have header tags to indicate that this data would be presented to students through a properly marked up HTML table. The data cells in this example have radio buttons, but depending on the construct, they could be checkboxes instead.
In an actual HTML table, the data cells would also be marked up, so this is just for illustrative purposes. With a properly marked-up HTML table students can use their screen readers and/or refreshable Braille devices to have randomized access to the table content.
Data Structure + Look and Feel + Functionality
Sample Table
<th scope=“col”>A Mid-Summer
Night’s Dream</th>
<th scope=“col”>Romeo and Juliet</th>
<th scope=“col”>The
Tempest</th>
<th scope=“row”>
Capulet</th>
<th scope=“row”>
Demetrius</th>
<th scope=“row”>
Lysander</th>
<th scope=“row”>
Prospero</th>
EVERY Student Gets the Whole EnchiladaOn the screen you see a plate filled with enchiladas. Each enchilada is unique. This is to illustrate that … If the diverse needs of students are considered
at the beginning of the design, item content can be built without compromising: Accessibility Rigor Interactivity Construct validity Security
What Happens with Progressive Enhancement? In terms of our drag and drop example:
Fully sighted participants with good motor ability can drag answers to the proper drop bays.
Other participants with certain types of assistive technology can use the table directly.
Everybody wins. AND, if we need a new interface we can build
it from the table.
Interdependencies
WCAG
ATAG UAAG
W3C Guidelines Content Authoring User Agents
Call To Action
Unfortunately…
1. For taxpayers,
2. For users of assistive technology
3. Providers of assistive technology, and
4. For assessment providers,
…it’s just not that easy!
Why? Because the ARIA integration of Browser technology with the Accessibility API and with Assistive Technology is very uneven and often unpredictable.
Unique Opportunities for Computer-based AssessmentsDave Edyburn, Professor
University of Wisconsin-Milwaukee
Ruben R. Puentedura, Ph.D explains that the goal of technology integration is to have students create content and learn in a technology rich environment with opportunities to apply their critical thinking skills. He outlines four levels of technology integration in the SAMR Model. SAMR stands for Substitution, Augmentation, Modification, and Redefinition. As technology integration progresses, the effects on student learning outcomes move from enhancement to transformation
Ehancement: Substitution – Technology acts as a direct tool substitute, with no
functional change Augmentation – Technology acts as a direct tool substitute with
functional improvement Transformation:
Modification – Technology allows for significant task redesign Redefinition – Technology allows for the creation of new tasks,
previously inconceivable
SAMR Model
Missed Opportunities
On the screen you see a name tag that says, “Hello, my name is Missed Opportunities.” We have missed:
The opportunity to develop a system that generates empirical data about the effects of various interventions on student performance. This would overcome the historical problems identified in the research literature that the lack of validity of teachers selecting accommodations for students.
The opportunity to design a system that would obtain value added data about student performance.
The opportunity to expand access. Not just physical and sensory access but cognitive access.
Workspace …
MY Roles
Responsibilities
Interests
Education
Abilities
Expertise
Habits
Used Items
MY Team
Manager
Colleagues
Competition
Partner
MY Process
Projects
Tasks
Activities
Decisions
Tools
Theory of the User
Cartoon with two dogs. The first dog is sitting in a desk chair in front of the computer, looking down at the second dog who is sitting on the floor. The first dog says, “On the internet, no one knows you’re a dog.”
The term theory of user refers to system analytics that are used to analyze and profile the user.
We are all familiar with the recommendation engines at Amazon and Pandora that make recommendations for us based on our historical preferences.
Why can't these interventions be applied to digital assessment? For example, can we determine/verify whether or not the user has a disability? That is, what type of interaction and response patterns are indicative of a
disability? Or, is the student an English Language Learner?
Embedded Support
• Given the purpose of assessing whether students are college and career ready, it is inappropriate to create assessment environments where students do not have access to the tools they routinely use.
• One way of thinking about how to implement universal design in the assessment context is known as embedded support.
• That is, tools such as a notepad, calculator, dictionary, etc. are embedded in the system. Embedded supports can be activated one of three ways
• always on for everyone• available only to students who prequalify, and• activated based on student performance
• The availability and use of embedded supports requires a new generation of psychometrics.
A New Generation of Psychometrics
The transition from paper-based assessments to digital assessments requires that we consider new approaches to establishing reliability and validity. Indeed, the field of psychometrics is likely to be transformed though learning analytics.
For example, dynamic norming is a term used to describe the process of creating norm groups based on demographics of low incidence groups (i.e., disability, use of specialized supports like assistive technology) to make performance comparisons that would never be possible with traditional instrument validation because of the extreme cost associated with identifying and reaching such low incidence populations.
Discussion Points
Measuring What Students Know and Can Do Purpose – gather information related to target
construct, while minimizing the construct-irrelevant variance, to make valid inferences
Techniques Universal Design for Learning Accessibility and Accommodations Framework
Always available Identified in advance for student Accommodations Embedded or non-embedded
Item modification (Progressive Enhancement) Accessibility Reviews
Understanding Risks … The legal landscape is changing
Section 508 of the Rehab Act is being revised. The foundation of the revision is the Web Content Accessibility Guidelines (WCAG), version 2.0, Level AA
The Americans with Disabilities Act (ADA ) is being revised. Title III of the ADA
Assessment: Places of “Public Accommodation”
Department of Justice has made it clear that they are interpreting web-based services as places of public accommodation (see recent edX settlement – inaccessible LMS)
OCR has defined “effective communication” as: Timeliness of delivery Accuracy of the translation, and Provision in a manner and medium appropriate to the significance of the
message and the abilities of the individual with the disability
Advocacy Groups:Entering Into the World of K-12 Education Through the Assessment Door… - For the past several years, advocacy groups, such as the National
Federation of the Blind (NFB) have focused the majority of their litigation efforts in higher education.
- They are now entering into the world of K-12 education through the assessment door.
- In October 2014, the NFB posted a notice to their listserv, asking families to contact them if they or their children have experienced inadequate practice opportunities for any high stakes assessment.
In November of 2014, the Texas Chapter of the NFB passed a resolution to make sure that the Texas state assessment is accessible.
- The NFB’s legal team is presenting to parents at the National Convention on July 5th about what they need to know to properly advocate for their children in instruction and assessment.
- They have learned a great deal about assessment over the past year and a half and they know where the weaknesses are in the industry.
What can states do? Do your research:
Know the needs of students with various disabilities
Know the tools they use to access digital contentKnow the laws that protect them
Make sure your RFP language accurately spells out accessibility requirements and that you specify quantitative measures for accessibility (e.g. WCAG 2.0, Level AA conformance)
Moving forward … Challenges
Standardization – No longer one-size-fits all Consistency between classroom and testing environments Changing technology 3rd party technologies not following W3C standards APIP-WCAG-QTI toolkit
Future Continued move to online assessment (Pearson –25% in 2012 to over
50% projected for 2016) Research on effects of use of accessibility features Data analytics and psychometric models
Accessibility standards matters They are our front-line defense for students with disabilities to ensure
that they have equality in education and assessment.
Questions
Top Related