Reliability of ADA- CAT for ADA Compliance Denis Anson Director of Research and Development...
-
Upload
yahir-woolworth -
Category
Documents
-
view
215 -
download
0
Transcript of Reliability of ADA- CAT for ADA Compliance Denis Anson Director of Research and Development...
Reliability of ADA-CAT for ADA
Compliance
Reliability of ADA-CAT for ADA
ComplianceDenis Anson
Director of Research and Development
Assistive Technology Research Institute
Denis AnsonDirector of Research and
DevelopmentAssistive Technology Research
Institute
What is ADA-CAT?What is ADA-CAT?
ADA-CAT is a combination of a website and set of physical tools
ADA-CAT is a screening tool for accessibility of the environment, based on ADA standards
ADA-CAT was designed to allow non-engineers and non-architects to perform reliable and accurate assessments of compliance with ADA requirements
The WebsiteThe Website
The ADA-CAT website provides assessment checklists (Audits) for features of the environment
The WebsiteThe WebsiteEach item in an audit includes:
The ADA (or other) requirement, in understandable language
A link to the ADA-ABA requirement on the web
An explanation of why this feature is important
Just-In-Time Training
Just-in-Time TrainingJust-in-Time TrainingTraining you get today will be forgotten when you need it
ADA-CAT allows you to review how to assess an item as you are assessing
This includes both text and pictures of the process
The training is based on the ADA-CAT toolkit
The ADA-CAT ToolkitThe ADA-CAT Toolkit
The toolkit includes eleven tools in a custom bag
The tools include three off-the-shelf tools
25-foot tape measure
24 inch spirit level
Stopwatch
Three Off-The-Shelf Tools
Three Off-The-Shelf Tools
The tools are easily found in sports and hardware stores
The assumption might be that everyone knows how to use them
Watching novices attempt to use these common indicates that this isn’t the case
The ADA-CAT ToolkitThe ADA-CAT Toolkit
The toolkit contains three hard to find, but off-the-shelf tools
Light meter reading in Lux
A sound meter reading decibels
A force gauge
Three Hard-To-Find Tools
Three Hard-To-Find Tools
It’s unlikely that many have ever used tools like this
Most people have little understanding of the concepts being measured
Inverse Squared Rule?
Although the tools are “simple” to use, if not used correctly, they won’t give good results
The ADA-CAT ToolkitThe ADA-CAT ToolkitThe toolkit contains five custom tools not available anywhere else
Magic Slope Block
MultiTool
Key Torque Tool
The Font Guide
The StoryStick
Five Custom ToolsFive Custom Tools
These tools were created specifically for use in ADA-CAT assessments
Although the principles are hidden from the user, they can be complex
No-one who has not used ADA-CAT will have any experience with these tools.
Little or No Training Required?
Little or No Training Required?
The assertion is that ADA-CAT, the combination of the website and the tools, will produce reliable results when used by non-architects
This is a bold claim, and requires support
It is also testable
What is Reliability?What is Reliability?
There are two types of reliability that are commonly recognized in tests and measurements
Test-Retest Reliability
Inter-Rater Reliability
Test-Retest Reliability
Test-Retest Reliability
Test-Retest reliability means that when a measurement is taken repeatedly, the outcome is the same
When you measure the width of a door, you should always get the same width for the same door
Tools that have large subjective components will not show this reliability
Inter-Rater ReliabilityInter-Rater Reliability
Formally, this means that the results of a measurement are not dependent on the person doing the measurement
If I measure the weight of a glass of water, then you do, we should get the same result
In accessibility assessment, the “I was/wasn’t able to do it” test depends on who “I” am
Testing ADA-CAT Reliability
Testing ADA-CAT Reliability
A group of four OT research students undertook this task
The students received an introduction to the tools of the toolkit, but no observed trial in the real world
The students used the Elevator Audit as their sample test
Why Elevators?Why Elevators?
The Elevator Audit uses almost all of the tools of the toolkit
We didn’t use the Font Guide, but did use all the other tools
Elevators are commonly available
There is a wide range of accessibility in elevators in daily use
MethodMethod
Identify eight elevators to be assessed
Each researcher measures each elevator
After at least three weeks, each researcher measures each elevator again
Test-Retest Reliability
Test-Retest Reliability
Each researcher’s measurements can be compared with their measurements three weeks later.
Assuming no damage or repair to the elevators, there should be good agreement between the results
Inter-Rater ReliablityInter-Rater ReliablityThe measurements of each researcher can be compared to show the inter-rater reliability
When the scores for all four researchers were compared, the inter-rater reliability was .6
Statistically significant, but not exciting
Why?
Inter-Rater ReliabilityInter-Rater Reliability
Examining the data shows that, in most cases, three of the researchers had closely related data, and one was different.
Removing this individual increased the inter-rater reliability score to above .9
So why was this researcher’s scores different?
We’ll look at that in a minute.
Test-Retest Reliability
Test-Retest Reliability
If the scores of the researchers on first measurement are compared with the measurements 3 weeks later, the correlation is .97
So, What’s Wrong with Inter-Rater?So, What’s Wrong with Inter-Rater?
Individual items were compared between researchers where there was not good agreement
The differences clustered around two features
Sound levels
Braille
Sound LevelsSound LevelsThe Elevator Audit includes standards for the relationship between elevator signals and background noise, and for background noise levels
The outlier researcher did her evaluations during times when there was a lot of surrounding activities
During a basketball game for the athletic center elevator
Sound LevelsSound Levels
The background sound levels at busy times represent the people in the area, not the elevator, and cannot be controlled by the manufacturer
The sound measurements should be taken at quiet times, as the other three researchers did
Braille?Braille?Visually dependent people have a very difficult time determining if Braille is appropriate
In some of the older, and more abused elevators used in this study, the “raised” component of Braille was lost due to damage and wear
This was interpreted differently by different researchers, resulting in some variability
ConclusionsConclusions
Some additional instructions are needed on when to take sound measurements
Additional examples and counter-examples of Braille need to be added to the assessments