Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies
-
Upload
michael-m-grant -
Category
Education
-
view
502 -
download
1
Transcript of Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies
![Page 1: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/1.jpg)
Arrogance or Apathy?
The need for formative evaluation + current & emerging strategies
Michael M. Grant, PhD University of South Carolina
Michael M. Grant 2015
![Page 2: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/2.jpg)
Michael M. Grant The University of South Carolina http://viral-notebook.com @michaelmgrant
![Page 3: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/3.jpg)
Arrogance or Apathy? We just
don’t have time to do evaluation.
Our HR folks
won’t let us do evals.
There’s really no point because we’re going to
deploy it anyways.
It’s just not going to make a difference.
We don’t have access to testers.
Our managers
don’t care.
We’re just doing it for compliance.
![Page 4: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/4.jpg)
![Page 5: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/5.jpg)
Name a reason for evaluation.
http://pollev.com/mgrant
![Page 6: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/6.jpg)
![Page 7: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/7.jpg)
Level 5: ROI
Level 4: Organization
Level 3: Transfer
Level 2: Learning
Level 1: Reaction
Kirkpatrick (& Phillips) Levels
![Page 8: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/8.jpg)
Level 2 Evaluations Level 5: ROI
Level 4: Organiza2on
Level 3: Transfer
Level 2: Learning
Level 1: Reac2on
Appeal
Effectiveness
Efficiency
![Page 9: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/9.jpg)
Level 5: ROI
Level 4: Organization
Level 3: Transfer
Level 2: Learning
Level 1: Reaction
Kirkpatrick Levels
91.3%
53.9%
22.9%
7.6%
2.1%
(ASTD, 2005)
in practice
![Page 10: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/10.jpg)
79%
38%
15% 9%
0%10%20%30%40%50%60%70%80%
Reaction Cognitive Behavior Results
Kirkpatrick Levels
R.A. Noe (2005) Employee Training and Development
in practice
![Page 11: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/11.jpg)
Level 5: ROI
Level 4: Organization
Level 3: Transfer
Level 2: Learning
Level 1: Reaction
Kirkpatrick Levels
(ASTD, 2009)
92%
53.9%
22.9%
7.6%
17.9%
in practice
![Page 12: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/12.jpg)
Level 5: ROI
Level 4: Organization
Level 3: Transfer
Level 2: Learning
Level 1: Reaction
Kirkpatrick Levels
(TrainingMag, 2013)
68 percent of applicants u2lize Return on Value; 71 percent u2lize Return on Investment; 56 percent u2lize balanced scorecards; and 47 percent u2lize Six Sigma. The Kirkpatrick Levels of Evalua2on are more widely used: Level 1 (97 percent), Levels 2 and 3 (94 percent), and Level 4 (88 percent).
97%
94%
94%
88%
71%
in practice
Training Magazine Top 125 Companies in 2013
![Page 13: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/13.jpg)
The New World Kirkpatrick 4 Levels
!!!
6
Required Drivers
!
Learning Context Performance Context
![Page 14: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/14.jpg)
Level 4 and Level 5
Level 5: ROI
Level 4: Organiza2on
How do we measure the impact on business?
How do we measure the return on investment?
![Page 15: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/15.jpg)
Level 4 and Level 5
Level 5: ROI
Level 4: Organiza2on
How do we measure the impact on business?
How do we measure the return on investment?
Compares benefits to cost/ Benefit Cost ra2o: ROI(%) = Net Monetary Benefits x 100
Program costs
Measures changes in business impact variables ( produc2vity, incidents, compliance discrepancies, customer service, etc,.
![Page 16: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/16.jpg)
Effectiveness Evaluation Activities include field tests, observations, interviews and performance assessments. Purpose? Determine whether the ILE accomplishes its objectives within the immediate or short-term context of its implementation.
![Page 17: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/17.jpg)
Formative Evaluation What ’s the purpose?
![Page 18: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/18.jpg)
A focus on improvement during development.
![Page 19: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/19.jpg)
User Review Observations from one-on-ones and small groups
![Page 20: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/20.jpg)
“Vote early and often.” The sooner formative evaluation is conducted during development, the more likely that substantive improvements will be made and costly errors avoided.
— Reeves & Hedberg (2003), p. 142
![Page 21: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/21.jpg)
Formative Evaluation Stages • Design team review • Expert review • One-to-one • Small group • Field trials
![Page 22: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/22.jpg)
Questions for Evaluation 1. What are the logistical
requirements for implementing the ILE?
– Hardware – Software – Adjunct materials – Help and support
2. What are the user reactions to the ILE?
– Appeal – Motivation – Usability – Comprehension
3. What are the trainer/instructor reactions to the ILE?
– Appeal – Utility
4. What are the expert reactions to the ILE?
– Content – Instructional design – Human-computer interface – Aesthetics
5. What corrections must be made to the ILE?
6. What enhancements can be made to the ILE?
![Page 23: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/23.jpg)
Data Collection Matrix
Methods
1. What are the logistical requirements?
2. What are user reactions?
3. What are trainer reactions?
4. What are expert reactions?
5. What corrections must be made?
6. What enhancements can be made?
Anecdotal records X X X X X
User questionnaires X X X X
User interviews X X X X
User focus groups X X X
Usability observations X X X X
Online data collection X X
Expert reviews X X X
![Page 24: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/24.jpg)
Evaluation informs development
• Project Conceptualization Review
• Design Needs Assessment
• Development Formative Evaluation
• Implementation Effectiveness Evaluation
• Institutionalization Impact Evaluation
• Project Re-conceptualization Maintenance Evaluation
from Reeves & Hedberg (2003)
![Page 25: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/25.jpg)
Contemporary Development Models
Michael Allen/Allen Interactions’ Successive Approximation Model (SAM)
![Page 26: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/26.jpg)
Contemporary Development Models
Image from h`p://www.intechopen.com/source/html/19453/media/image2.jpeg
Concurrent Design
![Page 27: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/27.jpg)
from Tripp, S., & Bichelmeyer, B. (1990)
Rapid Prototyping
Contemporary Development Models
![Page 28: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/28.jpg)
Contemporary Development Models
• Originated in manufacturing • ID hijacked from software
development • Focused on development
primarily
• Types of prototypes § Look-and-feel: colors,
effects, gross screen layouts
§ Media: use of sound effects, narration, 3D illustrations, video, etc.
§ Navigation: move through sections, access support (glossary, calculator, etc.)
§ Interactivity: content, activities, feedback
Rapid Prototyping
![Page 29: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/29.jpg)
Contemporary Development Models
1. Active user involvement is imperative
2. The team must be empowered to make decisions
3. Requirements evolve but the timescale is fixed
4. Capture requirements at a high level; lightweight & visual
5. Develop small, incremental releases and iterate
6. Focus on frequent delivery of products
7. Complete each feature before moving on to the next
8. Apply the 80/20 rule 9. Testing is integrated
throughout the project lifecycle – test early and often
10. A collaborative & cooperative approach between all stakeholders is essential
Agile Software Development
![Page 30: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/30.jpg)
What to consider with effectiveness . . . • An approved evaluation plan
– e.g., union, stakeholders, management
• Feasibility • Reliability • Validity • Implementation logs
![Page 31: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/31.jpg)
“Training evaluation provides the data needed to demonstrate that training does provide benefits to the company.”
(p. 311, R. Krishnaveni, 2008)
![Page 32: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/32.jpg)
“Vote early and often.”
The sooner forma6ve evalua6on is conducted during development, the more likely that substan6ve improvements will be made and costly errors avoided.
(Reeves & Hedberg, 2003, p. 142)
![Page 33: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/33.jpg)
Formative Evaluation
Expert review during development
User review during development
Usability tes2ng
3 Methods
![Page 34: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/34.jpg)
“Experts are anyone with specialized knowledge that is relevant to the design of your interactive learning
environment.”
(Reeves & Hedberg, 2003, p. 145)
![Page 35: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/35.jpg)
Expert Review • Scope • Sequence • Accuracy • Scenarios • Examples
SMEs
• Instructional strategies • Sequence • Practice • Mnemonics
Instructional experts
• Aesthetics • Metaphors • Icons • Navigation
Graphic designers
• Logistics
Teachers/ Trainers
• User experience • Story/narrative • Connections with
other systems
Interaction designers
• SCORM compliance/pkg • LMS integration • Metadata
LMS Administrators
![Page 36: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/36.jpg)
Interface Review Guidelines
from h`p://it.coe.uga.edu/~treeves/edit8350/UIRF.html
![Page 37: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/37.jpg)
How usable is this interface?
![Page 38: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/38.jpg)
Here’s what I wrote …
![Page 39: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/39.jpg)
User review in development
1-on-1 Observations
• Prototype Revision 1 • Try-out impressions;
obvious flaws; examples/scenarios
• 2 to 3 people • Instruments
– Observation Notes Form – Interview Protocol – Attitude Survey
Small Group Trials
• Prototype Revision 2 • Identify strengths and
weaknesses • 3 to 4 people • Instruments
– Observation Notes Form – Attitude Survey – Interview Protocol – Posttest/Learner
Performance
Observations from one-on-ones and small groups
![Page 40: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/40.jpg)
User review in development
In a contemporary model, users are likely involved early and through multiple iterations and multiple prototypes.
![Page 41: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/41.jpg)
What Is Usability?
![Page 42: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/42.jpg)
Two Major Methods to Evaluate Usability
Heuristic Evaluation
• Quick • Expert Analyses • No user involvement
User Testing
• Finds more problems • User involvement increases
validity • Seeing problems has a huge
impact on developers
![Page 43: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/43.jpg)
“The most common user action on a Web
site is to flee.” — Edward Tufte
![Page 44: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/44.jpg)
“At least 90% of all commercial Web sites are overly difficult to use …. the average outcome of Web
usability studies is that test users fail when they try to perform a test task
on the Web. Thus, when you try something new on the Web, the
expected outcome is failure.
— Jakob Nielsen
![Page 45: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/45.jpg)
Nielsen Web Usability Rules
1. Visibility of system status
2. Match between system and real world
3. User control and freedom
4. Consistency and standards
5. Error prevention 6. Recognition rather
than recall
7. Flexibility and efficiency of use
8. Help users recognize, diagnose, and recover from errors
9. Help and documentation
10. Aesthetic and minimalist design
![Page 46: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/46.jpg)
• Ease of learning - How fast can a user who has never seen the user interface before learn it sufficiently well to accomplish basic tasks?
• Efficiency of use - Once an experienced user has learned to use the system, how fast can he or she accomplish tasks?
• Memorability - If a user has used the system before, can he or she remember enough to use it effectively the next time or does the user have to start over again learning everything?
• Error frequency and severity - How often do users make errors while using the system, how serious are these errors, and how do users recover from these errors?
• Subjective satisfaction - How much does the user like using the system?
![Page 47: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/47.jpg)
![Page 48: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/48.jpg)
Heuristic Evaluation Process 1. Several experts
individually compare a product to a set of usability heuristics
2. Violations of the heuristics are evaluated for their severity and extent suggested solutions
3. At a group meeting, violation reports are categorized and assigned
4. average severity ratings, extents, heuristics violated, description of opportunity for improvement
![Page 49: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/49.jpg)
Heuristic Evaluation Comparisons Advantages • Quick: Do not need
to find or schedule users
• Easy to review problem areas many times
• Inexpensive: No fancy equipment
Disadvantages • Validity: No users
involved
• Finds fewer problems (40-60% less??)
• Getting good experts
• Building consensus with experts
![Page 50: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/50.jpg)
Heuristic Evaluation Report
![Page 51: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/51.jpg)
Heuristic Evaluation Report
![Page 52: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/52.jpg)
User Testing
![Page 53: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/53.jpg)
User Testing
• People whose characteristics (or profiles) match those of the Web site’s target audience perform a sequence of typical tasks using the site.
• Examines: – Ease of learning – Speed of task
performance – Error rates – User satisfaction – User retention over
time
![Page 54: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/54.jpg)
Image from (nz)dave at h`p://www.flickr.com/photos/nzdave/491411546/
![Page 55: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/55.jpg)
![Page 56: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/56.jpg)
![Page 57: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/57.jpg)
“For most companies…it 's fine to conduct tests in a conference room or an office — as long as you can close the door to keep
out distractions. What matters is that you get hold of real users and sit with them while they use the design. A notepad is the only
equipment you need.” — Jakob Nielsen
http://www.nngroup.com/articles/usability-101-introduction-to-usability/
![Page 58: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/58.jpg)
Elements of User Testing • Define target
users • Have users
perform representative tasks
• Observe users • Report results
Often called a user profile or persona.
Image from h`p://www.op2mum-‐web.co.uk/services/user-‐needs/personas/ & h`p://uxsuccess.com/2009/12/01/agile-‐personas-‐and-‐context-‐scenario/
![Page 59: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/59.jpg)
Why Multiple Evaluators?
• Single evaluator achieves poor results – Only finds about 35% of usability problems – 5 evaluators find more than 75%
![Page 60: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/60.jpg)
Why only 5 Users?
(Nielsen, 2000)
![Page 61: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/61.jpg)
Reporting User Testing
• Overall goals/objectives • Methodology • Target profile • Testing outline with test script • Specific task list to perform • Data analysis & results • Recommendations
![Page 62: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/62.jpg)
User Experience (UX)
from Jesse James Garrett | http://www.jjg.net/ia
![Page 63: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/63.jpg)
User Experience (UX)
from Peter Morville | http://semanticstudios.com/user_experience_design/
![Page 64: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/64.jpg)
User Experience (UX)
• Project Management • User Research • Usability Evaluation • Information Architecture (IA) • User Interface Design • Interaction Design (IxD) • Visual Design Content Strategy • Accessibility • Web Analytics
b Learner
![Page 65: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/65.jpg)
Current & Emerging Strategies for User Testing
![Page 66: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/66.jpg)
![Page 67: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/67.jpg)
![Page 68: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/68.jpg)
![Page 69: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/69.jpg)
![Page 70: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/70.jpg)
![Page 71: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/71.jpg)
Now defunct!
![Page 72: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/72.jpg)
![Page 73: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/73.jpg)
![Page 74: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/74.jpg)
A/B Testing
![Page 75: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/75.jpg)
usabilla
Morae
userzoom
![Page 76: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/76.jpg)
![Page 77: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/77.jpg)
10 Second Usability Test
1. Disable stylesheets 2. Check for the following:
1. Semantic markup 2. Logical organization 3. Only images related to content appear
![Page 78: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/78.jpg)
References & Acknowledgements
American Society for Training & Development. (2009). The value of evaluation: Making training evaluations more effective. Author.
Follett, A. (2009, October 9). 10 qualitative tools to improve your web site. Instant Shift. Retrieved March 18, 2010 from http://www.instantshift.com/2009/10/08/10-qualitative-tools-to-improve-your-website/
Image from http://www.flickr.com/photos/mutsmuts/4695658106/sizes/z/in/photostream/ Nielsen, J. (2000, March 19). Why you only need to test with 5 users. Jakob Nielsen’s Alertbox.
Retrieved from http://www.useit.com/alertbox/20000319.html Nielsen, J. (2012, January 4). Usability 101: Introduction to usability. NielsenNorman Group.
Retrieved from http://www.nngroup.com/articles/usability-101-introduction-to-usability/ Reeves, T.C. (2004, December 9). Design research for advancing the integration of digital
technologies into teaching and learning: Developing and evaluating educational interventions. Paper presented to the Columbia Center for New Media Teaching and Learning, New York, NY. Available at http://ccnmtl.columbia.edu/seminars/reeves/CCNMTLFormative.ppt
Reeves, T.C. & Hedberg, J.C. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications.
Usability.gov
Wu, S. (2015, June 22). 7 best pieces of user testing software. Creative Bloq. Retrieved from http://www.creativebloq.com/ux/best-user-testing-software-61515337
![Page 79: Arrogance or Apathy: The Need for Formative Evaluation + Current & Emerging Strategies](https://reader035.fdocuments.us/reader035/viewer/2022070513/58869aab1a28ab0c1d8b4955/html5/thumbnails/79.jpg)
Michael M. Grant 2015