Questionmark Breakfast Briefing: London, September 2012
-
Upload
questionmark -
Category
Technology
-
view
1.652 -
download
5
Transcript of Questionmark Breakfast Briefing: London, September 2012
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Breakfast Briefing
London
5-September-2012
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
9:00 – 9:30 Breakfast and Networking
9:30 – 10:45 • Welcomes and introductions • Brief Questionmark Overview • Assessment Technologies and Regulatory Compliance – John
Kleeman
10:45 – 11:00 Networking Break
11:00 – 12:30 Assessments Solutions enabled by Questionmark • Authoring, Delivery and Analytics • Mobile Delivery • New European Data Centre Case Study: HSBC – Matt Bushby Q&A
12:30 Close of Briefing
Agenda
Slide 2
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Slide 3
About Questionmark
Founded in 1988
Technologies and services that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes – Assessment Management System: On Demand
On Premise
More than 2,500 customers worldwide
Offices in UK, US, Belgium and Germany
Questionmark Perception available in English, Spanish, German, French, Dutch, Norwegian and Japanese Participant facing software in 20 languages
Questionmark Live SME Authoring available in 15 languages
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
John Kleeman, Questionmark
Chairman
Ivan Forward, Assessment Solutions
Consultant
Che Osborne, VP of Sales
Kerry Williams, UK Sales Manager
Chloe Mendonça, Marketing Assistant
Leila Homps, Account Manager
Carla Tavares, International Account
Executive
Questionmark Staff at Briefing
Slide 4
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Education
• Improve learning outcomes
• Increase student satisfaction
• Data for institutional effectiveness, accreditation
Learning, Training & HR in Business and Government
• Regulatory Compliance
• Heath and Safety
• Sales and Technical Channel Verification
• Employee Selection
Awarding Bodies / Certification / Test Publishers
• Efficiently create and maintain valid, reliable tests
• Generate revenue from exams and practice tests
Questionmark enables customers to address key objectives…
Slide 5
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessment Solutions Enabled
Diagnostic/ Placement
tests
Pre- and Post-course tests
Quizzes & Knowledge
Checks
Summative assessment
Course evaluation
Observational Assessments
Certification & licensing
exams
Job Task Analysis
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Global Manufacturing and Technology Company
Deploying OnDemand for Sales Channel Verification Assessment Program to Sales Reps Worldwide
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Power Generation and Distribution Utility in Midwest
Deploying Assessments for 7,000 employees to test for safety, regulatory compliance and certification of knowledge
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Large Automotive Parts Retailer with 50,000+ Employees
New OnDemand Customer Deploying Product Knowledge Assessment Program at 3,000+ US Retail locations
Slide 9
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Indian government agency responsible for Technical College entrance exams
All entrance exams will be delivered via Questionmark OnDemand
Slide 10
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Global Services Company Deploying Questionmark for Drivers Examinations
Helped firm address security concerns
Slide 11
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Large Medical Certification body deploying Questionmark for Observational Assessment
Objective Structured Clinical Examination (OSCE)
Slide 12
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Large Insurance Company deploys Questionmark for
Compliance and On-boarding 200,000+ assessments annual for new hires and compliance
testing
Slide 13
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Global IT company for partner certification
300,000 at home assessments
50,000 in test centers
Slide 14
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Large Transportation company deploys Questionmark with
SAP Connector 15,000 Pre-employment tests
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Nationwide College Entry Exams Deployed on Questionmark
in India 3 million exams per year
Slide 16
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessment Technologies and Regulatory Compliance
John Kleeman, Questionmark Chairman
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
3 huge fines in summer 2012
Slide 18
Can assessments make compliance easier and reduce risk?
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Setting the scene
5 reasons why assessments mitigate compliance risk 1. Demonstrate commitment
2. Find problems early
3. Document understanding after training
4. Increase knowledge / reduce forgetting
5. Reduce cost & time for compliance training
5 key pieces of practical advice on using assessments
Agenda
Slide 19
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Primarily focused on financial structure
Banks need capital and must not take risks …
But employee performance is part of this
Basel guidelines say: compliance function should assist senior management in
educating staff on compliance issues
Financial Services : Basel II and III
Slide 20
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Combat Terrorism Organized crime
FATF guidelines include a risk-based process
“testing to assess knowledge commensurate with the detail of information provided”
UK FSA example of poor practice: “employees were merely required to sign a form to confirm that
they were aware of the firm’s AML procedures, but their understanding was never tested.”
Financial Services : Anti-money laundering
Slide 21
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Two main impetuses US Foreign Corrupt Practices Act
UK Bribery Act
Both extra-territorial
Penalties can be large US$800m settlement with Siemens
US government being much more proactive
UK getting into act
Training and procedures against bribery can be a defence – and stop fines
All industries : anti-bribery and corruption
Slide 22
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Consumer protection Testing sales people
Industry technical regulations
General business regulation Accuracy of accounts
Anti-competition laws
HR (e.g. equality)
Health and Safety
Privacy and data protection
Other important regulatory drivers
Slide 23
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #1 : Demonstrate commitment
Slide 24
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Because employees follow their leaders Charles Jennings (August 2012)
“… common thread that runs through almost all high-profile compliance catastrophes. It is that the top-tier executives and middle managers in the organisations simply don’t model the behaviours that will lead to a culture of compliance”
Why is it important to demonstrate top level commitment?
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Require all employees and managers/executives to Take assessments every 6 or 12 months
Remedial training and retake assessment if fail
Assessments seen as fair
Major stride in demonstrating commitment
How do assessments demonstrate commitment?
Slide 26
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
UK FSA 2009: “Aon Ltd should have ensured that appropriate members of
staff – particularly those in the Aviation and Energy divisions – received focused training in relation to this area and were tested on their understanding of the relevant risks involved. Effective training and testing in this regard would have emphasised to staff the importance of carrying out effective due diligence prior to authorising an Overseas Third Party for payment.”
Fine of UK£5.5m for not testing, partly due to lack of commitment
Slide 27
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #2 : Find problems early
Slide 28
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Surveys identify issues in culture and ethics
Most employees want to help. From employee of bank involved in recent LIBOR scandal: “Always happy to help, leave it with me, Sir”.
UK Ministry of Justice Bribery Act Guidelines “Staff surveys, questionnaires and feedback from training can also provide an important source of information on effectiveness and a means by which employees and other associated persons can inform continuing improvement of anti-bribery policies”.
How surveys can find problems early
Slide 29
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Look at topic and assessment scores By department
By geography
Over time
Look at wrong answers
Look at test failures
Assessments touch every employee: a unique tool
How tests/quizzes/exams can find problems early
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #3 : Document understanding after training
Slide 31
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Because regulators require it common regulatory requirement, e.g. OECD recommends on
ethics/compliance programme:
measures designed to ensure periodic communication, and documented training for all levels of the company
Because makes business sense Trained employees more effective
“If you can not measure it, you can not improve it.“
Why document training for compliance?
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Take attendance
Get employee signatures
Test to check understanding
Common Methods of Documenting Training
Slide 33
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How to do Trainer records attendance or takes photo of attendees
Record online attendees
Pros Easy to do
Measures who turns up
Cons Did they stay the whole time?
Did they pay attention?
Did they get it?
Take attendance
Slide 34
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How to do Get employees to sign a form confirming
that they attended training AND that they understand the rules
Pros Gets employee confirmation
Employee committed to follow rules
Cons People may sign but not take seriously
People may sign without attending or understanding
Get employee signatures
Slide 35
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Not where safety is involved according to 2005 United States Appeal Court: “merely having an individual sign a form acknowledging his responsibility to read the safety manual is insufficient to insure that the detailed instructions contained therein have actually been communicated.”
Not according to this UK FSA example of poor practice: “… employees were merely required to sign a form to confirm that they were aware of the firm’s AML procedures, but their understanding was never tested.”
Does getting an employee to sign give protection?
Slide 36
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How to do: After training, test employees People who fail test undergo
remedial training
Pros Genuinely check understanding Allows improvement of weaknesses Feedback to improve training
Cons More effort You have to follow through & take
action on failures
Test to check understanding
Slide 37
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #4 : Increase knowledge / reduce forgetting
Slide 38
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Compliance Training
Slide 39
Receive information
Working memory
Long term memory
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Cue
• Question
• Other stimulus
Search • Retrieve from long
term memory
Act • Act or respond
Psychology research shows retrieval practice helps future retrieval
Successful retrieval modifies memory making it more retrievable in future
Training useful if can retrieve
Slide 40
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
In a Perfect World…
Slide 41
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
In a Perfect World… we would remember everything we learn!
Slide 42
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Learning in the Actual World They don’t get it All!
Slide 43
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Learning Curve
Slide 44
Ouch! The Forgetting Curve
Time
Kn
ow
led
ge/S
kills
Lear
nin
g
Exp
erie
nce
Lea
rnin
g
Exp
erie
nce
Lea
rnin
g
Exp
erie
nce
Qu
esti
on
s Q
ues
tio
ns
Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall
Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
In the Actual World… we forget some of it!
Slide 45
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Slow the Forgetting Curve
Slide 46
Time
Kn
ow
led
ge/S
kills
Lear
nin
g
Exp
erie
nce
Lea
rnin
g
Exp
erie
nce
Lea
rnin
g
Exp
erie
nce
Qu
esti
on
s Q
ues
tio
ns
Qu
esti
on
s Q
ues
tio
ns
Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve
Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reason #5 : Reduce cost & time for compliance training
Slide 47
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Take test If pass, skip
training Otherwise do training
Testing out of compliance training
Slide 48
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Management Report Example
A) Assessment Completions 1000
B) Pass the assessment and test out 375
C) Do the compliance training 625
D) Time for compliance training 1.5 hours
E) Hours of training avoided (B * D) 562.5
F) Average cost / hour $70
G) Money saved (E * F) $39,375
Example of how can save money
Slide 49
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
1. Demonstrate commitment
2. Find problems early
3. Document understanding after training
4. Increase knowledge / reduce forgetting
5. Reduce cost & time for compliance training
Summary on why to use assessments
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Good practice in using assessments
5 key practical steps for compliance assessments
Slide 51
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Medium and higher stakes Internal exams Post-course tests Diagnostic tests Observational/workplace assessments
Lower stakes assessments
Course evaluation surveys Employee attitude surveys Job task analysis surveys Practice tests Quizzes during learning 360 degree performance assessments
Common assessments in compliance
Slide 52
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Create scenario questions Assess application not just knowledge
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Evaluation
Synthesis
Analysis
Application
Comprehension
Knowledge
Questions better if use application / comprehension
NOT just knowledge
Blooms Taxonomy
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What does a yellow traffic light mean? Stop
Go
Caution
If you are driving toward an intersection and the light turns from yellow to red, what should you do? Speed up and cross the
intersection
Stop suddenly
Stop gradually
Example
Slide 55
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Use feedback at the topic level It is actionable
Slide 56
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What to do Divide assessment by topic areas
Give individual feedback
Topic scores
Resources or advice to improve weak topics
Analyze at department or organization level
Topic feedback is actionable Direct participant to specific resources to improve learning
Direct compliance to areas knowledge is weak
How topic feedback can help
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Feedback based topics of questions
Slide 58
Corrective feedback points participant to appropriate learning
resources
“Topics” of questions
answered in assessment
Score on topic
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Make assessments accessible Any large organization will have visually impaired or other participants with accessibility needs
Slide 59
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Accommodations to consider
Slide 60
Disability
Loss of vision
Visual impairment
D/deafness, hard of hearing
Motor impairment
Cognitive impairment
Accommodation
Screen reader support
Contrast and text size
Text alternatives
Keyboard accessible
Text size, extra time
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Screenshots in questions
Use of color
Long text that is hard to read
Simulations & graphics using Flash
Time limits (may need to extend for some)
Make sure your assessment software supports accessible assessments
Be careful about
Slide 61
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Have participants agree a code of conduct Help them be honest
Slide 62
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Fraud needs: Motivation
Opportunity
Rationalization
Fraud triangle
Slide 63
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Motivation depends on stakes
Slide 64
Higher Stakes
High
Medium
Low
Medium Stakes
Low Stakes
What’s at Stake?
Life and Limb
Promotion & Jobs & Legal Concern
Educational Exams
Tests
Elearning & Surveys
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Impersonation Content Theft Cheating
Opportunity
Slide 65
Monitoring/proctoring helpful, e.g. having manager supervise employees
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reduce rationalization to cheat Make the programme fair
Communicate widely that it is fair
A code of conduct (honor code) can help
Idea borrowed from universities
Employees commit at start of assessment to be honest
Rationalization
Slide 66
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Consider observational assessments Measure skills and behavior
Slide 67
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Traditional assessment
• Participant is presented with questions for direct response
Observational assessment
• Observer is presented with questions to ‘rate’ participant to measure knowledge, skills and abilities
Observational assessment vs. ‘traditional’ quiz, test or exam
Slide 68
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Preparation Performing the
task Wrap Up
Observer typically rates participant in 3 areas
Smartphones/tablets make more practical
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
1. Create scenario questions
2. Use feedback at the topic level
3. Make assessments accessible
4. Have participants agree a code of conduct
5. Consider observational assessments
Actionable good practice
Slide 70
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Excellent book on assessment Criterion-Referenced Test
Development
by Shrock and Coscarelli
More good practice
Slide 71
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark white paper
www.questionmark.com/uk/whitepapers
More good practice
Slide 72
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Your questions
Slide 73
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Break
Slide 74
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Create • Collaborative, secure
authoring of surveys, quizzes, tests and exams
Deliver • Blended delivery to
support a broad range of devices and security needs
Analyze • Evaluate results and
inform stakeholders with timely, meaningful reports
Assessment Management
System
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Create • Collaborative, secure
authoring of surveys, quizzes, tests and exams
Deliver • Blended delivery to
support a broad range of devices and security needs
Analyze • Evaluate results and
inform stakeholders with timely, meaningful reports
Software-as-a-Service
For OnPremise Deployment
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
New European Data Centre
Slide 77
ISO 2700 and 9001 accredited
State-of-the-art datacenter
SaaS / Rapid Deployment
24/7 Monitoring
Global 24/7 support available
99.9+% Uptime
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Authoring
Delivery
Analytics
Integration
Slide 78
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Author, Deliver and Analyse via the Cloud
Internet
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Distributed authoring and item management
Internet
Assessments
Questions
Windows based
Authoring Manager
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Slide 81
Wizards to create 21 item types
Drag and Drop *Essay *Explanation Fill in Blanks Hotspot *Likert Adobe Flash Adobe Captivate Matching Knowledge Matrix Survey Matrix
*Multiple Choice *Multiple Response Numeric File Upload *Pull-down List Ranking Select a Blank *Text Match *True/False *Yes/No
Enable authors to create more interesting and probing questions
Reveals more information during an assessment in a shorter time
Not offered by simpler assessment tools in LMS’s
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Subject Matter Experts Author
Questions in Questionmark Live
Questionmark Live: Simplify Item Writing, Content Harvesting from SMES
Perception Repository
SME content
SME questions
SME questions
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Interactive Translations Using Questionmark TMS
Translations using 3rd Party Tools
Translation Management System
Translation methods
Slide 83
Export to XLIFF (XML)
Translate using 3rd Party Tools
Import from XLIFF (XML)
Translation
Management
System
Manages translation process ….does not do translation
Set Up Project using TMS
On-The-Fly
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Demonstration
Slide 84
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Authoring
Delivery
Analytics
Integration
Slide 85
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Formative Quizzes Course Evaluations
Diagnostic Tests
Low Stakes Tests
Medium Stakes Exams
High Stakes Exams
Levels
of M
onitoring
Blended Delivery
Stakes of Assessment
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Browser Support Critical for Reliable Participant Experience
Slide 87
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Browser Support Critical for Reliable Participant Experience
Slide 88
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Support for Leading Browsers
Slide 89
Internet Explorer
Firefox Safari
for Mac Chrome
Safari for iOS 4 & 5
Android 2+
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Auto Sense Device/Browser
Slide 90
Senses Device/Browser
Questionmark Packages the Assessment for Device/Browser/iframe
On-the-fly auto-sizing of skin and controls
Sends HTML
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Demonstration
Slide 91
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Mobile Delivery of Assessments
Observational assessments
Mobile exam rooms
Deliver to the device in your
hand
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Mobile Apps
Android
iPhone, iPad and iPod touch
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Observer rates participants’
performance, skills and abilities
Preparation Performing the
task Wrap Up
Observational Assessment
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Uses of Observational Assessment
Certification Process
• Observational assessments can be used as critical components of the certification process – in some cases are required by regulatory authorities.
Medical and Dental School exams:
• Medical and dental schools make extensive use of observational assessments such as the Objective Structured Clinical Examination (OSCE).
Equipment operation:
• Observational assessment can be used to record and rate operators' skills, abilities, safety practices and adherence to required procedures
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
~70% from Experience
~20% from Others
~10% from Study
Effective application of 70+20+10 requires flexible learning and assessment technologies
Slide 96
Wikipedia: http://en.wikipedia.org/wiki/70/20/10_Model
Dr. Charles Jennings
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Wikis
Blogs
Forums
Social Networking
Portals
Technologies used to support social/informal learning initiatives
Slide 97
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Embedding Assessments in Wikis, blogs, portals
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Benefits of Embedding in Blogs, Portals, Wiki’s
For Learners For Organizations
Focus the learners’ attention
Practice memory recall
Identify what was learned
Provide feedback to assist the learner to: Correct misconceptions
Instill confidence
Point to relevant material
Embrace 70+20+10 with confidence
Easily use assessments throughout the learning process
Provide seamless user experience
Centralized management of assessment content and results
Slide 99
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Authoring
Delivery
Analytics
Integration
Slide 100
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Did the learning investment impact
performance?
Where are the skills and
knowledge gaps in my organization?
Are my questions and tests valid and
reliable?
What did participants think
of the learning experience?
What did participants learn from the training
or course?
How do I coach participants to
improve performance?
How did one group of test takers compare to
another?
Analyze and report on results
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Demonstration
Slide 102
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Authoring
Delivery
Analytics
Integration
Slide 103
Can I integrate Questionmark with various other Enterprise Systems?
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Integrating with SharePoint
Slide 105
Embedding assessment lists using
SharePoint Connector for OnDemand
Embedding assessment lists and assessments using
Windows Authentication
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
SAP Connector
Slide 106
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Blackboard Connector (v8 and 9)
Slide 107
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
• Content
• Delivery
• Results
Secure
• Millions of tests delivered
• 99.99% Uptime
• Tests available globally
Scalable
• Author in Browser
• Multilingual
• Mobile, Blended Delivery
Agile
• Accessible
• Reliable
• 24/7 Support
Trustable
Why Questionmark?
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Ways to Evaluate
Slide 109
Try Questionmark OnDemand
• 30 Day Trial – Software as a Service
• https://www.questionmark.com/ondemand
Try Questionmark Perception
• Download – 30 day trial
• https://www.questionmark.com/download
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Slide 110
Ways to Learn More
Questionmark Blog!
• http://blog.questionmark.com
Web Seminars
• http://www.questionmark.com/uk/seminars
White Papers
• http://www.questionmark.com/whitepapers
Manuals, Knowledge Base and Best Practices Guides
• http://www.questionmark.com/perception/help
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
The Role of Assessments in Mitigating Risk for Financial Services Organizations
Assessments through the Learning Process
Security of Questionmark’s OnDemand Service
Alignment, Impact and Measurement with the A-model
Embedded Assessments: Building Knowledge Checks, Surveys and Other Assessments into Learning Materials
How Questionmark Aids ISO 10667 Compliance
Recommended White Papers
Slide 111
https://www.questionmark.com/whitepapers
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Thank you for attending – Keep in Touch!
Slide 112
blog.questionmark.com
www.questionmark.com/newsletter
twitter.com/questionmark
linkedin.com Look for Questionmark under “Groups” and join
youtube.com/questionmarkvideo
flickr.com/questionmarkpictures
facebook.com Like us on Facebook