Introduction to Questionmark’s Assessment …€¢Assessment technologies and portal solutions...

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

A s s e s s f o r S u c c e s s

Questionmark Conference 2018 Savannah

Introduction to Questionmark’sAssessment Management Technologies

Rory Northam

Product Evangelist, Questionmark

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Assessment Solutions enabled by Questionmark Assessment Management System Overview

▪ Collaborative Authoring Demonstration

▪ Blended Delivery Demonstration

▪ Reporting and Analytics Demonstration

▪ Integration

Q&A

Agenda

Slide 2



• Founded in 1988

• Offices in US, UK, Canada; Partners worldwide

• Assessment technologies and portal solutions that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes

• ISO/IEC 27001 Certified (Learn more at www.questionmark.com/trust)

Background

• 1100 Enterprise customers worldwide

• Corporate, Government; Higher Education; Certification Providers; Awarding Bodies

Customers

About Questionmark

http://www.questionmark.com/trust



Assessment Management System

Collaborative, secure authoring of surveys, quizzes, tests, and exams

Reporting to analyze results and inform stakeholders

Blended delivery to support a broad range of devices and security needs

Multilingual: 36 Languages

Global 24/7 Support



OnDemand OnPremiseOnDemand for Government

Questionmark Deployment Options



Measure and Enhance Learning

•Measure knowledge, skills and attitudes –before, during and after learning

•Correct misconceptions and increase knowledge retention

Qualification and Certification

•Deliver reliable, defensible tests to certify employees

•Ensure sound personnel decisions

Regulatory Compliance

•Ensure knowledge of essential laws, regulations and health/safety guidelines

•Demonstrate compliance and reduce risk

Sales/Technical Channel

•Competence/product expertise of sales and technical professionals

•Increase revenues, customer satisfaction

•Reduce costs

Credentialing

•Efficiently create and maintain valid, reliable tests

•Generate revenue from exams and practice tests

Higher Education

•Increase student satisfaction

•Improve learning

•Reduce Cheating

Why our customers use assessments



Browser-based, collaborative environment▪ Role-based access control

Manage item bank▪ Organize items by topic, subtopic and

metatag▪ Item version control▪ Import items, assessment and other

content

Author items and assessments

Publish assessments for delivery

Authoring and Item Banking

Slide 8



Slide 9

Many item types to choose from…

Multiple choice

Multiple Response

Matching

Ranking

Pull-down List

Select-a-blank

Matching

Job Task Analysis

Knowledge Matrix

Survey Matrix

Likert scale

True/False

Yes/No

Text Match

Drag-and-Drop

Hotspot

Numeric

Essay

Fill-in-the-blanks

Explanation

File Upload



Demonstration

Slide 10



Question-level feedback

Slide 16

Question with participant’s answer

Corrective feedback



Feedback based topics of questions

Slide 17

Corrective feedback points participant to appropriate learning

resources

“Topics” of questions

answered in assessment

Score on topic



Provide users role-based access to content

Provide users role-based access to content

Roles & Permission Management



Item version history and rollback

Slide 19

Revision History

Audit trail of edits and option to “rollback”

Easily “track changes” made in collaborative authoring or review



Search description, stimulus, choices, status, meta tags,

dates and more

Item SearchSearch item bank to find items that contain certain words and/or tags



Multilingual Interfaces

Slide 22

Arabic ( (العربية Bulgarian (Български) Catalan (Català) Chinese, Simplified (简体中文)

Chinese, Traditional (繁體中文)

Croatian (Hrvatski) Czech (Čeština) Danish (Dansk) Dutch (Nederlands) English Finnish (Suomi) French (Français)

German (Deutsch) Greek (Ελληνικά) Hebrew ( (עברית Hindi (हिन्दी) Hungarian (Magyar) Indonesian (Bahasa

Indonesia) Italian (Italiano) Japanese (日本語) Korean (한국어) Norwegian Bokmål

(Bokmål) Persian ( (فارسی Polish (Polski)

Portuguese, Portugal (Português)

Romanian (Română) Russian (Русский) Serbian (Српски) Slovenian (Slovenščina) Spanish (Español) Swedish (Svenska) Thai (ภาษาไทย) Turkish (Türkçe) Ukrainian (Українська) Urdu ( (اردو Vietnamese (Tiếng Việt)



Translation Management for Multilingual Assessments



Blended, Secure Delivery

Slide 26



Author Once… Deliver Anywhere

Slide 27

Single results set

Available Securely

Author once Deliver to any devicein context



Responsive Design, Blended Delivery

One Assessment…

…many devicesand browsers



Demonstration

Slide 30



Print your own bubble sheets

Participants enter responses

Scan and upload for scoring

Analyze and report on results

Printing and Scanning:Deliver assessments online or on paper

Slide 32

Deliver assessments on paper or online Print your own question and bubble sheets Use your own scanner to read responses Upload scanned bubble sheets for scoring Scanned assessment results available for reporting



Delivering high stakes assessments

Quizzes & Surveys

Diagnostic Tests

Low Stakes Tests

Medium Stakes Tests

High Stakes Exams

S e c u r i t y R e q u i r e m e n t s

Sta

ke

s o

f A

ss

es

sm

en

t



“Save as you go”

Results saved at the server “as you go” so IT failures don’t lose data

Hope for the best; prepare for the worst

Slide 34



Secure browser for high-stakes assessments

Reduce content theft, cheating; improve exam integrity

Questionmark SecureOne Setting – Multiple Platforms

Slide 35

Windows Mac iPad

PrintingScreen captureTask switching

Control/system keysRight-click / context menu

optionsView source

Accidentally exiting exam

Prevents….



Require Proctoring

Slide 36



Convenience of online exam delivery at home or workplace under the watchful eye of a proctor▪ Works with most standard webcams/audio

▪ Live proctor can start, pause, and stop exam

More info:▪ Website

▪ Your account manager

Online Proctoring

Slide 37

www.questionmark.com/proctoring

Powered by



Accessibility

Also support for…- Non-mouse navigation; alternate pointing devices- Screen readers



Questionmark BadgingRecognize Assessment Outcomes with Verifiable, Portable Digital Credentials

Slide 39

Integrated with industry-leading digital credentialing platform



Single-Sign-On

• SAML

Launch and Track via AICC or SCORM

• SuccessFactors, Cornerstone, etc.

Learning Tools for Interoperability (LTI)

• Blackboard, Canvas, D2L, Moodle, etc.

Web services API

• QMWISe

OData APIs

• Results API

Integration

Slide 40



Publish to LMS



Reporting and Analytics

Slide 42



Reporting and Analytics

Slide 43

What can we learn from assessments?

Did they like it?

Did they learn it?

Can they do it?

Can we prove it?



What you can learn from Questionmarkreporting and analytics

How are our Items and Assessments doing? How are our Participants doing?

Did the learning investment impact

performance?Where are the skills and knowledge gaps in my

organization?

Are my questions and tests valid and reliable?

What did participants think of the learning

experience?

What did participants learn from the training

or course?

How do I coach participants to improve

performance?

How did one group of test takers compare to

another?



Item and Test Analysis Item Analysis report Test Analysis report Score list report Test center analysis report Assessment completion time report Question frequency analysis report Demographic report Question type report Assessment Overview report Assessment results over time report

Participant-focused reports Score list report Coaching Report Transcript Report Grade Book Report

Survey Reports Survey Frequency Analysis Job-task-analysis (JTA)

▪ JTA Summary▪ JTA dimension by demographic

Course Evaluation▪ Course Summary Report▪ Instructor Summary Report▪ Class Summary Report▪ Class Detail Report

Types of reports available

Slide 45



Score List Report

Slide 46

Demonstration



What it measures:▪ % answered correctly

How it is expressed:▪ Number from 0.0 to 1.0

How to interpret:▪ Too high = too easy▪ Too low could mean…

Too difficult

Confusing/ambiguous

Miscoded

What it measures:▪ Correlation between

item score and test (assessment) score

How it is expressed:▪ Number between

-1.0 and +1.0.

How to interpret:▪ Generally…

< 0.2 low

> 0.3 good

Item Analysis: Reliability of Question Results Data

Difficulty index (aka P “value”) Item correlation discrimination:



Slide 48

Difficulty/P-Value

Correlation -Discrimination



Slide 49



John Kleeman’s

Item Analysis for Beginners

Thursday, March 8, 13:30 – 14:45



Slide 51

Coaching ReportCompare against

benchmarks or other participants

Participant

Topiclevel

Overall



Job Task Analysis (JTA) Reports

Slide 52



Course SummaryInstructor Summary

Class Summary Class Detail

Four Course Evaluation Reports:

Slide 53



Built-in Reports Custom Reports and Dashboards

Microsoft ExcelPowerPivot

BI Tools

Big Data Applications: Results API

Slide 54

R e s u l t s W a r e h o u s e



•Planning

• Installation

• Integration

Deployment

•Assessments

• Importing items

•Templates / Look & feel

Authoring

• Item reports

•Other custom reports

Reporting

•Testing and Assessment Portals

•People Data syncing

•Results Data syncing

•Self Registration

•Auto Scheduling

•Assessment Clustering

•Certificate Generation

•Distributed Administration

•Assessment Catalogs

Solutions & Integration

Consulting / Solution Services

Slide 55

w w w. q u e s t i o n m a r k . c o m / c o n s u l t i n g



Summary of key benefits

Collaborative Authoring

• Secure

• Browser-based

• Available globally

Blended Delivery

• Responsive Design

• Browser support

• Tablets/smartphones

• Accessibility

• Secure delivery for high stakes exams

Reporting and analytics

• How are my participants doing?

• How are my tests doing?

• Results API

Multilingual

• Translation Management

• Screens/Controls in 36 languages

• Right-to-left languages

Integration

• Supports industry standards

• APIs

• Connectors



What questions do you have?



www.questionmark.com/whitepapers

Recommended eBooks and White Papers

Slide 59

http://www.questionmark.com/whitepapers

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

A s s e s s f o r S u c c e s s

Questionmark Conference 2018 Savannah

Follow us…

Thank you for attending!

We hope to see you at future webinars

Keep up to date at blog.questionmark.com!

https://www.questionmark.com/go/sn004




http://blog.questionmark.com/

Introduction to Questionmark’s Assessment …€¢Assessment technologies and portal solutions...

Documents

Transcript of Introduction to Questionmark’s Assessment …€¢Assessment technologies and portal solutions...