Assessments Through The Learning Process
description
Transcript of Assessments Through The Learning Process
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessments Through The Learning ProcessOctober 2013Eric Shepherd. Questionmark, [email protected] www.questionmark.com
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Thank youSlide 2
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
You have colored cards on your chairs/tables Get one of each color, lettered A, B, C and D Let’s practice holding up cards
During this presentation we’ll be using questions to show you the power of questions
Please answer quickly and don’t be embarrassed – we are here to learn!
Using the cards
Slide 3
A B C D
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learners generally prefer response cards to simple hand-raising
Response Cards Shown to Improve classroom engagement amount they learn amount they retain
-- Dr. Will Thalheimer, citing multiple examples of research on the use of response cards
Research Suggests…
Slide 4
A BC
D
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
100 chests of tea
34 chests of tea
342 chests of tea
342 tea bags
How much tea was dumped into Boston Harbor at the Boston Tea Party?
Slide 5
B
C
A
D
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
342 chests of tea
Over 46 tons of tea Worth (at the time) over £10,000
£678,827 using the retail price index £866,694 using the GDP deflator £7,422,564 using average earnings £10,025,165 using per capita GDP £25,350,172 using the GDP
Citation: Lawrence H. Officer, "Five Ways to Compute the Relative Value of a UK Pound Amount, 1830 - 2005" MeasuringWorth.Com, 2006.
Answer:
Slide 6
C
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning Without Questions
Slide 7
Content
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning Benefits of Questions
Slide 8
Content
Questions
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
…expose poor presentation skills
…stimulate thinking and keep you awake
…frustrate our color blind friends
All of the above
We are using answer cards to…
Slide 9
B
C
A
D
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Provide Assessment Management Systems To create, deliver and report on assessments For exams, tests, quizzes, and surveys
More than 1,500 customers have purchased Questionmark Corporate, Government, and Education markets
More than 14,000 authoring systems installed
Last month users created 40,000+ questions in Questionmark
Offices in US, UK, Canada, Germany, Belgium
Available in numerous languages
Questionmark
Slide 10
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Worldwide Practical Experience
Slide 11
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Alberta Health Services Aluminerie Alouette Inc. Apache Corporation ATB Financial British Columbia Hydro Canada Border Services Agency - CBSA CMLTO - College of Medical Laboratory
Technologists of Ontario Department of National Defence Everest Colleges Canada Inc ExxonMobil Flakeboard Company Limited Fortis BC Inc Grant MacEwan University International Competency Assessment Board Invesco Canada Ltd. Justice Institute of British Columbia
Loyalist College Manitoba Hydro Memorial University of Newfoundland Moneris Solutions National Association of Pharmacy Regulatory
Authorities Northern Ontario School of Medicine Omnivex Corporation Plains Midstream Safety Codes Council Scotiabank Telus : Telus Supporting Units University of Calgary : Faculty of Veterinary
Medicine - University of Calgary University of Manitoba - Faculty of Medicine University of Ottawa University of Ottawa - Faculty of Medicine University of Saskatchewan
Sample List of Canadian Customers
Slide 12
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Who Else Uses Questionmark?
Slide 13
Financial Services
Pharma / Health
High Tech / Manufacturing
Telcom Energy / Utilities
Public Sector EducationTransportation
Prof. Services
Retail
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
…brain dump; tell me everything
…I don’t know; say something interesting
…hurry up; I am only here for the networking
…why so many questions?
What would you like from this session?
Slide 14
B
C
A
D
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Things that Drive What-To-Do
Slide 15
Increase Revenue = € £ $ ¥
Reduce Costs = € £ $ ¥
Reduce Time to Market
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Practical Limitations for Planning
Slide 16
Mon
ey =
€ £
$ ¥
How Long Do We Have?
People or Talent
Resources
Laws
Health &
Safety
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Increase revenue, time-to-market, laws
Increase revenue, reduce costs, safety
Increase revenue, reduce costs, time-to-market
Money, time-to-market, laws and talent
What are the business drivers that drive the use of learning and assessments?
Slide 17
B
C
A
D
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Money, health and safety
Money, laws, health and safety
Money, laws, resources, and safety
Money, laws, time-to-market, health and safety and, access to talent and resources
What are the practical limitations of using assessments?
Slide 18
B
C
A
D
Strategy
Tactics &Plans
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Tactics &Plans
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Tactics &Plans
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Slide 19
Strategy
Tactics &Plans
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Tactics &Plans
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Tactics &Plans
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Job Roles
Resources
Tasks
Knowledge, Skills,
Abilities
Slide 20
Knowledge, Skills and Abilities Required
How To Do It
What Will They Do
What Will They Use
Who Will Do It
What To Do
Slide 21
Knowledge, Skills and Abilities Required
How To Do It
What Will They Do
What Will They Use
Who Will Do It
What To Do
Competencies
Slide 22 Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Mental Check to Confirm a Competency
“Given What Do What”
• Time• Tools• Materials• Circumstances• Data
Given What
• Do using Competencies
Do What • Very Well• Well• Poorly
Perform
Slide 23 Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Knowledge (a cognitive Skill)•Memory is a cognitive skills that provides
ability to recall knowledge:•“Knowledge How”•“Knowledge To”
Above Knowledge Skills• Cognitive skills above recalling knowledge
(Comprehension , Application, Analysis, Synthesis and Evaluation)
Observable Skills - Motor Skills i.e. physical movement/coordination- Inter-personal skills
Attitudes and Behaviors•Motivation•Ability to achieve goals•Team fit
Ability to Do/Perform“Competency”
Competency is the “Ability To Do”(or “Ability To Perform”) and requires….
Cognitive Skills
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Results Oriented Ability to meet and exceed goals, make difficult decisions, act on
opportunities that create value for the company. Demonstrate ability to exceed stated job responsibilities, take informed
risks, make principled decisions, and faces high-stakes issues head on. Create a work environment where others feel it is safe to take risks.
Customer Results Consistently meets Customer Needs. Provides solutions for
external and internal customers that meet and/or exceeds customer expectations. Identifies opportunities that will benefit the customer and works in partnership with the customer to drive the business forward.
Example - Attitudes and Behaviors
Slide 24
Slide 25
Competencies Required
Knowledge, Skills and Abilities Required
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
How do we know the Knowledge, Skills and Abilities required? Job Task Analysis
Survey people who do the job
MoSCoW analysis Knowledge, Skills and Abilities that we:
Must have Should have Could have Would be nice to have
Slide 26
Knowledge, Skills and Abilities Required
Assessment
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 27
Skill Survey, Quiz, or Test,
What We Need
Knowledge, Skills and Abilities Required
Knowledge
Skills
Abilities
What We Have
Knowledge
Skills
Abilities
Assessment
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 28
What We Need
Knowledge, Skills and Abilities Required
Knowledge
Abilities
What We Have
Knowledge
Skills
Abilities- =Skills
Topics ToTrain
Knowledge
Skills
Abilities
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Helps us survey skills
Helps us know what actions to take
Know what knowledge we have
Determine qualified candidates
Which is the best answer to the question… Why are skill surveys useful?
Slide 29
B
C
A
D
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning
Assessment Thru Learning Process
Evidence Accumulated
Windows CDPrepare For Learning
Windows CDIn/As LearningAssessments
Windows CDOf-LearningAssessments
Mentoring InstructorCollaborative & E-learning Work Place
Slide 30
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning
Assessment Thru Learning Process
Evidence Accumulated
Windows CDPrepare For Learning
Windows CDIn/As LearningAssessments
Windows CDOf LearningAssessments
Windows CDJob Task Analysis Windows CDCourse Evaluations
Windows CDNeeds Analysis Mentoring InstructorCollaborative & E-learning Work Place
Windows CDSlow ForgettingCurve
Slide 31
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning
Pre-Learning Assessments
Evidence Accumulated
Mentoring InstructorCollaborative & E-learning Work Place
Purpose:• Create intrigue• Route to course• Inform instructors• Establish benchmarks
Windows CDPrepare For Learning
Windows CDIn/As LearningAssessments
Windows CDOf LearningAssessments
Slide 32
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning
In-Learning Assessments
Evidence Accumulated
Mentoring InstructorCollaborative & E-learning Work Place
Purpose:• Encourages focus on learning• Provide instant feedback to
instructor • Stimulate and strengthen memory• Correct misconceptions
Windows CDPrepare For Learning
Windows CDIn/As LearningAssessments
Windows CDOf LearningAssessments
Slide 33
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
In a Perfect World
Slide 34
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.Slide 35
In the Actual WorldThey don’t get it All!
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning
Post-Learning Assessments
Evidence Accumulated
Mentoring InstructorCollaborative & E-learning Work Place
Purpose:• Was learning environment
okay?• Knowledge transfer occurred?• Reduce Forgetting Curve• Certifications
Windows CDPrepare For Learning
Windows CDIn/As LearningAssessments
Windows CDOf LearningAssessments
Slide 36
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Learning Curve
Slide 37
Ouch! The Forgetting Curve
Time
Know
ledg
e/Sk
ills
Lear
ning
Ex
perie
nce
Lea
rnin
g E
xper
ienc
e
Lea
rnin
g E
xper
ienc
e
Que
stion
s
Que
stion
s
Purpose:• Content repetition• Memory retrieval practice• Strengthens memory
recall
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
In the Actual WorldThey Forget It!
Slide 38
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Slow the Forgetting Curve
Slide 39
Time
Know
ledg
e/Sk
ills
Lear
ning
Ex
perie
nce
Lea
rnin
g E
xper
ienc
e
Lea
rnin
g E
xper
ienc
e
Que
stion
s
Que
stion
s
Que
stion
s
Que
stion
s
Purpose:• Memory retrieval practice• Strengthens memory
recall• Reduce Forgetting Curve
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Questions Thru the Learning Process
Slide 40
Time
Know
ledg
e/Sk
ills
Que
stion
s
Que
stion
s
Que
stion
s
Que
stion
s
Q
uesti
ons
Create Intrigue
Strengthen Memory
Reduce Forgetting
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
30 minutes of instruction and 30 minutes of questioning
30 minutes of instruction and 30 minutes of review and study time
Both will produce similar results
Which of the following combination of instructional activities will increase learning and reduce forgetting? Question borrowed from the work of Dr. Will Thalheimer
Slide 41
B
C
A
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Remember what is learned
Retrieval Practice
Develop misconceptions
Feedback
Forget what’s been learned
Repetitions, Feedback, Retrieval Practice, SpacingDon’t absorb everything
Repetitions
Attention wanders
Pre-questions; directs attention
Why Don’t They Get It?
Slide 42
Dr. Will Thalheimer research:
Learning Benefits from Questions Learning Benefit
Min Max
Pre-questions (to focus attention) 5% 40%
RepetitionsInitial
Subsequent30%15%
110%40%
Feedback 15% 50%
Retrieval Practice 30% 100%
Questions Spaced Over Time 5% 40%
100% 380%
Slide 43
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
30 minutes of instruction and 30 minutes of questioning
30 minutes of instruction and 30 minutes of review and study time
Both will produce similar results
Which of the following combination of instructional activities will increase learning and reduce forgetting? Question borrowed from the work of Dr. Will Thalheimer
Slide 44
B
C
A
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Tests and Exams That Measure
Slide 45
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Regulatory Governments Health and Safety Examples
Driving Tests Pilots Exams Doctor Certification
Non-Regulatory, certification Vendor Specific
Microsoft Oracle SAP
Vendor Neutral ECDL/ICDL
Tests and Exams That Measure
Slide 46
Regulatory Compliance Demonstrating compliance to government
Health/Safety/Environmental Financial Services
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Norm Referenced(pass/fail determined after test)
Criterion Referenced(pass/fail score determined before test)
Setting a Pass/Fail score?
Slide 47
250 people
Only 50 peoplecan pass
250 people
1st group ofpeople
65 pass
2nd group ofpeople45 pass
3rd group ofPeople
75 pass
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
How each type of assessment measures the participant…
Norm ReferencedParticipant is compared to others in a group?• Typically used for standardized college placement tests
like the SAT, ‘competition’ tests (e.g. advancement test for limited number of openings)
• Pass/Fail can only be determined after the assessment
Criterion ReferencedParticipant is compared to a benchmark?• Typically used for certification tests, licensing exams, end
of course exams, etc.• Typically Pass/Fail score is determined before the
assessment
Slide 48
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Norm referenced
Criterion referenced
Neither of A nor B
Both A and B
Are Regulatory Compliance tests Norm or Criterion referenced?
B
C
A
D
Slide 49
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How Do I Know It’s a Good Test
Slide 50
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Choices# of
Respondents
% Selecte
d
Difficulty(also p-value)
0 = Too difficult
1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A
B
C
D
Item Statistics- Choices
Slide 51
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Choices# of
Respondents
% Selecte
d
Difficulty(also p-value)
0 = Too difficult
1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21
B 52
C 323
D 112
Item Statistics- # of respondents
Slide 52
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Choices# of
Respondents
% Selecte
d
Difficulty(also p-value)
0 = Too difficult
1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4%
B 52 10%
C 323 64%
D 112 22%
Item Statistics- % of respondents
Slide 53
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Choices# of
Respondents
% Selecte
d
Difficulty(also p-value)
0 = Too difficult
1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4%
B 52 10%
C 323 64% .64
D 112 22%
Item Statistics- Difficulty (also P-value)
Slide 54
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Choices# of
Respondents
% Selecte
d
Difficulty(also p-value)
0 = Too difficult
1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4% 36%
B 52 10% 47%
C 323 64% .64 66%
D 112 22% 52%
Item Statistics- Participant Mean
Slide 55
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Choices# of
Respondents
% Selecte
d
Difficulty(also p-value)
0 = Too difficult
1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4% 36%
B 52 10% 47%
C 323 64% .64 66% 0.49
D 112 22% 52%
Item Statistics- Outcome Discrimination
Slide 56
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Choices# of
Respondents
% Selecte
d
Difficulty(also p-value)
0 = Too difficult
1 = Too easy
Participant Mean
OutcomeDiscrimin
-ation
Outcome Correlatio
n
A 21 4% 36% -0.32
B 52 10% 47% -0.22
C 323 64% .64 66% 0.49 0.24
D 112 22% 52% -0.15
Item Statistics- Outcome Correlation
Slide 57
Cronbach’s Alpha
Internal consistency test reliability: A measure of how well the assessment questions relate to one another to measure the same topic
Crucial information for legal defensibility – is the test reliably measuring what is was designed to measure?
Speaks to the quality of assessment questions: Are they of high statistical quality?
)(1 2
22
T
pT
SSS
kk
Slide 58
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Skew Mode Kurtosis Mean (arithmetic) Median Mode Standard Deviation Variance Standard Error of measurement Standard Error of the Mean
Other Measures
Slide 59
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Informed Judgment Poll SMEs to determine criteria for passing (no. of correct
answers, no. of attempts) Not recommended for high-stakes assessments
Borderline Group Method Candidates are interviewed prior to an exam and borderline cases are
identified After the exam, the scores of borderline candidates can be used to determine
the cut score Contrasting Group
Compare SME to others Angoff Method
Poll SMEs Consider marginal candidates and the probability of getting specific questions
right (0-100%) Make a mathematical calculation to determine cut score
Setting Cut Scores for Criterion Reference Tests
Slide 60
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
SME A SME B SME C Total
Q1 75% 75% 75% 75.00%
Q2 70% 80% 80% 76.67%
Q3 65% 75% 70% 70.00%
Q4 60% 85% 90% 78.33%
Q5 80% 80% 85% 81.67%
Q6 80% 80% 80% 80.00%
Q7 75% 80% 75% 76.67%
Q8 65% 90% 65% 73.33%
Q….. 75% 80% 75% 76.67%
Q50 65% 85% 65% 71.67%
Totals 71% 81% 76% 76%
Angoff MethodWhat is the % chance that a borderline candidate will get question right?
Slide 61
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Emerging Trends and Practices
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Mobile Delivery of Assessments
Observational assessments
Mobile exam rooms
Deliver to the device in your
hand
Slide 63
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
QR = Quick Response Type of barcode Allows more data than
traditional barcode Used in marketing, advertising,
entertainment Increasing use in
e-learning/m-learning
What are QR Codes?
Slide 64
Can encode 4000+ characters of data!
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
QR Codes are read by Apps
Slide 65
www.url.com/xxx
www.url.com/xx
x
iPhone•Redlaser•Neoreader• ScanLife
Android•Google Goggles•QR Droid• Scanlife
Windows Mobile•QR Reader•BeeTag•Quickmark
Blackberry• ScanLife•BeeTag•QR Code
Scanner
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Books PowerPoint Signage
Enables quick launching of quizzes and course evaluations
Slide 66
Benefits• Ease of access for increased response rates• Demographic data easily captured• Better data accuracy• Easier analysis of survey results by demographic
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Embed in a Wiki
Slide 67
Embedded in a Wiki
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Embedding in Blogs, Portals, Wiki’s
For Learners For Instructors
Focus the learners’ attention Practice memory recall Identify what was learned Provide feedback to assist the
learner to: Correct misconceptions Instill confidence Point to relevant material
Easily use assessments throughout the learning process
Provide seamless user experience Centralised management of
assessment content and results
Slide 68
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
QuestionmarkAssessments Management System
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Assessment Management System
Slide 70
Software-as-a-Service
For OnPremise Deployment
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Criterion-Referenced Test Development Sharon A. Shrock & William C. Coscarelli ISBN: 1-890289-09-4
Work-Learning Research white papers Will Thalheimer www.work-learning.com
Tests That Work Odin Westgaard ISBN 0-7879-4596-X
Performance-Based Certification: How to design a valid, defensible, cost-effective program Judith Hale, Ph.D., CPT ISBN: 0787946400
Recommended Books
Slide 71
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Assessments through the Learning Process
Security of Questionmark’s OnDemand Service
Alignment, Impact and Measurement with the A-model
Embedded Assessments: Building Knowledge Checks, Surveys and Other Assessments into Learning Materials
How Questionmark Aids ISO 10667 Compliance
Recommended White Papers
Slide 72
https://www.questionmark.com/whitepapers
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Keep in Touch!
Slide 73
blog.questionmark.com
www.questionmark.com/newsletter
twitter.com/questionmark
linkedin.com Look for Questionmark under “Groups” and join
youtube.com/questionmarkvideo
flickr.com/questionmarkpictures
facebook.com Become a “fan” for news, updates and more
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are
acknowledged.
Session Evaluation
Slide 74
https://www.questionmark.com/go/709015
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What questions do you have?Eric Shepherd. Questionmark, CEO
[email protected] www.questionmark.com