Quality Assurance and the question of Validity · ensure quality control. Validity ... The elixir...
Transcript of Quality Assurance and the question of Validity · ensure quality control. Validity ... The elixir...
The first school merged Rudolf Steiner’s philosophical, spiritual concepts with Emil Molt’seconomic reality
Two things you need to know about validity
“If one were to select a sample of psychometricians from each of the last five to ten decades and gather them together in, say, a bar, it is quite likely that all would drink a toast to validity as the paramount concept in the field of testing.”
Fast, E. and Hebbler, S. with ASR-CAS Joint Study Group on Validity in Accountability Systems. (2004). A Framework for Examining Validity in State Accountability Systems. Washington, DC: Council of Chief State School Officers.
Two things you need to know about validity
“If one were to select a sample of psychometricians from each of the last five to ten decades and gather them together in, say, a bar, it is quite likely that all would drink a toast to validity as the paramount concept in the field of testing. However, a mêlée would ensue if they were asked to define what validity is.”
Fast, E. and Hebbler, S. with ASR-CAS Joint Study Group on Validity in Accountability Systems. (2004). A Framework for Examining Validity in State Accountability Systems. Washington, DC: Council of Chief State School Officers.
But don’t worry about that because…
The general idea of validity is basically simple
• it’s whether we’re assessing the right thing, in the right way, to produce accurate and usefulassessment results
1. Validity(which is the water in your bucket)
2. Threats to validity(that risk piercing your bucket)
This session will focus upon
Illustration by Sal Murdocca
So this is what Ofqual and Crossfields Institute mean by validity• The validity of a particular qualification or certification is
• the degree to which it is possible to measure• whatever that qualification or certification needs to measure• by implementing its assessment procedure.
• The validity of a particular qualification or certification is determined by• ALL of the features and processes that are put in place to
ensure that results are as accurate as possible and as usefulas possible.
This version of validity is pretty broad
VALIDITY
Authenticity
Reliability Sufficiency
Currency
And other
stuff too!
VALIDITY
Comparability
Reliability Minimal Bias
Manageability
And other
stuff too!
Five critical steps in the lifecycle of any assessment
1. Measurement objectives (LOs) are clarified
2. Multiple performances (evidence) are elicited from each student (via tasks) to provide evidence of achievement or proficiency
3. All evidence is evaluated in terms of what it implies about student achievement or proficiency
4. The evaluations, for each student is combined into an overall result
5. The result is interpreted by those for whom it has been provided
A qualification specification
A set of tasks for each candidate
A set of evaluations for each candidate
An overall result for each candidate
An interpretation of the result for each candidate
O
U
T
P
U
T
S
Step 1: learning objectives are clarified
■ Manual Metal Arc Welding
□ Welding skills typically found in industry and associated underpinning knowledge to a level that will enable them to complete welded joints in simple welding positions.
■ Learning outcome 1
□ The learner will:
▪ 1. produce beads on a plate in the PA flat position
■ Assessment criteria 1
□ The learner can:
▪ 1.1 use manual metal arc welding techniques safely to produce beads on a plate in simple welding positions
▪ 1.2 check joints are aligned and welds are sound and of uniform appearance
▪ 1.3 identify defects in the weld using visual checks
■ Etc.
Step 2: Multiple performances are elicited from each student (via assessment tasks) to provide evidence of proficiency
Credit: TechShop
Multiple welds Answers to multiple questions
Credit: Alberto G.
Step 3: Each task completed by the student is evaluated in terms of what it implies about their proficiency
Good products? Safe processes?
Correct answers?
Credit: garycycles7 Credit: genebrooks
Credit: Oliver Tacke
Step 4: The IQA moderates a sample from across all assessment tasks, assessors and students
• A range of evidence from assessment tasks is compared to ensure that assessment decisions are valid
Step 5: The measurement result is interpreted by those for whom it has been provided
Credit: Alan Cleaver
Level 1 Award in
Introductory
Manual Metal Arc
(MMA) Welding
… that’s a good
foundation for our
apprenticeship
A validity argument, expressed fairly informally
1. IF the learning objective is properly specified, &
2. IF evidence of achievement for that outcome is properly elicited, &
3. IF the elicited evidence is properly evaluated, &
4. IF the assessment is properly moderated, &
5. IF the results are properly interpreted,
THEN measurement interpretations will be both accurate and useful.
■ Manual Metal Arc Welding
□ Welding skills typically found in industry and associated underpinning knowledge to a level that will enable them to complete welded joints in simple welding positions.
■ Learning outcome 1
□ The learner will:
▪ 1. produce beads on plate in the PA flat position
■ Assessment criteria 1
□ The learner can:
▪ 1.1 use manual metal arc welding techniques safely to produce beads on a plate in simple welding positions
▪ 1.2 check joints are aligned and welds are sound and of uniform appearance
▪ 1.3 identify defects in the weld using visual checks
■ Etc.
Step 1: We need to specify the proficiency that needs to be measured
• But how faithfully does our proficiency specification represent the ‘target proficiency’ (the skillset that reallyneeds to be acquired)?
Is this LO up-to-date?
Have we omitted any
critical LOs?
Two examples of many
potential validity threats
Step 2: We need to generate and capture evidence
• But how faithfully does the evidence in each student’s portfolio represent their ‘true proficiency’?
Credit: TechShop
Credit: Alberto G.
Do students
interpret our questions as
we intend them to be
interpreted?
Is our simulated task
sufficiently authentic to test
real-world proficiency
(as opposed to
test-world proficiency)?
Two examples of many
potential validity threats
Step 3: We need to assess the evidence
• But how faithfully does the assessment feedback in each student’s portfolio represent the ‘true quality’ of their task performances?
Credit: garycycles7
Credit: Oliver Tacke
How susceptible are
assessor judgements to
proficiency-irrelevant
factors?
Does the m/c scoring ‘key’
identify the correct answer
for all questions?
Two examples of many
potential validity threats
Credit: genebrooks
Step 4: We need to moderate (IQA) the assessments
• How faithfully does each student’s overall result represent the ‘true significance’ of their evaluation profile?
Is it justifiable to award a
pass (or higher) when
certain ACs have not
been achieved?
Is it justifiable to award
grades purely on the basis
of written components?
LO1
AC1: P, AC2: P, AC3: P, AC4: F
LO2
AC1: P, AC2: P, AC3: P
LO3
AC1: P, AC2: P, AC3: P, AC4: P
LO4 and LO5
71/80 = DISTINCTION
OVERALL RESULT
DISTINCTION
Two examples of many
potential validity threats
Step 5: They need to interpret our results
• But how faithfully does each interpretation of a student’s result represent the ‘true meaning’ of their result?
Is the qualification
title sufficiently clear?
Has sufficient information
been provided to users on
how to interpret grades?
Credit: Alan Cleaver
Two examples of many
potential validity threats
And across all five steps Did you establish the credentials (expertise/integrity)
teachers, assessors and IQAs before appointing them?
Have those staff been given
the right training, guidance and supervision?
Have safeguards been put in place to prevent
human error and deliberate malpractice?
Have you satisfactorily ensured the
accuracy and security of the data
that you record and store?
And so on…
CLARIFICATION
STEP
The proficiency
specification
ELICITATION
STEP
A set of task
performances for
each candidate
EVALUATION
STEP
A set of
evaluations for
each candidate
COMBINATION
STEP
An overall result
for each candidate
INTERPRETATION
STEP
An interpretation
of the result for
each candidate
TARGET PROFICIENCY
Candidate
Assessor
Aggregator
Qual. User
Qual. Designer
Four examples of many
potential validity threats
CLARIFICATION
STEP
The proficiency
specification
ELICITATION
STEP
A set of task
performances for
each candidate
EVALUATION
STEP
A set of
evaluations for
each candidate
COMBINATION
STEP
An overall result
for each candidate
INTERPRETATION
STEP
An interpretation
of the result for
each candidate
TARGET PROFICIENCY
Candidate
Assessor
Aggregator
Qual. User
Qual. Designer
You probably are going
to lose a little trickle of
validity here…
CLARIFICATION
STEP
The proficiency
specification
ELICITATION
STEP
A set of task
performances for
each candidate
EVALUATION
STEP
A set of
evaluations for
each candidate
COMBINATION
STEP
An overall result
for each candidate
INTERPRETATION
STEP
An interpretation
of the result for
each candidate
TARGET PROFICIENCY
Candidate
Assessor
Aggregator
Qual. User
Qual. Designer … and
here…
CLARIFICATION
STEP
The proficiency
specification
ELICITATION
STEP
A set of task
performances for
each candidate
EVALUATION
STEP
A set of
evaluations for
each candidate
COMBINATION
STEP
An overall result
for each candidate
INTERPRETATION
STEP
An interpretation
of the result for
each candidate
TARGET PROFICIENCY
Candidate
Assessor
Aggregator
Qual. User
Qual. Designer
… and
so on!
CLARIFICATION
STEP
The proficiency
specification
ELICITATION
STEP
A set of task
performances for
each candidate
EVALUATION
STEP
A set of
evaluations for
each candidate
COMBINATION
STEP
An overall result
for each candidate
INTERPRETATION
STEP
An interpretation
of the result for
each candidate
TARGET PROFICIENCY
Candidate
Assessor
Aggregator
Qual. User
Qual. Designer
But you can’t afford to
lose a big load of
validity here…
CLARIFICATION
STEP
The proficiency
specification
ELICITATION
STEP
A set of task
performances for
each candidate
EVALUATION
STEP
A set of
evaluations for
each candidate
COMBINATION
STEP
An overall result
for each candidate
INTERPRETATION
STEP
An interpretation
of the result for
each candidate
TARGET PROFICIENCY
Candidate
Assessor
Aggregator
Qual. User
Qual. Designer
… or
here!
CLARIFICATION
STEP
The proficiency
specification
ELICITATION
STEP
A set of task
performances for
each candidate
EVALUATION
STEP
A set of
evaluations for
each candidate
COMBINATION
STEP
An overall result
for each candidate
INTERPRETATION
STEP
An interpretation
of the result for
each candidate
TARGET PROFICIENCY
Candidate
Assessor
Aggregator
Qual. User
Qual. Designer
And you can’t afford to
lose a constant stream
of validity throughout…