Www.ioe.ac.uk Research perspectives and formative assessment ASME Conference: Researching Medical...
-
Upload
naomi-wright -
Category
Documents
-
view
221 -
download
0
Transcript of Www.ioe.ac.uk Research perspectives and formative assessment ASME Conference: Researching Medical...
www.ioe.ac.uk
Research perspectives and formative assessment
ASME Conference: Researching Medical Education, November 2009: RIBA, London
Dylan Wiliam
OverviewThe nature of educational researchWhat should educational research try to do?How should it try to do it?
Formative assessmentDefinitions ImplementationsResearching formative assessment
Educational research“An elusive science” (Lagemann, 2000)A search for disciplinary foundations
Making social science matter (Flyvbjerg, 2001) Contrast between analytic rationality and value-rationalityPhysical science succeeds when it focuses on analytic rationalitySocial science
fails when it focuses on analytic rationality, but succeeds when it focuses on value-rationality
Research methods 101: causalityDoes X cause Y?
In the presence of X, Y happened (factual) Problem: post hoc ergo propter hoc
Desired inference: If X had not happened, Y would not have happened (counterfactual) Problem: X did happen
So we need to create a parallel world where X did not happen Same group different time (baseline measurement) Need to assume stability over time Different group same time (control group) Need to assume groups are equivalent Randomized contolled trial
Plausible rival hypothesesExample: Smoking cigarettes causes lung cancer
Randomized controlled trial not possible
Have to rely on other methods
Logic of inference-making
Establish the warrant for chosen inferences
Establish that plausible rival interpretations are less warranted
KnowledgeNot justified-true-belief
Discriminability (Goldman, 1976)
Elimination of plausible rival hypotheses
Building knowledge involves:marshalling evidence to support the desired inferenceeliminating plausible rival interpretations
‘Plausible’ determined by reference to a theory, a community of practice, or a dominant discourse
Inquiry systems (Churchman, 1971)
System Evidence
Leibnizian Rationality
Lockean Observation
Kantian Representation
Hegelian Dialectic
Singerian Values, ethics and practical consequences
The Lockean inquirer displays the ‘fundamental’ data that all experts agree are accurate and relevant, and then builds a consistent story out of these. The Kantian inquirer displays the same story from different points of view, emphasising thereby that what is put into the story by the internal mode of representation is not given from the outside. But the Hegelian inquirer, using the same data, tells two stories, one supporting the most prominent policy on one side, the other supporting the most promising story on the other side (Churchman, 1971 p. 177).
Inquiry systems
The ‘is taken to be’ is a self-imposed imperative of the community. Taken in the context of the whole Singerian theory of inquiry and progress, the imperative has the status of an ethical judgment. That is, the community judges that to accept its instruction is to bring about a suitable tactic or strategy [...]. The acceptance may lead to social actions outside of inquiry, or to new kinds of inquiry, or whatever. Part of the community’s judgement is concerned with the appropriateness of these actions from an ethical point of view. Hence the linguistic puzzle which bothered some empiricists—how the inquiring system can pass linguistically from “is” statements to “ought” statements— is no puzzle at all in the Singerian inquirer: the inquiring system speaks exclusively in the “ought,” the “is” being only a convenient façon de parler when one wants to block out the uncertainty in the discourse. (Churchman, 1971: 202).
Singerian inquiry systems
Educational research…can be characterised as a never-ending process of assembling evidence that:particular inferences are warranted on the basis of the available evidence;such inferences are more warranted than plausible rival inferences; the consequences of such inferences are ethically defensible.
The basis for warrants, the other plausible interpretations, and the ethical bases for defending the consequences, are themselves constantly open to scrutiny and question.
Effective learning environmentsA prevalent, mistaken, viewTeachers create learningThe teacher’s job is to do the learning for the learner
A not so prevalent, not quite so mistaken, but equally dangerous viewOnly learners can create learningThe teacher’s job is to “facilitate” learning
A difficult to negotiate, middle pathTeaching as the engineering of effective learning environmentsKey features:
Create student engagement (pedagogies of engagement) Well-regulated (pedagogies of contingency) Develop habits of mind (pedagogies of formation)
Formative assessment: a definition“An assessment functions formatively to the extent that evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been taken in the absence of that evidence.
Formative assessment therefore involves the creation of, and capitalization upon, moments of contingency (short, medium and long cycle) in instruction with a view to regulating learning (proactive, interactive, and retroactive).” (Wiliam, 2009)
The formative assessment hi-jack…Long-cycleSpan: across units, termsLength: four weeks to one year Impact: Student monitoring; curriculum alignmentMedium-cycleSpan: within and between teaching unitsLength: one to four weeks Impact: Improved, student-involved, assessment; teacher cognition about
learningShort-cycleSpan: within and between lessonsLength:
day-by-day: 24 to 48 hours minute-by-minute: 5 seconds to 2 hours
Impact: classroom practice; student engagement
Unpacking assessment for learningKey processesEstablishing where the learners are in their learningEstablishing where they are goingWorking out how to get there
ParticipantsTeachersPeersLearners
Five “key strategies”…Clarifying, understanding, and sharing learning intentionscurriculum philosophy
Engineering effective classroom discussions, tasks and activities that elicit evidence of learningclassroom discourse, interactive whole-class teaching
Providing feedback that moves learners forward feedback
Activating students as learning resources for one another collaborative learning, reciprocal teaching, peer-assessment
Activating students as owners of their own learningmetacognition, motivation, interest, attribution, self-assessment
(Wiliam & Thompson, 2007)
A model for professional changeContentEvidence Ideas
ProcessChoiceFlexibilitySmall stepsAccountabilitySupport
KMO Formative Assessment Project24 teachers, each developing their practice in individual ways
Different outcome variables
No possibility of standardized controls
“Polyexperiment” with “local design”
Synthesis by standardized effect size
5
05
6
0 64 6 8 8 80 3 4 5 6 6 7 95 9
3
4
1.51.41.31.21.11.00.90.80.70.60.50.40.30.20.10.0
-0.1-0.2-0.3-0.4 5 9
Jack-knife estimate of mean effect size: 0.32; 95% C.I. [0.16, 0.48)
Effect size by comparison typeI Parallel set taught by same
teacher in same yearS Similar set taught by same
teacher in previous yearP Parallel set taught by different
teacher in same yearL Similar set taught by different
teacher in previous yearD Non-parallel set taught by
different teacher in same yearN National norms
SummaryEducational research is a never-completed process of assembling evidence that:particular inferences are warranted on the basis of the available evidence;such inferences are more warranted than plausible rival inferences; the consequences of such inferences are ethically defensible.
The basis for each of these is constantly open to scrutiny and question