Learning, assessment and technology: in that order Keynote address to AMEE conference September...
-
Upload
lane-haddaway -
Category
Documents
-
view
212 -
download
0
Transcript of Learning, assessment and technology: in that order Keynote address to AMEE conference September...
![Page 1: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/1.jpg)
Learning, assessment and technology: in that order
Keynote address to AMEE conferenceSeptember 2009; Malaga, Spain
Dylan Wiliam
Institute of Education, University of London
www.dylanwiliam.net
![Page 2: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/2.jpg)
Overview of presentationSome theoretical precepts About learning About teaching
• Pedagogies of engagement• Pedagogies of contingency• Pedagogies of formation
The role of technology Supporting, rather than replacing, teachers Classroom aggregation technologies
Some thoughts about supporting teachers in changing practice
![Page 3: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/3.jpg)
How do we improve learning?
![Page 4: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/4.jpg)
What gets learnt?
(Denvir & Brown, 1986)
![Page 5: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/5.jpg)
Key insights from C20th psychologyWhat gets learned as a result of a particular sequence of
instructional activities is impossible to predict, but
Student errors are not random
Conclusions Assessment is the bridge between teaching and learning Teaching is interesting because learners are so different, but only
possible because they are so similar
![Page 6: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/6.jpg)
Learning power environmentsKey concept: Teachers do not create learning Learners create learning
Teaching as the engineering of learning environments
Key features: Create student engagement (pedagogies of engagement) Well-regulated (pedagogies of contingency) Develop habits of mind (pedagogies of formation)
![Page 7: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/7.jpg)
Why pedagogies of engagement?Intelligence is partly inherited So what?
Intelligence is partly environmental Environment creates intelligence Intelligence creates environment
Learning environments High cognitive demand Inclusive Obligatory
![Page 8: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/8.jpg)
264 low and high ability grade 6 students in 12 classes in 4 schools; analysis of 132 students at top and bottom of each class
Same teaching, same aims, same teachers, same class work Three kinds of feedback: scores, comments, scores+comments
[Butler(1988) Br. J. Educ. Psychol., 58 1-14]
Engagement and feedback
Achievement Attitude
Scores no gain High scorers: positive
Low scorers: negative
Comments 30% gain High scorers : positive
Low scorers : positive
![Page 9: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/9.jpg)
[Butler(1988) Br. J. Educ. Psychol., 58 1-14]
Responses
What do you think happened for the students given both scores and comments?
A. Gain: 30%; Attitude: all positiveB. Gain: 30%; Attitude: high scorers positive, low scorers negativeC. Gain: 0%; Attitude: all positiveD. Gain: 0%; Attitude: high scorers positive, low scorers negativeE. Something else
Achievement Attitude
Scores no gain High scorers : positive
Low scorers: negative
Comments 30% gain High scorers : positive
Low scorers : positive
![Page 10: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/10.jpg)
Kinds of feedback (Nyquist, 2003)Weaker feedback only Knowledge of results (KoR)
Feedback only KoR + clear goals or knowledge of correct results (KCR)
Weak formative assessment KCR+ explanation (KCR+e)
Moderate formative assessment (KCR+e) + specific actions for gap reduction
Strong formative assessment (KCR+e) + activity
![Page 11: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/11.jpg)
Effect of formative assessment (HE)
N Effect size
Weaker feedback only 31 0.14
Feedback only 48 0.36
Weaker formative assessment 49 0.26
Moderate formative assessment 41 0.39
Strong formative assessment 16 0.56
(Nyquist, 2003; revised values)
![Page 12: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/12.jpg)
Effects of feedbackKluger & DeNisi (1996) review of 3000 research reports Excluding poorly designed studies left 131 reports, 607 effect
sizes, involving 12652 individuals
On average, feedback increases achievement, but Effect sizes highly variable 38% (50 out of 131) of effect sizes were negative
![Page 13: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/13.jpg)
Engagement in learningAttribution (Dweck, 2000) Personalization (internal vs. external) Permanence (stable vs. unstable) Good learners attribute failure and success to internal, unstable
causes. (It’s down to you, and you can do something about it.)
Views of ‘ability’ Fixed (IQ) Incremental (untapped potential) Essential that teachers inculcate in students a view that ‘ability’ is
incremental rather than fixed (by working, you’re getting smarter).
![Page 14: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/14.jpg)
Dual-pathway theory (Boekaerts, 2006)Long-term learning goals are translated into short-term
learning intentions
Dynamic comparisons of task and situational demands with personal resources
Resulting activation of energy along one of two pathways: Well-being Growth
![Page 15: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/15.jpg)
Motivation: cause or effect?
competence
challenge
Flow
apathyboredom
relaxation
arousalanxiety
worry control
high
low
low high
(Csikszentmihalyi, 1990)
![Page 16: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/16.jpg)
Why pedagogies of contingency?
Fuchs & Fuchs (1986)
Natriello (1987)
Crooks (1988)
Banger-Drowns, et al. (1991)
Kluger & DeNisi (1996)
Black & Wiliam (1998)
Nyquist (2003)
Dempster (1991, 1992)
Elshout-Mohr (1994)
Brookhart (2004)
Allal & Lopez (2005)
Köller (2005)
Brookhart (2007)
Wiliam (2007)
Hattie & Timperley (2007)
Shute (2008)
![Page 17: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/17.jpg)
Hinge-point questionAn experimental study of problem-based learning in undergraduate medical education reports that a result was significant (p<0.05). This means that:A. The experimental group out-performed the control group by at least
5%
B. There is only a 5% chance that the experimental group did not out-perform the control group
C. There is a 5% chance that there is no difference between the experimental group and the control group
D. There is only a 5% chance that the observed result would have happened if the experimental and control groups had the same achievement
![Page 18: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/18.jpg)
Other supports for contingencyAll-student response systems ABCD cards “Exit-pass” questions
![Page 19: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/19.jpg)
Exit-pass questionSummarize the key principles of the following schools of psychology on the appropriate coloured card Associationism (blue) Information processing (orange) Constructivism (red) Situated approaches (green)
![Page 20: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/20.jpg)
Pedagogies of formationInstilling disciplinary habits of mind History Philosophy Statistics
Instilling critical perspectives Values
![Page 21: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/21.jpg)
Three generations of pedagogyFirst generation Traditional pedagogy Negligible contingency
Second generation All student response systems Contingency dependent entirely on teacher skill
Third generation Automated aggregation technologies Contingency supported by technology
![Page 22: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/22.jpg)
Evidence-centred designQuality in assessment is essentially a matter of validity Validity is a property of inferences, not of instruments Assessments should be designed “backwards” from the intended
inferences
Four-process architecture for assessment design Task selection Task presentation Evidence identification Evidence accumulation
Almond, Steinberg and Mislevy (2002)
![Page 23: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/23.jpg)
Task selection/Task presentation
![Page 24: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/24.jpg)
Hinge-point questionWhich of the following is the most important difference between the theories of Piaget and Vygotsky?
A. Piaget places greater importance on the role of conservation in cognitive development
B. Vygotsky places greater importance on the role of cultural artifacts in cognitive development.
C. Vygotsky did not believe in distinct stages of cognitive development.
D. Piaget was a social constructivist while Vygotsky placed greater emphasis on cultural-historical activity theory
![Page 25: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/25.jpg)
Evidence identification
![Page 26: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/26.jpg)
Evidence identificationSingle student response systems
All-student response systems Flash-cards/dry erase boards Classroom ‘clickers’ Traditional keyboards (wired/wireless) Anoto pens
![Page 27: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/27.jpg)
Anoto pen Wireless pen
Special coated paper
Pen ‘knows where it is’
![Page 28: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/28.jpg)
Palm with wireless keyboard
Text-based input
Limited task-presentation capability
Portable
![Page 29: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/29.jpg)
Classroom ‘clickers’ (and their progeny)
![Page 30: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/30.jpg)
Discourse®
www.ets.org/discourse
![Page 31: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/31.jpg)
Evidence identificationAutomated essay scoring (e-rater)
Paraphrase analysers (c-rater)
Graphical and equation analysers (m-rater)
![Page 32: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/32.jpg)
Evidence identification technologies
unstructuredstructured
evidence structure
teacher-mediated
automatedclickers
aggr
egat
ion
ABCDcards
dry-eraseboards
c-rater
Discourse®
latent semantic analysis
m-rater
e-rater
![Page 33: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/33.jpg)
Evidence accumulation
![Page 34: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/34.jpg)
Evidence accumulationUnidimensional student models Useful for summative purposes Almost useless for formative purposesMultidimensional student modelsEvidence-centred design Bayesian inference networks
• Proficiency model• Task model• Evidence model• Student model
![Page 35: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/35.jpg)
Four-process architecture for ECD
Mislevy, Almond and Lukas (2003)
Task selection
Task presentation
Evidence accumulation
Evidence identification
![Page 36: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/36.jpg)
Evidence utilizationWhole-class
Sub-groups Homogenous Heterogenous
Individualization
![Page 37: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/37.jpg)
37
Actually, the technology is the easy partWhat’s hard is changing practice
Telling teachers what to do doesn’t work
Context always intrudes…
![Page 38: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/38.jpg)
38
The ancient yogis used logs of wood, stones, and ropes to help them practise asanas effectively. Extending this principle, Yogacharya Iyengar invented props which allow asanas to be held easily, and for a longer duration without strain.
Yogacharya Iyengar in setubandha sarvangasana
This version of the posture requires considerable strength in the neck, shoulders, and back requiring years of practice to achieve it. It should not be attempted without supervision.
![Page 39: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/39.jpg)
39
![Page 40: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/40.jpg)
A model for teacher learningContent, then process
Content (what we want teachers to change) Evidence Ideas (strategies and techniques)Process (how to go about change) Choice Flexibility Small steps Accountability Support
![Page 41: Learning, assessment and technology: in that order Keynote address to AMEE conference September 2009; Malaga, Spain Dylan Wiliam Institute of Education,](https://reader035.fdocuments.us/reader035/viewer/2022070307/551a4aa0550346cb358b59d5/html5/thumbnails/41.jpg)
SummaryLearning has to be done by the learner, not for the learner
Teaching as engineering effective learning environments
Features of effective learning environments Pedagogies of engagement Pedagogies of contingency
• All-student response systems• Classroom aggregation technologies
Pedagogies of formation
Learning milieu focused on growth, rather than well-being
Teachers supported to improve practice continually