Inquiry into effective teaching with one hand tied behind your back
description
Transcript of Inquiry into effective teaching with one hand tied behind your back
Inquiry into effective teaching with Inquiry into effective teaching with one hand tied behind your backone hand tied behind your back
22 November 2006UBC Institute for the Scholarship of Teaching and Learning
Dan Bernstein, University of [email protected]
Overview of session• Invited to discuss my research methods• Among methodological issues in research on teaching, design and measurement are key• Look at examples of some of my methods• Embed them in larger framework as I go• Inform some practices by research literature•Recommend a different path from my own• Propose a big tent, even for research done without the best possible methods
Research Methods Issues
Design and Analysis Measurement
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
EXAMPLE
Experimental Procedures
• Commercial and home-made multi-media • Used video and live lecture with same
content as multi-media• Created text samples with same content• Assessment of levels of understanding with
mixed methods of responding• Individual instruction with unlimited time• Participants from introductory psychology
Lecture, Reading, and Computer
0
10
20
30
40
50
60
<= 69 70-79 80-89 90-100
LectureReadingsComputer
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
Analyze components
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
Analyze components
EXAMPLE
Mean Instructional Time by Instructional Mode
0
5
10
15
20
25
30
Computer Lecture Readings
Instructional time*
Time on task (in minutes)
0
10
20
30
40
50
60
70
Computer Lecture Reading
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
EXAMPLE
Upgraded the lecture component
• Live lecture to portion of introductory class• Topic is reactive measurement and
quantum mechanics• Matched to locally authored multi-media
and text -- also irrelevant text control• Motivation sustained by possibility of test• Assessment layered by type of
understanding and format of question
Distribution of total test performance resulting from four methods of
instruction
0%
10%
20%
30%
40%
50%
60%
<= 59
60-69
70-79
80-89
90-100
<= 5960-6970-7980-8990-100
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
Statistical comparison
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
Statistical comparison
EXAMPLE
Pushed envelope on assessment
• Looked for performance assessment• Topic was searching a jail cell• Verbal assessment is recognizing and
stating guidelines for complete search• Authentic assessment is use of guidelines
in searching a jail cell assembled for testing
• High end interactive program developed by NETV for Nebraska Sheriff’s Association
Authentic v. Verbal Assessment
010
2030
4050
6070
8090
InteractiveComputer
Lecture
VerbalAuthentic *
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
¿Ethical questions?¿Permission?
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Benchmarks of achievement
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Benchmarks of achievement
EXAMPLE
Criteria for oral interviewTopic Threshold
understanding
(5-9 pts)
Good understanding (10-15 pts)
Advanced understanding (16-20 pts)
Free will and determinism
Acknowledges that people can take position of free will and position of determinism; provides an example of someone’s beliefs that are influenced
Identifies some form of evidence that supports determinants of beliefs; identifies a form of evidence that supports free will in values; likely takes soft determinism view, that both are true
Provides strong evidence related to conventional criteria for free will or determinism; identifies and uses more than one criterion for either view; articulates criteria in reference to own beliefs or practices
Criteria for oral interviewTopic Threshold
understanding
(5-9 pts)
Good understanding (10-15 pts)
Advanced understanding (16-20 pts)
Scientific community
Recognizes the distinction between empirical claims and assertions of value; can give an example of a scientific analysis of a specific topic; thinks of psychology in broad ways of knowing
Can provide an example of scientific question and a non-scientific question; describes a procedure characteristic of an empirical analysis;
thinks of psychology as an empirical enterprise
Provides an original example of a scientific and non-scientific question; describes more than one analytic tool with understanding of its interpretation; recognizes that psychology involves value assertions as well as empirical questions
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Benchmarks of achievement
Movement ofachievement
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Benchmarks of achievement
EXAMPLE
Movement ofachievement
[Universal] Design for Success
• Presume students can learn
• Discount need to sort or differentiate
• Maximize overall course performance
Benjamin Bloom promoted mastery • Based on practice
and feedback• Divide course into
many smaller units• Take examinations
and get results• Require taking exam
again until high score• IFF 95% correct =>
study next unit
Fred Keller promoted mastery• IFF 95% correct =>
study next unit• Course grade is number
of units passed• No penalty for repeating
and learning• All who pass 12 units =>
grade of A• Do A work on 10 units
=> grade of C
Also taught conventional lecture
• Mastery Class
• 95% contingency
• No penalty for learning
• Immediate feedback
• No lecture required
• Lecture class
• Same exam questions
• Two attempts per test
• In class feedback
• No contingency
Total amount learned
• Nearly twice as many at the high end of learning• Virtually no one failed to learn• Maximized learning for many students
Showed in amount and accuracy
• Many more questions answered
• Took 12 15-item tests• Lecture was three tests
of 20 items each• Certify more learning• Overall percent correct
also higher
No magic -- students studied better
• They put in more time to their learning
• There was more work asked for by the course
• Note that they report doing the reading more
• Preparation for class is key issue (later also)
• Guideline in US -- 2 hours outside for every 1 hour in class
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
What do you think about the relative utility of these approaches?
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Cognitive Affective
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Cognitive Affective
¿EXAMPLES?
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Repeatknowledge
Abstract understanding
Contextualapplication
Affect
EXAMPLE
Added online questions on reading
Feedback until mastery of topics
Addition of EDU Out of Class
05
10152025303540
Percent Students
<=69 70-79 80-89 90-100
Levels of Achievement
New Questions Web-aided
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Repeatknowledge
Abstract understanding
Contextualapplication
Affect
YOUR EXAMPLES
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Repeatknowledge
Abstract understanding
Contextualapplication
Affect
EXAMPLE
Reasonably Conceptual Items• Describe the results of [Razran’s] second experiment on Describe the results of [Razran’s] second experiment on
semantically mediated generalization (numbers) done semantically mediated generalization (numbers) done with a 13-year old boy. Be sure to identify the with a 13-year old boy. Be sure to identify the respondent behavior, the training stimuli, and the test respondent behavior, the training stimuli, and the test stimuli. Why is this example semantically rather than stimuli. Why is this example semantically rather than physically mediated generalization?physically mediated generalization?
• What do chicken scratches, raccoon rubbings and pig What do chicken scratches, raccoon rubbings and pig rooting have in common? What stimuli have pre-existing rooting have in common? What stimuli have pre-existing relations these bits of behavior? What does the relations these bits of behavior? What does the “misbehavior of organisms” have to do with the generality “misbehavior of organisms” have to do with the generality of principles of learning? of principles of learning?
Transform into Problems• Old question:
– What were the reinforcing consequences in the Welsh, Bernstein & Luthans (restaurant) study? How were the consequences identified?
• Problem-based Assessment :– Suppose you were asked to implement a motivational
program in a business with 25 employees engaged in the following activities: production planning, inventory delivery, direct production, packaging, and marketing. Based on your understanding of the restaurant study, how would you improve the quality of the employees work by using access to activities as a motivator? Describe the costs and benefits of the program and make a recommendation about implementation.
• Old question:– The behavioral context into which punishment is
added must be considered in evaluating the likely effect of a given stimulus. What conditions are essential in this analysis?
• New question:– Suppose it is your task to decrease the frequency of
college students engaging in binge drinking or games involving high levels of intoxication. You have been asked to set up a punishment program to eliminate this program before someone is hurt. What context for binge drinking would you identify first? What punishing consequence would you use? What characteristics would you include in your punishing system to maximize effectiveness? Please give an example of an additional element you would need to include to make the plan maximally effective.
• Old question:– Describe the results of the experiment on semantically mediated
generalization done with a 13-year-old Soviet boy. Be sure to identify the respondent behavior, the training stimuli, and the test stimuli. Why is this example semantically rather than physically mediated generalization?
• New question:– Generate your own example of classical conditioning using the
neutral stimuli “idea” and “rock” along with the eliciting relation between a very loud noise and increased heart rate. Your example should include all of the following components: a description of a conditioning procedure that would produce differing reactions to the two stimuli, a description of the procedure that tests for the direct effects of conditioning, a description of a procedure that would test for physically mediated generalization, a description of a procedure that would test for semantically mediated generalization, and the likely results of the three test procedures.
Distribution on 1st Attempt
05
10152025303540
Percent Students
<= 69 70-79 80-89 90-100
Levels of Achievement
Old questions
Change of Questions
05
101520253035404550
Percent Students
<= 69 70-79 80-89 90-100
Levels of Achievement
Old questions New questions
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Repeatknowledge
Abstract understanding
Contextualapplication
Affect
GeneralinterestIntent to act
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Repeatknowledge
Abstract understanding
Contextualapplication
Affect
FISHBEIN
GeneralinterestIntent to act
Prediction of action through self reports
• Huge problem in social psychology
• Three decades of poor correlations
• Martin Fishbein saved the day– Specific action– Context– Time– Target
• Resulted in a more limited construct
• Gone was the global attitude
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Repeatknowledge
Abstract understanding
Contextualapplication
Affect
MISCHEL
GeneralinterestIntent to act
Global traits also challenged
• Personality tests based on general traits
• Three decades of data showed modest correlations, largely ~.30 (10% of variance)
• Walter Mischel offered less global theory
• Interaction between persons and situations
• Not useful in predicting what people do
• Replies have all been more specific, less global
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Bounded surveyof perception
Reflectionon learning
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Bounded surveyof perception
Reflectionon learning
Amount oflearning
Process of learning
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Bounded surveyof perception
Reflectionon learning
Amount oflearning
Process of learning
FISHBEIN NISBETT &WILSON
Introspection is the first problem• Academic psychology grew from this issue• More general problem of human judgment• Many complications lead to non optimal
performance• Richard Nisbett and Timothy Wilson
reviewed decades of work• In general people do notdo not accurately report
the why of consciousness, only the what• Alternatives from Simon, from Kahneman &
Tversky -- all stress practice and feedback• Open question is capacity given training
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Bounded surveyof perception
Reflectionon learning
Amount oflearning
Process of learning
FISHBEIN NISBETT &WILSON
NISBETT & WILSON
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
WHAT DO YOU THINK ABOUT THE UTILITY OF DIRECT AND INDIRECT MEASURES FOR RESEARCH ON TEACHING? YOUR EXPERIENCES?
Research Methods Issues
Design and Analysis Measurement
Random AssignmentExperiment
DescriptiveStudy
Treatment Control
THIS IS THE PATH I TOOK, STARTINGFROM EXPERIMENTAL PSYCHOLOGY
Research Methods Issues
Design and Analysis Measurement
Direct Indirect
Authentic Verbal surrogate
Repeatknowledge
Abstract understanding
Contextualapplication
Affect
I RECOMMEND MOST FACULTY MEMBERS START WITH RICH DESCRIPTION
What kind of research on their What kind of research on their teaching is appropriate for typical teaching is appropriate for typical
university faculty members?university faculty members?Is this properly called scholarship?Is this properly called scholarship?
Rigorous EducationalResearch
NarrativeDemonstration Of Learning
Inquiry intoPromotion ofLearning
Benchmark of Exceptional Achievement
All are Public
Rigorous EducationalResearch
NarrativeDemonstration Of Learning
Inquiry intoPromotion ofLearning
Benchmark of Exceptional Achievement
All are Public
Rigorous EducationalResearch
NarrativeDemonstration Of Learning
Inquiry intoPromotion ofLearning
Benchmark of Exceptional Achievement
Range of Purposes
DemonstrationOf Quality
Discover NewWays to Teach
Rigorous EducationalResearch
NarrativeDemonstration Of Learning
Inquiry intoPromotion ofLearning
Benchmark of Exceptional Achievement
Range ofMethods
Social Science Humanities
http://www.cte.ku.edu
Your Insights?
Ernest Boyer:
The work of the scholar remains incomplete
until it is understood and used by others.
Three functions of grading
• Certification of learning
• Motivation for learning
• Differentiation among learners
• Each has a legitimate purpose
• No one system does all well
Variability in conventional course
• Students learn at different rates• When course ends, fast learners get best results• Very good at identifying fast learners (differentiate)• Less good at motivating for more work
Variability in a mastery course
• Everyone learns until material is mastered• Reward is for work; subjective probability of success• Very good at certification of learning• Provides incentive for studying, no penalty if slow
When is mastery the right approach?
• Foundation courses -- want knowledge• Programs in which rate of learning is not a
criterion for success• Situations in which performance will not be
timed• Professions in which high skill is expected• Why tolerate ineffective teaching?• If we don’t care or think it can not be learned
by all
Scholarship Assessed (1997)• All forms of scholarship
include:
– Clear goals– Adequate preparation– Appropriate methods– Significant results– Reflective critique– Effective presentation
Glassick, Huber, & Maeroff