1 Designing and Evaluating Assessments for Introductory Statistics Minicourse #1 Beth Chance...
-
Upload
francine-peters -
Category
Documents
-
view
215 -
download
1
Transcript of 1 Designing and Evaluating Assessments for Introductory Statistics Minicourse #1 Beth Chance...
1
Designing and Evaluating Assessments for Introductory Statistics
Minicourse #1Beth Chance ([email protected])Bob delMasAllan RossmanNSF grant PI: Joan Garfield
2
Outline
Today: Overview of assessment Introductions Assessment goals in introductory statistics Principles of effective assessment Challenges and possibilities in statistics
Overview of ARTIST database Friday: Putting an assessment plan together
Alternative assessment methods Nitty Gritty details, individual plans
3
Overview
Assessment = on-going process of collecting and analyzing information relative to some objective or goal Reflective, diagnostic, flexible, informal
Evaluation = interpretation of evidence, judgment, comparison between intended and actual, use information to make improvements
4
Dimensions of Assessment
Evaluation of program Evaluate curricula, allocate resources
Monitoring instructional decisions Judge teaching effectiveness
Evaluating students Give grades, monitor progress
Promoting student progress Diagnose student needs
5
Types of Assessment
Formative Assessment In-process monitoring of on-going efforts in
attempt to make rapid adjustments Summative Assessment
Record impact and overall achievement, compare outcomes to goals, decide next steps
Example: teaching Example: learning
6
Bloom’s Taxonomy
Knowledge Comprehension Application Analysis Synthesis
Interrelationships Evaluation
7
An Assessment Cycle
1. Set goals
2. Select methods
3. Gather evidence
4. Draw inference
5. Take action
6. Re-examine goals and methods
Example: Introductory course
Example: Lesson on sampling distributions
8
Reflect on Goals
What do you value? Instructor and student point of view Content, abilities, values
At what point in the course should they develop the knowledge and skills?
Translate learning outcomes/objectives What should students know and be able to do by
the end of the course Must be measurable!
9
Some of My Course Goals
Understand basic terms (literacy) Understand the statistical process
Not just the individual pieces, be able to apply Be able to reason and think statistically
role of context, effect of sample size, caution when using procedures, belief in randomness, association vs. causation
Communication and collaboration skills Computer literacy Interest level in statistics
10
Possible Future Goals
Process (not just product) of collaboration Learn how to learn Appreciate learning for its own sake Develop the necessary skills to understand
both what they have learned and what they do not understand
11
Assess what you value
Students value what they
are assessed on
12
Example
Given the numbers 5, 9, 11, 14, 17, 29
(a) Find the mean
(b) Find the median
(c) Find the mode
(d) Calculate a 95% confidence interval for
13
“Traditional” Assessment
Good for assessing: Isolated computational skills, (short-term) memory
retrieval Use and tracking of common misconceptions How many right answers?
Provides us with: Consistent and timely scoring Predictor of future performance
14
“Traditional” Assessment
Less effective at assessing: Can they explain their knowledge? Can they apply their knowledge? What are the limitations in their knowledge? Can they make good decisions? Can they evaluate? Can they deal with messy data? Role of prior knowledge
15
Focus on what and how students learn, what students can now do Not on what faculty teach
16
Nine Principles (AAHE)
Start with educational values Multi-dimensional, integrated, over-time Clearly stated purposes Pay attention to outcomes and process On-going Student representation Important questions Support change Accountability
17
Select Methods
Need multiple, complimentary methods observable behavior adequate time
Need to extend students less predictable, less discrete
Needs to provide indicators for change Need prompt, informative feedback loop
On-going, linked series of activities over time Continuous improvement, self-assessment
Students must believe in its value
18
Focus on the most prevalent student misconceptions
19
Repeat the Cycle
Focus on the process of learning Feedback to both instructors and students Discuss results with students, motivate
responsibility for their own learning Consider other factors
Collaborate External evaluation
Continual refinement Consider unexpected outcomes
Don’t try to do it all at once!
20
Use the results of the assessment to improve student learning
21
Challenges in Statistics Education Doing statistics versus being an informed
consumer of statistics Statistics vs. mathematics
Role of context, messiness of solutions, computers handling the details of calculations, need to defend argument, evaluate based on quality of reasoning, methods, evidence used
Have become pretty comfortable with lecture/reproduction format Traditional assessment feels more objective
22
Challenges in Statistics Education Reduce focus on calculation Reveal intuition, statistical reasoning Require meaningful context
Purpose, statistical interest Meaningful reason to calculate Careful, detailed examination of data
Use of statistical language Meaningful tasks, similar to what will be
asked to do “in real life”
23
Some Techniques Multiple choice
with identification of false response with explanation or reasoning choices with judgment, critique (when is this appropriate)
“What if”, working backwards, “construct situation that”
Objective-format questions e.g., comparative judgment of strength of
relationship e.g., matching boxplot with normal prob plots
Missing pieces of output, background
24
Some Techniques
Combine with alternative assessment methods, e.g., projects: see entire process, messiness of
real data collection and analysis e.g., case studies: focus on real data, real
questions, students doing and communicating about statistics
Self-assessment, peer-evaluation
25
What can we learn?
Sampling Distribution questionsWhich graph best representsa distribution of sample meansfor 500 samples of size 4? A B C D E
26
What can we learn?
Asking them to write about their understanding of sampling distributions Now place more emphasis in my teaching on
labeling horizontal and vertical axes, considering the observational unit, distinguishing between symmetric and even, spending much more time of the concept of variability
Knowing better questions to ask to assess their understanding of the process
27
ARTIST Database
First…
28
HW Assignment
Assessment Framework WHAT: concept, applications, skills, attitudes,
beliefs PURPOSE: why, how used WHO: student, peers, teacher METHOD ACTION/FEEDBACK: and so?
29
HW Assignment
Suggest a learning goal, a method, and an action Be ready to discuss with peers, then class, on
Friday
Sample Final Exam (p. 17) Skills/knowledge being assessed Conceptual/interpretative vs.
mechanical/computational
30
Day 2
31
Overview
Quick leftovers on ARTIST database? Critiquing sample final exam Implementation issues (exam nitty gritty) Additional assessment methods Holistic scoring/Developing rubrics Your goal/method/action Developing assessment plan Wrap-up/Evaluations
32
Sample Final Exam
In-class component (135 minutes)
What skills/knowledge are being assessed? Conceptual/interpretative vs.
Computational/mechanical?
33
Sample Exam Question 1
Stemplot Shape of distribution Appropriateness of numerical summaries
C/I: 5, C/M: 3
34
Sample Exam Question 2
Bias Precision Sample size
C/I: 8, C/M: 0 No calculations No recitation of definitions
35
Sample Exam Question 3
Normal curve Normal calculations
C/I: 4, C/M: 3
36
Sample Exam Question 4
Sampling distribution, CLT Sample size Empirical rule
C/I: 4, C/M: 0 Students would have had practice Explanation more important than selection
37
Sample Exam Question 5
Confidence interval Significance test, p-value Practical vs. statistical significance
C/I: 7, C/M: 2 No calculations needed Need to understand interval vs. test
38
Sample Exam Question 6
Experimentation Randomization Random number table
C/I: 4, C/M: 4 Tests data collection issue without requiring
data collection
39
Sample Exam Question 7
Experimental design Variables Confounding
C/I: 13, C/M: 0 Another question on data collection issues
40
Sample Exam Question 8
Two-way table Conditional proportions Chi-square statistic, test Causation
C/I: 5, C/M: 9 Does not require calculations to conduct test
41
Sample Exam Question 9
Boxplots ANOVA table Technical assumptions
C/I: 7, C/M: 3 Even calculations require understanding
table relationships
42
Sample Exam Question 10
Scatterplot, association Regression, slope, inference Residual, influence Prediction, extrapolation
C/I: 15, C/M: 0 Remarkable in regression question!
43
Sample Exam Question 11
Confidence interval, significance test Duality
C/I: 9, C/M: 2 Again no calculations required
44
Sample Exam
C/I: 79, C/M: 28 (74% conceptual) Coverage
experimental design, randomization bias, precision, confounding stemplot, boxplots, scatterplots, association normal curve, sampling distributions confidence intervals, significance tests chi-square, ANOVA, regression
45
Nitty Gritty
External aids… Process of constructing exam… Timing issues… Student preparation/debriefing…
46
Beyond Exams
Combine with additional assessment methods, e.g., projects: see entire process, messiness of
real data collection and analysis e.g., case studies: focus on real data, real
questions, students doing and communicating about statistics
generation instead of only validation…
47
Beyond Exams (p. 8)…
Written homework assignments/lab assignments
Minute papers Expository writings Portfolios/journals Student projects Paired quizzes/group exams Concept Maps
48
Student Projects
Best way to demonstrate to students the practice of statistics
Experience the fine points of research Experience the “messiness” of data Statistician’s role as team member From beginning to end
Formulation and Explanation Constant Reference
49
Student Projects
Choice of Topic Ownership
Choice of Group In-class activities first
Periodic Progress Reports Peer Review Guidance/Interference
Early in process Presentation (me, alum, fellow student) Full Lab Reports statweb.calpoly.edu/chance/stat217/projects.html
50
Project Issues
Assigning Grades, individual accountability Insignificant/Negative Results Reward the Effort Iterative
Encourage/expect revision Long vs. Short projects
Coverage of statistical tools Workload
51
Holistic Scoring
Not analytic - each part = X points Problem is graded as a whole Calculations are one of many parts Strengths in one section can balance
weaknesses in another
52
Holistic Scoring
Did the student demonstrate knowledge of the statistical concept involved?
Did the student communicate a clear explanation of what was done in the analysis and why?
Did the student express a clear statement of the conclusions drawn?
53
Holistic Scoring
May lose points if don’t clearly explain why method was chosen assumptions of method line of reasoning final conclusion in context
54
Developing Rubrics
Scoring guide/plan Consistency, inter-rater reliability
Focus on goal/purpose of the question What information do you hope to learn based on
the student’s performance Identify valueable student behavior/correct
characteristics List those characteristics in observable terms
(Best/Good/Fair/Poor)
55
Developing Rubrics
Multiple procedures, variety of techniques Can you see the students’ thought
processes, knowledge of assumptions
56
Developing Assessment Plan
Match (most important) instructional goals Start with learning outcomes, own questions
Multiple and varied indicators Inter-related, complementary
Well-defined, well-integrated throughout course Detailed expectations, part of learning process
Goals understood by students Promote self-reflection, responsibility, trust
57
Developing Assessment Plan
Timely, consistent feedback indicators for change, feedback loop,
reinforcement Individual and group accountability Openness to other (justified) interpretations,
reward thoughtfulness, creativity Not all at once, Not too much Collaborate Continual reflection, refinement Assess what you value
58
Cautions!
Consider time requirements for students and instructor! Easier to solve than to explain With experience, become more efficient
Provide sufficient guidance Provide students with familiarity and clear
understanding of your expectations May not be used to being required to think! Less comfortable writing in complete sentences
59
Wrap-Up