Part I: The 3 Ts of Online Assessment: Tools, Techniques, and (Saving) Time Curtis J. Bonk,...
-
Upload
kelton-hendrickson -
Category
Documents
-
view
216 -
download
3
Transcript of Part I: The 3 Ts of Online Assessment: Tools, Techniques, and (Saving) Time Curtis J. Bonk,...
Part I: The 3 T’s of Online Assessment: Part I: The 3 T’s of Online Assessment: Tools, Techniques, and (Saving) TimeTools, Techniques, and (Saving) Time
Curtis J. Bonk, Professor, Indiana UniversityPresident, CourseShare
http://php.indiana.edu/~cjbonk; [email protected]
Vanessa Paz Dennen, Assistant ProfessorFlorida State University
http://www.vanessadennen.com, [email protected]
Session ObjectivesSession Objectives
Detail online assessment techniques Discuss how to match learning activities with
learner assessments Examine instructor time and comfort issues Discuss ways to limit and detect cheating and
plagiarism Document online tools and resources for
assessment
Online Assessment Online Assessment TechniquesTechniques
(with some time-saving(with some time-saving
tips added in…)tips added in…)
Is this motivating? Is this motivating? How would you feel?How would you feel?
You take an online class.You take an online class. You read some Web pages.You read some Web pages. Maybe you watch some videos or hear some Maybe you watch some videos or hear some
audio clips.audio clips. Maybe you ponder some study review Maybe you ponder some study review
questions.questions. You take a multiple choice test online.You take a multiple choice test online. You receive an automated score on the test.You receive an automated score on the test. Class is over.Class is over.
How about this scenario?How about this scenario?
You take an online class.You take an online class. You “meet” your fellow students on the d-board.You “meet” your fellow students on the d-board. You read some materials. You find and share You read some materials. You find and share
some materials too.some materials too. You participate in some discussions of course You participate in some discussions of course
concepts.concepts. You take a multiple choice test.You take a multiple choice test. You receive an automated score on the test.You receive an automated score on the test. Class is over.Class is over.
Commentary on Scenario 1Commentary on Scenario 1
No interaction with peers. No interaction with peers. Students don’t feel “missed” if they don’t Students don’t feel “missed” if they don’t
participate.participate. Not clear why course is online (except Not clear why course is online (except
perhaps for media elements).perhaps for media elements). Potential for immediate feedback is nice -- Potential for immediate feedback is nice --
but assessment format is limiting.but assessment format is limiting.
Commentary on Scenario 2Commentary on Scenario 2
Interaction with peers is great. Serves as a Interaction with peers is great. Serves as a motivator.motivator.
Community is likely to develop.Community is likely to develop. Students will feel involved and important if Students will feel involved and important if
they share examples and resources.they share examples and resources. Assessment format may not be well Assessment format may not be well
aligned given the activities.aligned given the activities. Class lacks closure in a manner Class lacks closure in a manner
appropriate to the activities.appropriate to the activities.
Assessment and LearningAssessment and Learning
Course objectives, activities, and Course objectives, activities, and assessments should be in alignmentassessments should be in alignment This tends to be an issue in courses This tends to be an issue in courses
regardless of medium.regardless of medium. Example:Example:
In class students conduct a debateIn class students conduct a debate Students are tested on their ability to recall Students are tested on their ability to recall
factsfacts
Mis-aligned Online Learning Mis-aligned Online Learning and Assessmentand Assessment
A not-uncommon scenarioA not-uncommon scenario Discussion is used as a learning activityDiscussion is used as a learning activity Students are required to participateStudents are required to participate Participation is noted by how many messages were Participation is noted by how many messages were
composed by a studentcomposed by a student But does this method measure learning?But does this method measure learning?
Common Online Common Online Assessment ComplaintsAssessment Complaints
Instructor perspectiveInstructor perspective There’s too much to assess!There’s too much to assess! I don’t know what activities to assess!I don’t know what activities to assess! I don’t know if students really are ready for the I don’t know if students really are ready for the
test!test! How do I know the student actually did the How do I know the student actually did the
work/took the test?work/took the test?
Common Online Common Online Assessment ComplaintsAssessment Complaints
Student perspectiveStudent perspective If they’re supposed to discuss, why If they’re supposed to discuss, why
doesn’t that count as part of their grade?doesn’t that count as part of their grade? If they’re just supposed to do something, If they’re just supposed to do something,
why does quality matter?why does quality matter? I just got a number, no feedback.I just got a number, no feedback. I didn’t get participation feedback.I didn’t get participation feedback.
The Feedback IssueThe Feedback Issue
Students participating in online activities look for Students participating in online activities look for feedback to know:feedback to know: A. the instructor is reading their contributionsA. the instructor is reading their contributions B. their participation is valuedB. their participation is valued C. their participation is adequate, in terms of quality C. their participation is adequate, in terms of quality
and quantityand quantity Feedback need not be individualized to be Feedback need not be individualized to be
effectiveeffective Whole class commentary provided on a regular basis Whole class commentary provided on a regular basis
was found to be just as satisfactory from the student was found to be just as satisfactory from the student point of view (Dennen, 2001)point of view (Dennen, 2001)
The Assessment IssueThe Assessment Issue
Often, online activities go Often, online activities go unassessedunassessed ““Add-on” syndrome: Adding an Add-on” syndrome: Adding an
online activity to a previously online activity to a previously designed class because it designed class because it sounds like a good ideasounds like a good idea
The Assessment IssueThe Assessment Issue
Students are more likely to participate when Students are more likely to participate when then know there is impact on their gradethen know there is impact on their grade Direct impact: graded on participation Direct impact: graded on participation
(quality, quantity or both)(quality, quantity or both) Indirect impact: participation should Indirect impact: participation should
bolster performance on other assessmentsbolster performance on other assessments Students quickly become aware if an Students quickly become aware if an
online activity is not related to assessed online activity is not related to assessed learning objectiveslearning objectives
The Assessment Issue, Cont.The Assessment Issue, Cont.
Sometimes the wrong things are assessedSometimes the wrong things are assessed Examples:Examples:
Assessing students’ online moderation Assessing students’ online moderation skills when the course topic/learning skills when the course topic/learning objectives have nothing to do with online objectives have nothing to do with online moderationmoderation
Assessing quantity of participation, but Assessing quantity of participation, but not quality not quality • which, granted, is easier, but encourages which, granted, is easier, but encourages
sloppy message posting rather than thoughtful sloppy message posting rather than thoughtful learning dialogueslearning dialogues
Bloom’s Revised TaxonomyBloom’s Revised Taxonomy
A useful tool for checking alignmentA useful tool for checking alignment Also great to guide your course design!Also great to guide your course design!
QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.
To Find Out More…To Find Out More…
A TaxonomyA Taxonomy
For LearningFor Learning
Teaching andTeaching and
AssessingAssessing
By AndersonBy Anderson
And KrathwohlAnd Krathwohl
QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.
Assessment TechniquesAssessment Techniques
Options…Options… Formative or SummativeFormative or Summative Student-led Student-led (Self or Peer)(Self or Peer) or Teacher-led or Teacher-led Public or PrivatePublic or Private Process or ProductProcess or ProductOther issues to consider…Other issues to consider… ““Objective” or InterpretiveObjective” or Interpretive Rubric-based or WholisticRubric-based or Wholistic
Formative vs. Summative Formative vs. Summative AssessmentsAssessments
FormativeFormative Alleviate student Alleviate student
anxieties re: anxieties re: expectationsexpectations Seem especially high Seem especially high
in online classesin online classes
Encourage working Encourage working toward masterytoward mastery
Can be informalCan be informal
SummativeSummative Used for student’s Used for student’s
gradegrade Assumed to be “best” Assumed to be “best”
efforteffort
Example: Online Example: Online Formative AssessmentFormative Assessment
Paper draft discussion forumPaper draft discussion forum Start a discussion forum for papers-in-Start a discussion forum for papers-in-
progressprogress Have each student start a thread and post Have each student start a thread and post
elements of their papers as they complete elements of their papers as they complete them (e.g., topic, major claims, research them (e.g., topic, major claims, research sources)sources)• A schedule for each element is usefulA schedule for each element is useful
Monitor and provide feedbackMonitor and provide feedback
Student vs. Teacher LedStudent vs. Teacher Led
Teacher-ledTeacher-led ““Traditional” Traditional”
assessmentassessment Most often summativeMost often summative
Student-ledStudent-led Students may assess Students may assess
self or peersself or peers May be formative or May be formative or
summativesummative Can greatly relieve Can greatly relieve
instructor burdeninstructor burden Students reinforce Students reinforce
concepts through concepts through feedback processfeedback process
Examples: Online Self-Examples: Online Self-AssessmentAssessment
Self-tests: Use test tool to create self-tests Self-tests: Use test tool to create self-tests (multiple choice, true false)(multiple choice, true false) May wish to track student effortsMay wish to track student efforts Can incentivize use (essentially, use as a Can incentivize use (essentially, use as a
learning tool)learning tool) Reflection papers: Have students submit Reflection papers: Have students submit
brief, focused papers expressing the brief, focused papers expressing the strengths and assessments of their strengths and assessments of their assignment(s)assignment(s)
Example: Online Example: Online Peer Assessment (Formative)Peer Assessment (Formative)
Feedback groups: Assign students in Feedback groups: Assign students in groups to provide formative feedback on groups to provide formative feedback on projects and papersprojects and papers Often raises quality of assignmentsOften raises quality of assignments Need a structure with clear deadlinesNeed a structure with clear deadlines Need prompts and models to guide students Need prompts and models to guide students May wish to assess feedback May wish to assess feedback
process/contributionsprocess/contributions
Example: Online Peer Example: Online Peer Assessment (Summative)Assessment (Summative)
Conference Presentations: Have students Conference Presentations: Have students “present” their work and ask questions/provide “present” their work and ask questions/provide feedback to others.feedback to others. In d-board, have students attach papers to In d-board, have students attach papers to
messages; post a message with a synopsis; or attach messages; post a message with a synopsis; or attach a powerpoint presentationa powerpoint presentation
Each student/team should have their own threadEach student/team should have their own thread Feedback should occur during a defined period of Feedback should occur during a defined period of
time.time. May consider allowing students to rate assignments May consider allowing students to rate assignments
on certain dimensionson certain dimensions
Public vs. PrivatePublic vs. Private
PrivatePrivate Work is submitted to Work is submitted to
the teacher onlythe teacher only Entire burden of Entire burden of
feedback is on feedback is on teacherteacher
Important if assessing Important if assessing at fact levelat fact level
PublicPublic Peers can see each Peers can see each
others’ work (either in others’ work (either in process or process or completed)completed)
Peers may comment Peers may comment on each others’ workon each others’ work
Often increases Often increases quality of work quality of work submittedsubmitted
Process vs. ProductProcess vs. Product
ProductProduct The end deliverableThe end deliverable Look for polish, Look for polish,
accuracyaccuracy
ProcessProcess How the student got How the student got
therethere Look for Look for
thoughtfulness of thoughtfulness of approach, intentapproach, intent
Assessing ProcessAssessing Process
Easy to doEasy to do Many technology tools will archive student Many technology tools will archive student
work/interactionswork/interactions Students create a document trail in processStudents create a document trail in process
Helps students develop metacognitive Helps students develop metacognitive knowledgeknowledge Instructors structure/model/encourage productive Instructors structure/model/encourage productive
work processeswork processes Students learn how to manage their own work Students learn how to manage their own work
processesprocesses
Why Assess Process?Why Assess Process?
For the instructor …For the instructor … Provides formative feedback on course Provides formative feedback on course
(e.g., helps gather data about why students (e.g., helps gather data about why students have difficulty with product-oriented have difficulty with product-oriented assessments)assessments)
Clarifies who is doing most work in small Clarifies who is doing most work in small group assignmentsgroup assignments
Helps prevent cheatingHelps prevent cheating
Why Assess Process?Why Assess Process?
For the student …For the student … Typically improves the quality of their Typically improves the quality of their
productsproducts Helps them develop productive work Helps them develop productive work
processesprocesses Puts on a schedulePuts on a schedule Shows that you care about individual growthShows that you care about individual growth
Assessment Project CycleAssessment Project Cycle
From From Classroom Assessment TechniquesClassroom Assessment Techniques by Angelo & Cross (1993)by Angelo & Cross (1993)
Step 1: PlanStep 1: Plan Choose classChoose class Focus on assessable questionFocus on assessable question Design project to answer questionDesign project to answer question
Assessment Project Cycle [2]Assessment Project Cycle [2]
Step 2: ImplementStep 2: Implement Teach target lessonTeach target lesson Collect assessment dataCollect assessment data Analyze dataAnalyze data
Step 3:Step 3: Interpret resultsInterpret results Communicate resultsCommunicate results Evaluate assessment projectEvaluate assessment project
I. Term PapersI. Term Papers
How to do online:How to do online: Have students each start their own thread Have students each start their own thread
and post topic of interestand post topic of interest Peers and instructors give feedbackPeers and instructors give feedback Students post thesis statements, research Students post thesis statements, research
sources, etc., with iterations of feedbacksources, etc., with iterations of feedback Final paper is postedFinal paper is posted
Term Paper AssessmentsTerm Paper Assessments
Product: the paperProduct: the paper Process: quality and timeliness of Process: quality and timeliness of
student work from time when paper is student work from time when paper is assignedassigned
Process: quality and timeliness of Process: quality and timeliness of feedback provided to peersfeedback provided to peers
Process: responsiveness to feedback Process: responsiveness to feedback received from instructor and peersreceived from instructor and peers
II. Discussion AssignmentsII. Discussion Assignments
1. 1. Chain of thoughtChain of thought Have students develop a solution to a Have students develop a solution to a
problemproblem Have students indicate what led them to Have students indicate what led them to
a particular conclusion, method or a particular conclusion, method or approachapproach
Can be done in a discussion boardCan be done in a discussion board
Discussion AssignmentsDiscussion Assignments
2. 2. Theory to PracticeTheory to Practice Have students match up theories you are Have students match up theories you are
learning about to actual problemslearning about to actual problems Present students with problems and have Present students with problems and have
them explain what theories they would them explain what theories they would use to solve these problems and how they use to solve these problems and how they would approach itwould approach it
Debrief the assignmentDebrief the assignment
Discussion AssignmentDiscussion Assignment
3. 3. Synthesizer (i.e., wrapper)Synthesizer (i.e., wrapper) Have students take roles being the Have students take roles being the
weekly synthesizer of class discussionweekly synthesizer of class discussion Add a “meta” level in which students Add a “meta” level in which students
narrate their own experiences while narrate their own experiences while reading the weekly discussionreading the weekly discussion
Reflect on how life relates to discussionReflect on how life relates to discussion
III. Group ProjectsIII. Group Projects
Tools usedTools used Chat:Chat: brainstorming ideas, making group decisions, brainstorming ideas, making group decisions,
regular way to feel connected (should be archived)regular way to feel connected (should be archived)
Discussion board:Discussion board: commenting on drafts commenting on drafts
E-mail:E-mail: quick feedback quick feedback
File exchange:File exchange: sharing project files sharing project files
MS Word:MS Word: Track changes Track changes
Group Project AssessmentsGroup Project Assessments
Product: project files that are turned inProduct: project files that are turned in Process: online archive demonstratingProcess: online archive demonstrating
Who contributed whatWho contributed what Who provided peer feedbackWho provided peer feedback Who worked in a timely mannerWho worked in a timely manner How collaborative a group wasHow collaborative a group was
Process: peer ratingsProcess: peer ratings Process: interim instructor consultationsProcess: interim instructor consultations
Group Project AssignmentsGroup Project Assignments
1. Work Plans1. Work Plans Have students develop a plan of work Have students develop a plan of work
for their projectfor their project Make them outline topic, schedule, Make them outline topic, schedule,
resources needed, division of labor and resources needed, division of labor and anticipated form of final deliverablesanticipated form of final deliverables
At end of project, have students At end of project, have students evaluate how well they followed their evaluate how well they followed their own plan and how useful it wasown plan and how useful it was
Project AssignmentsProject Assignments
2. Research Trail2. Research Trail Have students document the steps Have students document the steps
they took in the research process and they took in the research process and the resultsthe results
Ask for a brief reflection on how Ask for a brief reflection on how effective their process was and what effective their process was and what they might change the next timethey might change the next time
Project AssignmentsProject Assignments
3. 3. Process PresentationsProcess Presentations Have students focus on their process as well Have students focus on their process as well
as their product in class presentationsas their product in class presentations To maintain focus, ask them to share 3 main To maintain focus, ask them to share 3 main
lessons learnedlessons learned Might ask for some process documents to be Might ask for some process documents to be
shared, like an early draftshared, like an early draft
Project AssignmentsProject Assignments
4. 4. Design JournalDesign Journal Have students maintain a journal of all Have students maintain a journal of all
ideas related to their projectideas related to their project Encourage sketches, lists, organizational Encourage sketches, lists, organizational
charts, etc.charts, etc. Require journals to be turned in with final Require journals to be turned in with final
projectsprojects
IV. Reflection AssignmentsIV. Reflection Assignments
Have students keep a weekly journal of Have students keep a weekly journal of their thoughts on readings and course their thoughts on readings and course content AND real-world related content AND real-world related instances that they noticedinstances that they noticed
May make these public, with each May make these public, with each student having their own discussion student having their own discussion threadthread
Making it HappenMaking it Happen
Learners need to see that process is valuable:Learners need to see that process is valuable: Model appropriate processesModel appropriate processes Provide students with scaffolding (guide Provide students with scaffolding (guide
sheets) to structure their processessheets) to structure their processes Give students feedback on their processGive students feedback on their process Require students to reflect on their processesRequire students to reflect on their processes Grade students on processGrade students on process
Vanessa’s TopVanessa’s TopTime-Saving TipsTime-Saving Tips
Before you assign it, ask yourself “can I reasonably Before you assign it, ask yourself “can I reasonably assess it?”assess it?”
Rely on students/peers for providing some feedbackRely on students/peers for providing some feedback Let students know what to expect up frontLet students know what to expect up front Choose the right tool for the jobChoose the right tool for the job
Get to know the editing and commenting features of your Get to know the editing and commenting features of your favorite programsfavorite programs
Use rubrics!Use rubrics! I make them in Word, and then while grading I highlight I make them in Word, and then while grading I highlight
or bold the section that applies to the student’s projects. or bold the section that applies to the student’s projects. Add a few comments at the bottom = speedy grading!Add a few comments at the bottom = speedy grading!