Human Centred Collaborative Design: An Institutional Approach
Collaborative Assessment: Working Together Toward Institutional Change
-
Upload
elizabeth-nesius -
Category
Education
-
view
858 -
download
0
description
Transcript of Collaborative Assessment: Working Together Toward Institutional Change
TYCA Northeast Annual Conference 2010Capitol Improvements: The Two-Year College as an Agency of Change
Collaborative Assessment:
Working Together Toward Institutional Change
Kelly Bender and Elizabeth NesiusPassaic County Community College
This presentation can be found onSlideShare
This presentation looks at methods of collaboration for course-and program-level assessment and data-sharing to help both faculty and program administrators. Instructors can improve the WAC elements of their courses without sacrificing content, while still furthering the program mission of helping students to write well and think critically in any discipline. Writing program administrators, on the other hand, can improve both the writing program implementation and the training of its faculty to teach WAC courses. Students will benefit by experiencing WAC courses that are clearly defined, engaging, and optimized for their learning.
PCCC Writing Initiative
Title V Grant: $2.5 million over 5 yearsDevelop 20 Distinct, Writing Intensive
Courses (with multiple sections, across 3 campuses)
Develop and provide online resources for technology and writing
Support students and faculty with an on-ground writing center and online tutoring
Students required to take 2 WI courses (1 if getting AAS degree)
Writing Intensive Courses
19 Distinct WI courses currently developed or in development across the curriculum
Departments include: Computer Information Systems, English, Humanities, Math, and Science
Courses designed to contain writing, critical thinking, and information literacy elements
What We Assess
Students◦ Assessed in writing, critical thinking, and information
literacy skills◦ Assessed in the course by instructors
Courses◦ Course materials created collaboratively in Faculty
Institutes◦ Course materials checked and approved by WI
AdministratorsProgram
◦ Students assessed across courses by WI administrators in the 3 areas
◦ Students and instructors self-assess by means of a survey
How We Assess
Students◦Instructors: student grades◦Administrators: student performance on CWE
Courses◦Course materials collected at the beginning of each
semester◦Evidence of materials being used by student use of
ePortfoliosProgram
◦Student writing assessed with 3 rubrics: writing, critical thinking, information literacy
◦Student and faculty surveys
Assessing Writing
First we had to determine the criteria for writing assessment:◦Appropriateness◦Unity & Focus◦Development◦Organization◦Sentence Structure◦Usage◦Mechanical Conventions
Additional criteria for course writing are left up to individual instructors
Writing Rubric Student __________________________________ Assignment ______________________________________ Beginning Developing Competent Accomplished
Vocabulary/Language
appropriate to this course
Words used incorrectly; displays fundamental errors in vocabulary usage; lacks variety.
Generally correct usage but little variety; some variety but words used incorrectly.
The majority correctly used with some variety in vocabulary.
Varied vocabulary; correct and sophisticated use of language.
Conventions mechanics, spelling, sentence structure,
word forms (problems noted below)
Many serious errors which persistently interfere with meaning.
Some serious errors that may interfere with meaning.
A few errors which do not interfere with meaning.
No serious errors; few minor errors.
Organization
includes: appropriate length, order (logical, chronological etc.) and
transitions
No clear structure; organizational requirements not met.
Some structure and order evident; minimally, an introduction, body & conclusion present.
Generally well-organized; clear and logical flow of ideas. Fulfills the assignment.
Well-organized with appropriate structure and smooth transitions.
Content
includes addressing the assigned topic
Lacks thesis; incomplete development of evidence and/or minimal relevance to topic.
May lack thesis. Some evidence underdeveloped or off-topic.
Has supporting evidence and examples for thesis.
Strong, convincing evidence to support thesis and argument.
Tense formation subject-verb agreement pronoun usage/agreement word choice/meaning proper modifiers Variety of formations correct construction Spelling Capitalization punctuation
COMMENTS:
WI v10.1
Assessing Writing
EN 205 – Introduction to Literature Grading Rubric
Student __________________________________ Assignment ______________________________________
Accomplished Competent Developing Beginning
Vocabulary/Language appropriate to this course
Varied vocabulary; correct and sophisticated use of language.
The majority correctly used with some variety in vocabulary.
Generally correct usage but little variety.
Words used incorrectly; displays fundamental errors in vocabulary usage; lacks variety.
Conventions (problems noted below)
No fundamental errors; few minor errors.
A few errors which do not interfere with meaning.
Some fundamental errors that may interfere with meaning.
Many fundamental errors which persistently interfere with meaning.
Organization includes: appropriate length, order
(logical, chronological etc.) and transitions
Well-organized with appropriate structure and smooth transitions.
Generally well-organized; follows basic assignment guidelines; clear and logical flow of ideas.
Some structure and order evident, attempt at following assignment guidelines; minimally, introduction, body & conclusion.
No clear structure; does not follow assignment guidelines; incoherent.
Content includes addressing the assigned topic
Strong, convincing evidence to support thesis and argument.
Has supporting evidence and examples for thesis.
May lack thesis or conclusion. Evidence underdeveloped or off-topic.
Lacks thesis and/or conclusion. Minimal relevance to topic.
Citation of sources
Uses citations when appropriate and demonstrates mastery of formatting; goes beyond the amount of citation or sources required.
Uses required amount of citations when appropriate and uses proper formatting most of the time.
Uses citations infrequently and shows inconsistent formatting; understands some elements of citation but not others.
Uses little or no citation and displays little knowledge of formatting; clearly cannot differentiate between most of the elements of citation.
USAGE
Tense formation
subject-verb agreement
pronoun usage/agreement
word choice/meaning
proper modifiers SENTENCE CONSTRUCTION
Variety of formations
correct construction MECHANICS
Spelling
Capitalization
punctuation
COMMENTS:
Revised 11/1/2010
Assessing Writing
Faculty use rubric with students for formative writing assessment
Extra space to add content or other elements for assessment based on assignment
WI Administrators use rubric for summative assessment of writing (without additional elements)
TOOLS: The Writing Center
The Writing Center provides support for students at all stages of the writing process, whether it is brainstorming a topic, structuring an essay, or editing and revising.It also offers support to students for their Writing Intensive courses in the forms of face-to-face and small group tutoring/instruction.
The Writing Center
While the Writing Center made the assessment of student writing “easier,” it was also helpful in terms of assessing the course.
Attendance at the Writing Center was made mandatory. This created a direct correlation between the outcome of student writing and the effectiveness of a support tool.
Assessing Critical Thinking
Criteria for critical thinking assessment◦Identifying arguments and counter-arguments◦Assessing validity of both sides◦Drawing conclusions based on those
assessmentsFaculty use rubric with students for
formative critical thinking assessmentAdministrators assess critical thinking
across courses (in writing assignments that require critical thinking) using same rubric
Assessing Critical Thinking
Critical Thinking Rubric STUDENT _________________________________________ Assignment ______________________________
Beginning Developing Competent Accomplished
Evidence
Biased interpretations Insufficient search for
evidence or inappropriate
evidence choices
Misinterprets some evidence
Limited information search
limited source evaluation or attribution
Accurately interprets the majority of evidence
Sufficient search for evidence
Shows some evaluation of sources
Appropriate evidence Accurately interprets the
evidence presented
Arguments
Fails to identify argument
or counter-arguments
Has argument but inadequate identification of relevant counter-arguments.
Identifies relevant argument(s)
including counter-arguments
Identifies pertinent arguments, pro and con
Rebuts counter arguments
Conclusion(s)
Does not reach any conclusion justified by evidence.
Conclusion based on incorrect, irrelevant , or insufficient evidence
Conclusion(s) are reasonably justified or reasonably explained.
Draws thoroughly justified conclusions by appropriate interpretations of evidence
Comments
V4 June 2010
Assess Information Literacy
Criteria for information literacy assessment◦Defines and articulates need for information◦Information retrieval◦Citation of sources◦Evaluation of Sources◦Uses information effectively
Information literacy required in at least one assignment
Assessment by faculty and administrators using the same rubric
Assessing Information Literacy
Information Literacy Rubric Student ________________________ Reader ____________________ Score _____________ Beginning Developing Competent Accomplished Defines and articulates the need for information
No clear statement of purpose; does not associate information need with the assignment
Shows purpose but not clearly articulated; associates information need with the assignment but does not articulate a thesis statement
Purpose is clearly articulated and information need is clearly recognized
Purpose is highly focused and/or nuanced; thesis statement or research question takes multiple viewpoints into account
Information retrieval
Retrieves little or none of the information required for the assignment; or, information retrieved clearly does not meet the information need
Retrieves some of the information required for the assignment; some of the information retrieved may not meet the information need
Retrieves the required amount of information for the assignment and the majority of it meets the information need
Retrieves much more than the required amount to complete the assignment; information retrieved meets the information need; multiple viewpoints are considered
Citation of sources
Uses little or no citation and displays little knowledge of formatting; clearly cannot differentiate between most of the elements of citation
Uses citations infrequently and shows inconsistent formatting; understands some elements of citation but not others
Uses citations when appropriate and uses proper formatting most of the time
Uses citations when appropriate and demonstrates mastery of formatting; goes beyond the amount of citation or sources required
Evaluation of Sources
Little or no recognition of the relevance and quality (authority, currency, objectivity) of information sources
Recognizes the relevance of information sources but has difficulty evaluating their quality
Recognizes both the quality and relevance of information sources
Information sources are highly relevant and evaluation of sources is highly nuanced
Uses information effectively in their writing assignments
Uses little or no useful information from outside sources
Attempts to incorporate information from outside sources; sometimes off-topic
Uses information from outside sources to support the topic; shows some synthesis with own ideas
Use of information from outside sources is highly refined, articulated, and synthesized with own ideas
Comments:
Revised 6/6/08
Assessing the Course
Course assessment is essential in determining:
Both the quality and quantity of learning for students
An appropriate response to their personal development needs
Whether students have acquired the knowledge, skills and competencies consistent with the aims of the course
The effectiveness of the curriculum and faculty development
Methods of Course Assessment
By Instructors◦Faculty institutes◦ePortfolios◦Course Survey
By Administrators◦Course materials◦ePortfolios
Course Assessment: Faculty Institutes
Faculty Administrators
Institute Part 1◦ Overview of Initiative◦ Overview of WI course
components◦ Look at course as it
stands for writing, critical thinking and information literacy elements
◦ Begin development of new materials with rubrics and templates
Institute Part 1◦ Collect current course
materials (syllabus and assignments)
◦ Assist in development of new materials
◦ Collect and file new materials
◦ Survey faculty on success of workshop components
Faculty Institutes
Faculty Administrators
Institute Part 2◦ Some materials tested
in regular section of courses
◦ Materials refined based on in-course testing
◦ More materials developed
◦ Integration of media and other resources
Institute Part 2◦ Discuss piloting of
materials from part 1◦ Assist in refining
materials and development of new materials
◦ Assist in incorporating media, other resources and support services
Assessing the Initiative
The program is assessed for accomplishment of its goals◦For faculty development◦For student development
Methods of Program Assessment◦ePortfolios◦Student Survey◦Faculty Survey◦Focus Groups
ePortfolio Information
Electronic portfolios offer three distinct benefits: 1) They are a creative, lively form in which to show
a student’s best work; 2) They allow students to gain an awareness of a
larger audience (a portfolio can be published on an intranet or the Internet or saved to a CD and given to college representatives, employers, or anyone a student wants to impress);
3) They can evolve and grow over a student’s entire academic career.
"From Worn-Out to Web - Based : Better Student Portfolios." Phi Delta Kappan 85.10 (June 2004): 792-794. Academic Search Premier. EBSCO. [Library name], [City], [State abbreviation]. 27 Aug. 2009 <http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=13292501&site=ehost-live>.
ePortfolio Information
The portfolio not only offers a tool for authentic assessment but also a means for students to be reflective practitioners, emphasizing the how and why as much as the what. Time spent in portfolio assessment is not time taken away from teaching or academics, but time refocused and redefined, with the portfolio viewed as a natural complement to learning.
Lombardi, Judy. "To Portfolio or not to Portfolio: Helpful or Hyped?." College Teaching 56.1 (Winter2008 2008): 7-10. Academic Search Premier. EBSCO. [Library name], [City], [State abbreviation]. 27 Aug. 2009 <http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=31161341&site=ehost-live>.
ePortfolio Information
Assessment findings derived from students, tutors, and teachers indicate that electronic portfolios have had several positive effects on student learning. These portfolios vividly record writing as a process, providing students and teachers with an effective means of assessing the development of that process over a semester. In addition, they effectively display the final results of the semester, including a student’s self-assessments in the form of written reflections.
Click, Ben A., and Sarah C. Magruder.. "Implementing Electronic Portfolios for Performance Assessment:A Pilot Program Involving a College Writing Center." Assessment Update 16.4 (July 2004): 1-15. Academic Search Premier. EBSCO. [Library name], [City], [State abbreviation]. 27 Aug. 2009 <http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=13998617&site=ehost-live>.
Assessment via ePortfolios
Faculty Administrators
Used for student and course assessment
Students upload all formal assignments to portfolios
Allows instructors to look at student progress throughout the course
Students write reflections on various aspects of the course, or on the course as a whole
Used for course and program assessment
Random selection of portfolios to review across courses
10% of portfolios assessed
Look at assignments for writing, critical thinking, and information literacy elements
Sample ePortfolios
www.stevecuello.v2efolioworld.mnscu.edu www.kristycancel.v2efolioworld.mnscu.ed
u
www.luzcampusano.v2efolioworld.mnscu.edu
Course Survey
Used by instructor for course assessmentProvides the instructor with feedback from
his or her studentsThrough narrowly focused questions,
specific feedback is received on:◦Exact assignment◦Support systems in place (i.e. The Writing
Center, eTutoring.org)◦Text (book and reading selections)◦Instructor◦Grading Rubric
Student Survey & Results
1. When you registered for this course, were you aware that it was a Writing Intensive course? 68% Yes
2. What is your overall opinion of the material covered in the course?
3. Would you take another WI course? 93% Yes4. What is your experience with eTutoring.org?5. What is your experience with the Writing
Center?6. What did you like best about this course?7. What, if anything, would you
change/add/remove from the course?
Student Testimonials
“I like to read more because of this class and Professor Bender.”
“I never thought I was going to have so much fun in a Literature course or on an intensive writing course. I’m glad it was a great experience.”
I definitely improved my writing skills. I think if this was just an intro to lit class it would be boring, the writing is what made it fun. I feel so much more confident in my writing.”
WI Student Survey
Used by WI Administrators for program assessment
Conducted across all WI coursesNeither students nor courses identifiedAllows students to report on various
aspects of the WI experience◦Workload◦Support services◦Self-assessment on improvement in writing,
critical thinking and information literacy
WI Student Survey Results
Were you aware that you registered for a WI course? ◦57% Yes◦43% No
Rating of services◦eTutoring
Excellent or Good 68.2% Fair or Poor 20.5% Did not use 11.4%
◦Writing Center Excellent or Good 62.2% Fair or Poor 6% Did not use 31.8%
WI Student Survey Results
Rating of services◦Libguides
Excellent or Good 50.4% Fair or Poor 13.5% Did not use 36.1%
◦ePortfolios Excellent or Good 44.7% Fair or Poor 24.2% Did not use 31.1%
Rating of skills◦Writing
Improved 71.5% About the same 27.7%
WI Student Survey Results
Rating of skills◦Information literacy
Improved 63.4% About the same 31.3%
◦Critical Thinking Improved 66.4% About the same 29.1%
◦Technology Improved 50.7% About the same 42.5%
Faculty Survey
All WI Faculty surveyedSurvey administered online, anonymousAllows faculty to report on services offered in
the Writing Initiative◦Institutes◦Professional development opportunities◦Support services
Faculty report on observations of student improvement in the three areas
Report on own improvement in technology and media use and use of support services
Faculty Survey Results
Familiar with Initiative goals for students◦Use the writing process: 100%◦Support of claims with specific evidence: 89%◦Edit writing according to the rules of standard
academic English: 100%◦Evaluate sources for credibility and academic
appropriateness: 89%◦Employ techniques for integrating information:
67%◦Exhibit the ability to think critically: 100%◦Cite sources using an appropriate
documentation style: 89%
Faculty Survey Results
100% found the goals helpful for improving student learning
67% used rubrics for evaluation before teaching a WI course
87% used rubrics for evaluation in WI courses
33% made use of available tutoring services for their classes before teaching WI courses
Faculty Survey Results
100% incorporated tutoring into their WI courses
67% felt their students were more comfortable doing and integrating research after taking a WI course
78% felt their students were more aware of critical thinking after taking a WI course
89% felt that WI components had value for their other courses
Focus Groups
Conducted by Institutional ResearchRecorded discussion of strengths and
weaknesses of programWI Administrators and WI Faculty met
separatelyIR wrote report analyzing the results for
each group and making recommendations
Focus Group Results
Faculty Administrators
Felt that there were too many technological elements required in the course
Wanted an increase in communication among WI faculty. They felt well-supported by the staff and often communicated with them, however.
Desire more help from the college community as a whole to overcome obstacles to program success
See communication, especially among faculty and program administrators, as key to program success
For the Future
Intro. to Literature Writing Initiative
Work more closely with counselors/registrar/advisors to ensure that students are aware of the course they are taking
Possibly incorporate more use of the Writing Center
Increase the amount of available tech support to allow for easier ePortfolio use
Better marketing of WI courses to students
Better marketing of the Writing Center to students
Better marketing of professional development opportunities to WI faculty
Allow faculty to use either paper or online portfolios for their classes
Thank you!
Kelly Bender: [email protected] Nesius: [email protected]
Portfolio Results
Writing◦Minimum pass rate or higher: 77%◦Below passing: 23%
Critical Thinking◦Minimum pass rate or higher: 25%◦Below passing (or no evidence of CT): 75%
Information Literacy◦Minimum pass rate or higher: 50%◦Below passing in one or more areas: 50%
Sample ePortfolios
Sample ePortfolios
Sample ePortfolios