GCAT-SEEK Workshop, 2013Dr. Tammy Tobin, Susquehanna University

Adapted from Mary J. Allen, SACS-COC Summer Institute

Developing and Using Rubrics to Assess Vision and Change

Outcomes

Designing and Assessing Student Activities for

Vision and Change Learning Goals - Overview

• Use “Backwards” Course Design

Identify student learning goal(s) first....

THEN decide on activities/topics that best allow students to gain the relevant skills, knowledge and/or abilities

THEN decide on a rubric (if needed) and make sure that it actually addresses the learning goal in ways that are appropriate for the activity

Identifying Student Learning Goals: V&C Core

Concepts• Evolution

• Structure and Function

• Information Flow, Exchange and Storage

• Pathways and Transformations of Energy and Matter

• Systems

Identifying Student Learning Goals: V&C Core Competencies

• Ability to Apply the Process of Science

• Ability to Use Modeling and Simulation

• Ability to Use Quantitative Reasoning

• Ability to Tap Into the Interdisciplinary Nature of Science

• Ability to Communicate and Collaborate with Other Disciplines

• Ability to Understand the Relationship Between Science and Society

Decide How to Best Assess Student

Learning• Direct vs. Indirect Assessment

Authentic Assessment

• Formative vs. Summative Assessment

• Imbedded vs. External Assessment

The GCAT SEEK Assessment Instrument

The Challenge: Develop an assessment instrument that is broad enough to assess the outcomes of ALL GCAT-SEEK projects for ALL V&C concepts and competencies…

Stage One: Develop broad learning goals for GCAT SEEK – Using V&C categories as guide

Stage Two: Develop and implement a survey/quiz in Google Forms to assess participant progress on (some of) these learning goals

Stage Three: Refine the instrument/identify gaps in V&C coverage

Stage Four: Use data to improve GCAT-SEEK activities

Rubrics• Classify continuous behaviors (V&C Competencies!) into categories

• Can be developed to classify any student product

• Can be used to provide formative feedback to students, to grade students, and/or to assess programs

• Are most effective when they are generated at the same time as the assignment

Creating Rubrics

• Adapt a pre-existing rubric to your needs

Loyola Marymount University has a good database of samples, including those from the AAC&U VALUE project.

•http://www.lmu.edu/about/services/academicplanning/assessment/Assessment_Resources/Rubrics/Example_Rubrics.htm

Steps for Creating Your Own Rubrics

1. Write down the learning outcome(s) to be assessed:

Vision and Change Core Competencies:• (STUDENTS WILL BE ABLE TO) TAP INTO THE INTERDISCIPLINARY NATURE OF SCIENCE.

• (STUDENTS WILL BE ABLE TO) COMMUNICATE AND COLLABORATE WITH OTHER DISCIPLINES.

• (STUDENTS WILL) UNDERSTAND THE RELATIONSHIP BETWEEN SCIENCE AND SOCIETY.

Steps for Creating Your Own Rubrics

2. Identify measurable central components or skills for the learning goals, using the Vision and Change descriptors as as your guide.

Ability to tap into the interdisciplinary nature of science

Interdisciplinary science practice may be achieved in several ways. For future biologists, one way is through developing expertise not just in an area of biology, but also in a related discipline. That way, students will develop the vocabulary of both disciplines and an ability to think independently and creatively in each as well. A second, less intensive approach is to develop deep expertise in one area and fluency in related disciplines. A third option is to serve as a biologist on a multidisciplinary team. All of these routes develop a student’s facility to apply concepts and knowledge across traditional boundaries. For those not majoring in biology, the inherent interdisciplinary nature of biology practice lends itself to forming connections between biology and other sciences and, in so doing, can help all students understand the way science disciplines inform and reinforce each other.

Steps for Creating Your Own Rubrics

3. Develop an assessment instrument:

GCAT-SEEK Essay:Science is an interdisciplinary, collaborative process that can be of great use to society, but only if it is conveyed in a manner that makes its importance clear to both scientists and non-scientists. We are interested to see if you have this ability. Write a short (2-3 paragraph) essay in which you try to convince a charitable foundation to fund an interdisciplinary next generation sequencing project. In your essay you should briefly explain the project that you will be performing, why it is important to society, and how next generation sequence data will help you complete the project. Also convince the foundation to fund the work of at least one individual who is not a molecular biologist, but without whom your research project would be less meaningful or effective.”

Steps for Creating Your Own Rubrics

4.Decide which rubric type to use (holistic or analytic).

Holistic Scoring - One global, holistic score for a product or behavior.

• See rubric examples 1-2 at end of packet

Analytic Rubrics - Separate, holistic scoring of specified characteristics of a product or behavior

• See rubrics 3-4 at end of packet

ABILITY TO TAP INTO THE INTERDISCIPLINARY NATURE OF SCIENCE.

55Accurately and completely describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective.

Steps for Creating Your Own Rubrics5. Create a descriptor for the highest student performance level. Keep it short, simple and measurable. Make sure the rubric asks the student to perform the task!Science is an interdisciplinary, collaborative process that can be of great use

to society, but only if it is conveyed in a manner that makes its importance clear to both scientists and non-scientists. We are interested to see if you have this ability. Write a short (2-3 paragraph) essay in which you try to convince a charitable foundation to fund an interdisciplinary next generation sequencing project. In your essay you should briefly explain the project that you will be performing, why it is important to society, and how next generation sequence data will help you complete the project. Also convince the foundation to fund the work of at least one individual who is not a molecular biologist, but without whom your research project would be less meaningful or effective.”

Creating Your Own Rubrics

6. Identify the word(s) in the descriptor that can vary to convey different levels of performance

Accurately and completely describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective.

• Vary these words when writing your lower performance level descriptions.

Creating Your Own Rubrics

7. Create a descriptor for the lowest student performance level.

Accurately and completely describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective

Does not accurately describe how either of the chosen techniques are necessary to make the research project chosen more meaningful or effective

ABILITY TO TAP INTO THE INTERDISCIPLINARY NATURE OF SCIENCE

55Accurately and completely describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective

44

Accurately and completely describes how either next generation sequencing or at least one other discipline is necessary to make the research project chosen more meaningful or effective. However, the description of the role of the other technique, while accurate, is superficial

33

Accurately but superficially describes how both next generation sequencing and at least one other discipline are necessary to make the research project chosen more meaningful or effective

22

Accurately describes how either next generation sequencing or at least one other discipline is necessary to make the research project chosen more meaningful or effective. However, the description of the role of the other technique is inaccurate

11Does not accurately describe how either of the chosen techniques are necessary to make the research project chosen more meaningful or effective

8. Decide how many levels to use, and write descriptors for the intermediate levels.

Practice, Practice, Practice …In teams, identify the central components or skills for the learning goals below, using the Vision and Change descriptors as as your guide (V&C page 15).

• (STUDENTS WILL BE ABLE TO) COMMUNICATE AND COLLABORATE WITH OTHER DISCIPLINES.

• (STUDENTS WILL) UNDERSTAND THE RELATIONSHIP BETWEEN SCIENCE AND SOCIETY:

Practice, Practice, Practice…

Next, create a descriptor for the highest and the lowest student performance levels. Make sure they are measurable, and that they measure something the student was actually asked to do.

55

44

33

22

11

Varying the appropriate terms, write descriptors for all of the levels of achievement in the chart below

Using RubricsPilot Testing the Rubric

Managing Group ReadingsOne reader per document

Requires careful reader calibration!Two independent readers per documentPaired readers

Analyzing the Data

• Which level of performance is "good enough“?

• Consider using a preformed computer spreadsheet to quickly enter and process your data as you go.

• Modify the rubric/prompt for the next cycle, as needed.

• Reward yourselves!