Slide 1

Slide 2

USING DATA TO IMPROVE LEARNING Pre-Conference Workshop Presenters Rob Johnstone, Senior Research Fellow, The Research & Planning Group for California Community Colleges, Berkeley, CA Kurt Ewen, Assistant Vice President, Assessment and Institutional Effectiveness, Valencia College, Orlando, FL

Slide 3

Overview of the Workshop Introductions Discussion about the Nature and Use of Data in Higher Education Implications for practice Playing with Actual Data and discussions about data display

Slide 4

INTRODUCTIONS Name The name and location of your College Your roll at the College What are you hoping for from the workshop?

Slide 5

USING DATA TO IMPROVE STUDENT LEARNING Think Pair Share In your work in higher education, what is the best piece of data you have ever seen? The most effective / actionable data? Why? An example of Great DataGreat Data What is different about data from learning assessment and student success?

Slide 6

LESSONS LEARNED ABOUT DATA COLLECTION AND USE

Slide 7

LESSON 1: THE PURPOSE OF ASSESSMENT (DATA) IS TO IMPROVE STUDENT LEARNING Assessment of learning creates the possibility of better conversations Course Level Assessment Faculty Student Student Students Program Level Assessment Faculty Faculty

Slide 8

LESSON 2: DATA FROM LEARNING OUTCOMES ASSESSMENT AT THE PROGRAM LEVEL IS MESSY AND THE MESSINESS IS AN IMPORTANT PART OF THE PROCESS An Example What should we expect from Students exiting Comp1 at a Community college?

Slide 9

LESSON 3: THE SEEMINGLY MORE USEFUL / UNDERSTANDABLE DATA ABOUT STUDENT LEARNING IS TO AN OUTSIDE AUDIENCE THE LESS ACTIONABLE THAT DATA WILL BE TO FACULTY. An Example What kinds of data can be reported as a result of assessment efforts using this rubric?

Slide 10

Slide 11

LESSON 4 A culture of evidence requires the development of an institutional practice that 1.gives careful consideration to the question being asked 2.gives careful consideration to the data needed in order to answer the question. A genuine culture of evidence is dependent on a culture of inquiry

Slide 12

WHAT IS A CULTURE OF INQUIRY? Institutional capacity for supporting open, honest and collaborative dialog focused on strengthening the institution and the outcomes of its students.

Slide 13

CULTURE OF INQUIRY: FEATURES Widespread sharing and easy access to user-friendly information on student outcomes Encouraging more people to ask a wider collection of questions and use their evidence and conclusions to enhance decision making Shared, reflective and dynamic discussions

Slide 14

CULTURE OF INQUIRY: MORE FEATURES Multiple opportunities to discuss information within and across constituency groups Continuous feedback so adjustments can be made along the way and processes can be adapted Culture that values curiosity, questions and robust conversations

Slide 15

1000s of opportunities tried. Maintain a Research and Development Component. Climate of Innovation Level I Level II Level III Eye for Evidence: More rigorous at each level. Standard of evidence and reflection increases at each level. 100 are selected for support as Phase I Innovations. Angel Capital Stage Prototype 10 supported as Phase II Innovations. Venture Capital Stage Pilot Implementation (Limited Scale) 1 or 2 are brought up to scale and Institutionalized. Level II Innovations must be scalable and must show potential to bring systemic change and business-changing results. The challenge is moving from Level II to Level III. We have yet to figure out how this will work in the new structure Strategy Institutionalization Hypothesis

Slide 16

When gathering evidence, make sure you are focusing on the right data. 16

Slide 17

Retention Rate Success Rate 19892008 20 YEAR TREND FOR CALIFORNIA CC COURSE SUCCESS RATES What does that tell us about the usefulness of these metrics in setting institutional strategies?

Slide 18

LESSON 5 A culture of evidence is one that seeks data supported decisions. Data driven decision making runs the risk of over looking / underestimating the human factor which is very often concealed by the desire for statistical significance Data driven decision making runs the risk of underestimating the role / significance / importance of evidence informed hunches to inform our decisions

Slide 19

LESSON 5 (cont) Data supported decisions concerning student success oriented programs requires a consideration of "meaningful improvement" and may require balancing all or most of the following: Statistically significant improvement in target measures. Reflection on the human impact Economic efficiency in relationship to difficulty of the task at hand. A consideration of perception as it relates to benefit versus cost.

Slide 20

LESSON 6 Structured reflection and dialogue allows for data to be transformed into meaningful (actionable) information The meaning of data in Higher Education is not generally self-evident and requires the benefit of the intersection of multiple perspectives The more meaningful learning data is to an outside audience the less actionable it is to those who work with students

Slide 21

Data do not speak for themselves. 21

Slide 22

THE VITAL ROLE OF CONVERSATION In order to make data useful, ample time and space are needed to discuss and analyze the information and connect it back to the original research question. Answers are not always immediately apparent, so skilled facilitation may be needed to dig out the deeper meaning. Multiple perspectives and types of information are often needed to make sense of individual data points. !

Slide 23

DIALOGUE VS. DISCUSSION AN ETYMOLOGICAL DISTINCTION Dialogue: Seeing the whole among the parts Seeing the connections between parts Inquiring into assumptions Learning through inquiry and disclosure Creating shared meaning Discussion: Breaking issues / problems into parts Seeing distinctions between parts Justifying / defending assumptions Gaining agreement on one meaning / result

Slide 24

LESSON 7 Meaningful information promotes consensus about lessons learned and a shared vision / plan for the future No data should be shared as information until it has been processed in a collaborative and thoughtful way.

Slide 25

Slide 26

LESSON 8 Meaningful information from data does not generally emerge from a single data point but from the intersection of multiple and varied (quantitative and qualitative) sources of data There is rarely a silver bullet (and if there is, then the question being asked is probably not particularly interesting) What data can be add to learning data to make it more meaningful Student Assessment of instruction data CCSSE Grade distribution report ??

Slide 27

LESSON 9 Data about student learning / success must be tailored to the needs / questions of particular groups Data concerning Students ability to write at the college-level Course Program Department Institution External Audience Etc.

Slide 28

LESSON 10 A simple assessment measures does not necessarily produce less meaningful / actionable data. Checklist 4 question multiple choice test

Slide 29

LESSON 11 The best insights often come as an unintended result of simple questions asked about things you were not planning to question.

Slide 30

LESSON 11 How are our new students doing? Data was provide on FTIC Degree Seeking Students Who are our new students? Development of our Philosophy statement on the New Student All Students with less than 15 College-level credits at Valencia Who are our new Students?

Slide 31

WHO ARE OUR NEW STUDENTS?

Slide 32

HOW ARE OUR NEW STUDENTS DOING?

Slide 33

National Institute for Learning Outcomes Assessment (NILOA) From Gathering to Using Assessment Results: Lessons from the Wabash National Study

Slide 34

FROM GATHERING TO USING ASSESSMENT RESULTS Most institutions have routinized data collection, but they have little experience in reviewing and making sense of data. It is far easier to sign up for a survey offered by an outside entity or to have an associate dean interview exiting students than to orchestrate a series of complex conversations with different groups on campus about what the findings from these data mean and what actions might follow.

Slide 35

LESSON 12 The sharing of Information should reflect standards of scholarly communication and evidence Biases and conclusions about the information should be clearly articulated Open and unanswered questions should be articulated (but should not be allowed to stop the process) Differing perspectives on the meaning of the information should be given equal time The use of programmatic and academic jargon should kept to a minimum The visual presentation of information should be monitored for consistency