10/30/01 EDUCAUSE 2001 1

Copyright Statement

Copyright [Mzumara, Gavrin, & Chisholm] [2001]. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

10/30/01 EDUCAUSE 2001 2

WebScience: Assessment of Just-in-Time Teaching (JiTT)

at IUPUIHoward R. Mzumara

Andrew D. Gavrin & Robin ChisholmIndiana University Purdue University Indianapolis

Tel: 317-278-2214 Fax: 317-274-3400E-mail: hmzumara@iupui.edu

http://webphysics.iupui.edu/webscience/assessment.html

10/30/01 EDUCAUSE 2001 3

Presentation Outline

Brief background information about JiTTWhat is JiTT? Components of JiTT StrategyJiTT DiversityMain Instructional Objectives for JiTT ClassAssessment of JiTT: Evaluation QuestionsMethods of Data CollectionMethods of Data AnalysisPreliminary Results To DateSummary of Findings and Conclusions

10/30/01 EDUCAUSE 2001 4

The Problem

Many students come to college underprepared to take college-level coursesIntroductory science and math classes have a mix of traditional and nontraditional students with a wide range of backgrounds and skillsStudent retention is a major problemGoal: Meet students at their current ability level and help them learn and gain from college courses they enroll in, and help to improve retention.

10/30/01 EDUCAUSE 2001 5

What is JiTT?

JiTT involves Web assignments and an interactive classroom that fundamentally rely upon one another

JiTT is a strategy for improving student learning, retention, and attitudes

http://webphysics.iupui.edu/jitt/jitt.html

10/30/01 EDUCAUSE 2001 6

JiTT Requires Feedback

Feedback Loop between the Web and Classroom Activities

10/30/01 EDUCAUSE 2001 7

Web-based Components of JiTT

“Warm-Up”- short quiz or conceptual exercise with an

electronic due date just hours before class

“Puzzle” - weekly assignment

“What is (Physics) Good For?”

– weekly essay on a related topic (“extra credit”)

Information and communication pages– Chapter Outlines, News, etc.

10/30/01 EDUCAUSE 2001 8

JiTT Diversity

Several Science & Math Courses use JiTT: – Physics 152 (Mechanics and Sound) &

Physics 251 (Heat, Electricity and Optics)– Biology N100 (Contemporary Biology)– Chemistry 105 (Principles of Chemistry)– Mathematics M118 (Finite Mathematics) &

Mathematics 164 (Integrated Calculus & Analytic Geometry)

10/30/01 EDUCAUSE 2001 9

JiTT Course Homepages at IUPUI:http://webphysics.iupui.edu/webscience/webscience.html

Physics 152 & 251: http://webphysics.iupui.edu/introphysics.html

Biology N100:http://www.biology.iupui.edu/biocourses/N100/index.html

Chemistry 105:

http://webphysics.iupui.edu/webscience/courses/chem105/chem105home.html

Mathematics M118:

http://webphysics.iupui.edu/webscience/courses/mathm118/mathm118home.html

10/30/01 EDUCAUSE 2001 10

Main Instructional Objectives:

Encourage frequent, short study sessions (facilitate development of good study habits and time

management skills)

Connect textbook to the real worldEncourage the development of students’

• critical thinking skills, • estimation skills, and • ability to deal with ill-defined problems

Develop cooperative work habits and communication skills

• Emphasize teamwork, collaborative learning, and effective communication

10/30/01 EDUCAUSE 2001 11

Using the Warm-Ups: “The Interactive Lecture”

Before class:– ½ hour to review responses– 15 minutes to select quotes– Lecture emphasis may be adjusted

During Class:– Lectures are built around student answers to

Warm-Ups– Quotes from scafford– Lecture content is unaffected– Students gain ownership of “lecture” session

10/30/01 EDUCAUSE 2001 12

The Recitation Section

Format of collaborative recitation/discussion session:

– 15 minute “sermon” from instructor (short review of homework)

– Students form groups (teams of 2-4 students)

– Groups work on white-boards (and practice problem solving, teamwork & communication)

– Faculty, peer mentors circulate in classroom

10/30/01 EDUCAUSE 2001 13

High Tech vs. Low Tech Approach

High Tech:– “Warm-Up Questions”– Electronic communication– Other online activities (e.g., Web search)

Low Tech:– Workplace skills– Course community– Mentoring

10/30/01 EDUCAUSE 2001 14

For more about Just-in-Time Teaching

Book by Novak, et al. (1999)Just-in-Time Teaching: Blending Active Learning with Web Technology

(published by Prentice Hall)

JiTT Website: http://webphysics.iupui.edu/jitt.html

Physics Text: http://www.prenhall.com/giancoli/

10/30/01 EDUCAUSE 2001 15

Assessment of JiTT: Evaluation Questions

To what extent is the JiTT strategy helping students?In what specific areas is the introduction of JiTT making an impact in facilitating teaching and learning in science and mathematics courses?In what ways can the JiTT courses be significantly improved?

10/30/01 EDUCAUSE 2001 16

Data Collection Methods

Mixed-method evaluation approach included:

Document Review

Student Attitude Surveys

Classroom Observation

Focus Groups

Achievement Tests or Common Final Exams

10/30/01 EDUCAUSE 2001 17

JiTT Assessment: Evaluation Instruments

Student Attitude Surveys– Pre- and Post-Course Survey consisting of six (6)

sub-scales or dimensions:• Confidence• Expectations• Rapport• Study Skills• Group Work• Computer

10/30/01 EDUCAUSE 2001 18

JiTT Assessment: Evaluation Instruments – cont’d

Classroom Observation– Rating Scale and Narrative Form

• Purpose: To obtain a view of the climate, rapport, interaction, and functioning of JiTT classes (particularly with respect to checking the adequacy and effectiveness of the JiTT strategy in facilitating interactivity, teamwork, or collaborative learning among students and faculty)

10/30/01 EDUCAUSE 2001 19

JiTT Assessment: Evaluation Instruments – cont’dStudent Focus Groups– Purpose: To obtain students’ perspectives and

experiences regarding use of Web resources to enhance learning, particularly with respect to making classroom activities more interactive, efficient, and more appropriate for meeting needs.

– Focus groups were designed to evaluate the impact and effectiveness of the JiTT strategy (particularly use of Web resources) in support of teaching and learning in science and math.

10/30/01 EDUCAUSE 2001 20

URL for JiTT Assessment

Copies of assessment instruments are available online at:

http://webphysics.iupui.edu/webscience/assessment.html

10/30/01 EDUCAUSE 2001 21

Methods of Data Analysis

Qualitative Data Analysis:– Content Analysis– Formulation of Coding Categories or Theme

Quantitative Data Analysis – Descriptive and Inferential Statistics– Use of Frequency Counts and/or

Percentages

10/30/01 EDUCAUSE 2001 22

Reporting of Evaluation Findings

Three Broad Areas of Assessment:

Course Attrition Rates (retention/persistence)

Student Cognitive Gains (cognitive factors)

Student Attitudes (affective area)

10/30/01 EDUCAUSE 2001 23

Assessment: Attrition Rates (PHYSICS)

Measure No JITT Using JiTT

DFW rate (Mechanics)

48% 29%

DFW rate (E&M)

33% 17%

Cognitive Gain<g> on FCI Not Available 0.31

10/30/01 EDUCAUSE 2001 24

Assessment: Attrition Rates (PHYS 152)

First Semester Attrition (152)

0%

10%

20%

30%

40%

50%

60%

70%

Sp94 Sp95 SP96 Sp97 Sp98 Sp99 Sp00 Sp01

Semester

Att

riti

on

(D

FW

)

10/30/01 EDUCAUSE 2001 25

Assessment: Attrition Rates (PHYS 251)

Second Semester Attrition (251)

0%

10%

20%

30%

40%

50%

60%

70%

80%

Semester

Att

riti

on

(D

FW

)

10/30/01 EDUCAUSE 2001 26

Cognitive Area: GPA Comparisons

Preliminary GPA Comparison in Physics (Note: USAFA classes only):

GROUP Incoming GPA Outgoing GPA

No JiTT 2.693 2.719

Consistent JiTT 2.796 2.967

10/30/01 EDUCAUSE 2001 27

Cognitive Area: Performance on FCI

Force Concept Inventory (FCI):– Nationally accepted achievement test– Conceptual understanding– <g> = 0.23 (traditional courses)– <g> = 0.31 (IUPUI Phys 152)– <g> = 0.48 (good interactive courses)

10/30/01 EDUCAUSE 2001 28

Affective Area: Student Attitudes

Student responses to end-of-semester voluntary anonymous survey on the use of JiTT strategy were overwhelmingly positive94% of students indicated they thought Warm-Ups were a good idea

Students commented repeatedly that Warm-Ups helped them to learn rather that memorize, be prepared for class, and keep up (pace) with material throughout the semester

10/30/01 EDUCAUSE 2001 29

Affective Area - continued

Post-course satisfaction survey:– Students were asked “If you could pick your

instructor and you had a choice between two equal instructors, except that one instructor used Warm-Ups and the other didn’t, which instructor would you pick?”

91% preferred Warm-Ups.

10/30/01 EDUCAUSE 2001 30

Affective Area – continued

Post-course satisfaction survey:

– Overall I would rate the Warm-Up system as Excellent (10) – Very good (7.5) – Satisfactory (5) – poor (2.5) – Very poor (0).

Result (overall rating): 8.21

10/30/01 EDUCAUSE 2001 31

Affective Area – continued

Are Warm-Ups a good idea? Why?– “The Warm-Ups are OK. They seem to help us

think before lecture into the main points that will be discussed, so that we are not that lost when we walk into class.”

– “I think the Warm-Ups are a good idea because without them most students would not even look at the material before class.”

– “I think that Warm-Ups are a good idea in that it makes us get in the book and search things out.”

10/30/01 EDUCAUSE 2001 32

Affective Area – continued

Are Warm-Ups a good idea? Why?– “Yes! They force me to look at the new stuff a

little before coming to class, and help me get a general idea on if I need to pay special attention in class.”

– “Yeah! I think the Warm-Ups are a very good idea. It helps me to get a head start on understanding the new materials better.”

10/30/01 EDUCAUSE 2001 33

Affective Area - continued

Improvement in Students’ Study Habits:– Discourage cramming– Encourage daily work

Practice Skills:– Analytic thinking– Estimation skills– Ill-defined problems– Communication and teamwork

10/30/01 EDUCAUSE 2001 34

JiTT Assessment:References and Resources

Gavrin & Novak (1997)Novak, Patterson, Gavrin, & Enger (1998)Novak, Patterson, Gavrin, Christian, (1999) Department of Biology (June 2001) Annual ReportDepartment of Physics (July 1999, June 2001) Annual ReportsHake, R. R. (October 2001) Evaluation of Progress on the WebScience (JiTT) Project at IUPUI

10/30/01 EDUCAUSE 2001 35

Conclusion

There is no single way to “do JiTT”JiTT works with a wide variety of courses, students, schedules, etc.Affective assessment: Excellent results so farCognitive assessment: Good – Additional research and/or analysis of relevant

data is needed to establish statistical and/or practical significance of findings.

10/30/01 EDUCAUSE 2001 36

Question and Answer Session

Question and Answer Session