NOTICE

This presentation and its contents are the jointly held property of the three coauthors and The Board of Regents of The University of Texas System.

Please do not cite any of the materials presented herein without appropriately referencing the work per the information on the next slide.

My colleagues and I welcome your feedback; send your e-mail to djsilva@uta.edu.

If you plan to use any part of this presentation, please let us know. Thank you.

-David Silva, Araya Maurice, David Purkiss

To move through the presentation, press the “enter” key.

The Development and Use of UT Arlington’s Active Learning Assessment

Tool (ALAT)

Araya Maurice*, David Purkiss*, and David J. Silva^*Office of Institutional Research, Planning and Effectiveness / ^Office of

the Provost

Southern Association of Institutional Research Annual Conference

October, 20th 2008

Active Learning at UT Arlington

…puts the student at the center of the learning process, making him/her a partner in discovery, not a passive receiver of information.

…requires students to interact with and integrate course material by reading, writing, discussing, problem-solving, investigating, reflecting, and engaging in such higher order thinking tasks ...

…draws upon such teaching strategies as question-and-answer sessions, short in-class writing exercises, student research, internships and community service, team learning, and problem-based learning. 3

What is the ALAT?

“Active Learning Assessment Tool”

Closely modeled after the Active Learning Inventory Tool (ALIT) developed at Northeastern University (School of Pharmacy)

“[A]n ‘inventory tool’ to characterize the use of Active Learning techniques by faculty” in Pharmacy classes (Devlin et al.)

Tremendous thanks to colleagues at Northeastern University, including Dr. Richard Porter

4

Comparing ALIT and ALAT

NortheasternALIT

UT ArlingtonALAT

Impetus Accreditation Council for Pharmacy Education (ACPE) and American Association of Colleges of Pharmacy (AACP)

Quality Enhancement Plan (QEP) as formulated for reaffirmation by the Southern Association of Colleges and Schools (SACS)

Scope School of Pharmacy(with plans to generalize)

University-wide

Personnel

Multiple trained faculty observers

Single trained student or staff observer with periodic second observer (staff) to verify / calibrate

Coding Scheme

22 activity types3 levels of complexity

17 activity types5 levels of complexity6 interactive domains

5

Active Learning Inventory ToolNortheastern University

6

“Active Learning: Pathways to Higher Order Thinking at UT Arlington”

12 faculty-designed / committee-selected pilot projectsTime Frame = 3 years (2007-’08, ‘08-’09, ‘09-’10)Differing Course Formats

Large lecture class Capstone Seminar

Multiple Disciplines Architecture, Business, Education, Engineering (3), History, Honors,

Library / English, Mathematics, Nursing, Political ScienceVarying Interventions

Personal Response Systems (“clickers”) Blogging / On-Line Community-Building Fieldwork On-line Problem Solving (with immediate feedback) Design & Production Projects

7

QEP Measurements

Student LevelAcademic BackgroundDemographic InformationSelf-Reported Engagement + AttendancePerformance

Critical Thinking Test (CTT) Evaluations from a Criterion-Based Course Rubric

Course LevelAggregated student-level dataCourse Descriptives including indications of “control” vs.

“active learning”IDEAFocus Group TranscriptsActive Learning Assessments (ALAT) 8

UT Arlington ALAT: First Iteration

1. Student offers unsolicited comment.2. Student responds to question.3. Student asks question.4. …5. …6. …7. …8. … . .22. …23. Students teach each other in small groups.24. Students draw diagrams which interrelate various concepts.25. Student exercise in problem solving…26. Student problem solve in small groups. 9

UT Arlington ALAT: First Iteration

SPEAKWRITESOLVE PROBLEMORGANIZEPLAY“CLICKER”QUIZPRESENTATION

10

UT Arlington ALAT: Second Iteration

11

The Deming Cycle

12

The Assessment Cycle

13

UT Arlington ALAT: Third Iteration

14

UT Arlington ALAT: Current Version (2008-09)

15

ALAT Coding, Complexity Level and Duration Categories

Activity (ALAT Code) Complexity Level

No Active Learning Activities 0

Active Learning Activities: 1, 5, 12 1

Active Learning Activities: 4, 8, 11, 13, 15, 17 2

Active Learning Activities: 2, 6, 9 3

Active Learning Activities: 3, 7, 10, 14, 16 4

Duration Category

Active Learning Activity Lasted Less Than 1 Minute

1

Active Learning Activity Lasted Between 1 and 5 Minutes

2

Active Learning Activity Lasted More Than 5 Minutes

3

16

Data Entry: Active Learning Info System

17

ALAT Reliability Issues

Inter-rater: Performed for the first two semesters of useAssessment Specialists performed concurrent ALAT

observations and then compared results between raters

ALAT Comparisons (visual): Good match on codes; some variation on number of occurrences

Intra-rater: Beginning the third semester of useShifted focus to ongoing training and norming of

ALAT observers to promote rater consistency

…Mathematical Model? 18

Data: Active Learning Indices for 2007-08

Stratum Course Active Control

Introductory Engineering (Mixed) 0.84 0.72

History 0.68 0.61

Electrical Engineering 0.12 0.10

Political Science 0.18 0.33

Seminar Honors (Sophomore) 2.16 (F07), 1.09 (S08)

Honors (Senior) 1.42 (F07), 0.90 (S08)

Capstone Mathematics 0.45 (F07)

Nursing 1.57 (F07), 1.83 (S08)

Utility

Measurement of Classroom Activity for IR purposes

Providing instructional faculty with feedbackComparing perception with reality

Realigning as appropriate

Creating an inventory of praxis for chairs and deans

Providing information to central administration onRange of teaching praxis campus-wide

Generalizations as stratified by college, course level, course format

Directions for future professional development needs(specific to teaching)

20

Caveats

Measures only what happens in the classroom

Makes no direct claims regarding learning per se

21

Questions / Comments

Araya Maurice, Ph.D.Assistant Director for Assessment of Student Successamaurice@uta.edu

David Purkiss, M.A.Undergraduate Research and Assessment Specialistdpurkiss@uta.edu

David J. Silva, Ph.D.Vice Provost for Academic Affairsdjsilva@uta.edu

22