OATS: Georgia Tech’s Online Assessment Tracking System

SAIR 2005 ConferenceCharleston, SC October 24, 2005

Joseph HoeyJon GordonOffice of Assessment, Georgia Tech

OATS Presentation Outline

Development of OATS to document Institutional EffectivenessDescription of OATS features OATS structure and elementsIssues encountered, solutions generatedPath forwardQuestions and discussion

Documenting Assessment With a Web-Based System

Web-based documentation of assessment for demonstrating progress towards Institutional Effectiveness is an option, not a requirement – we have found it very useful at Georgia Tech.Georgia Tech’s approach to documenting systematic assessment processes is our Online Assessment Tracking System (OATS).Similar systems have been developed by other institutions and commercial vendors.

OATS: Background Annual Assessment Update concept was generated by GT unit coordinators in 1998 to document Georgia Tech’s responsiveness to SACS recommendations re: assessment of student learning.Requests received to move to online environmentOnline process provides structure, formalizes best practices in assessment, serves as a cross-campus communication vehicle, and thus facilitates demonstration of compliance. Having an online documentation system proved to be a key feature in Georgia Tech’s compliance review with the Principles of Accreditation in 2004.

Deciding What to Document and At What Level

We document and track assessment at the degree program levelFor each degree program, we document: Faculty expectations for student learning How these expectations are defined in

practice How these expectations are assessed What results are gained (summary level) Based on results gained, what actions are

being taken

Old Paper-Based Annual Assessment Update Structure

Three open-ended sections: What did you look at? What did you find? What did you do about it?

Submission as MS Word or .pdf documents

Web-Based Annual Assessment Update Structure: OATS

Feature ComparisonPaper-Based Method

Many different formats Hard copy only Difficult to track progress

over time Flexibility (but no

consistency across Institute)

Difficult to provide feedback internally and to facilitate institutional sharing of good practices

Online Method Consistent format Database storage Ability to track progress

over time Flexibility maintained Process facilitates

accreditation e-review Easier to provide

feedback; facilitates institutional sharing

OATS: Integrates Assessment and Annual Reporting

Annual Assessment Update now done online via OATS.Becomes a longitudinal record of assessment and changes to each instructional program.Ideally, can be used to satisfy multiple internal and external information and accreditation needs.

OATS Application: Features

Includes user id/password login and Georgia Tech authentication Web accessible from any location; software plug-in required for editing capability Defined format structure—Objectives/Outcomes, Methods, Results, and Actions/ImpactAllows posting of formatted text (tables, charts, etc.)Default annual rollover feature (except for assessment results and actions taken) saves timeAssessment Updates due: November 1 each year

OATS Application: Features

Archiving of all past submissions by degree program, Print-preview feature that produces a laddered report of objectives, methods, results, and actions taken, Notes feature that permits the storage of submission notes and commentary that won’t be “seen” officially; could be notes to colleagues Feedback feature is built in that permits units to obtain formative feedback on their update prior to final submission. College and Institute-level views, whereby deans are able at a click to view the online assessment update submissions of all units within their colleges.

Description of OATS Elements

Main MenuProgram MenuUnit Summary SectionObjectives/OutcomesMethodsResultsActionsPrint Preview

OATS Main Menu

Includes links to instructions and changes in OATSLink to administrative page for those with administrative accessSelections for: editing current updates, viewing current updates (print preview format

only), and viewing updates from previous years (print

preview format for 2003; .pdf for earlier years)

Main Menu: Current Year and History

OATS Program Menu

Searchable by CollegeIncludes enumeration of objectives/outcomes, methods, results, actions/impacts and date last modifiedIndicates status/progress towards completionOptions to combine programs, view notes, edit individual update, and delete update (“Are you sure?” screen appears)Notes feature can be used as information storage area by user (“here’s why we did this last year…”) or for feedback from Office of Assessment

College Level: Ivan Allen College

Unit Summary Section

Summary section of each OATS update includes portions for program purpose, responsibility for implementation, and operational (programmatic) objectivesPrograms use this section to describe the program purpose, set the context, describe operational objectives for the program, and to provide a brief summary of overall direction.

Objectives/Outcomes - 1

We ask programs to limit OATS updates to 3-8 most important student learning objectives/outcomes faculty expect from program graduates. Engineering undergraduate programs:

prioritize and develop a rotational timeline for assessing ABET EC 2000 program outcomes – don’t try to assess all of them each year.

Objectives/Outcomes - 2

For each objective/outcome, we ask programs to operationally define the activities and competencies to be assessed. What will students actually demonstrate? Describe what is to be achieved.State in terms of expected behaviors.Measurable.Aggregate (group) level – not individual.

School Level: History, Technology & Society

Assessment Methods - 1

We ask programs to specify the method by which student competencies will be assessed (presentations, embedded exam questions, projects, etc.)Optional: Indicate location in curriculum.Optional: Specify performance standards to be achieved.

Assessment Methods - 2We remind programs that the same method may be used to cover more than one learning objective/outcome. e.g., rubrics or checklists developed to assess

senior design projects might be used to assess technical skills, teamwork, and communication skills.

Consider triangulation (multiple methods) to increase reliability of measurement.Consider the reliability, validity, and quality of instrumentation. Will we be willing to trust the results?

School Level: History, Technology & Society

Assessment Results

Succinctly summarize assessment results found e.g., “95% achieved passing scores

on relevant portion of Fundamentals of Engineering Exam.”

Provide brief analysis and interpretation of results.

School Level: History, Technology & Society

Action: Use of Assessment Results

Include a discussion of actions taken or pending in context of the instructional program based on consideration of results obtained.Refer to instrumentation used if URL is available.Give timetable for completion of actions. e.g., “The school has redesigned our

undergraduate structures sequence and hired an additional faculty to teach the additional course beginning next semester.”

Print Preview

OATS includes an option to create a formatted report for each assessment updateSummary section of update appears as distinct from learning outcomes sectionUpdate report is “laddered” such that methods, results, and actions are indentedPrint preview may be saved as an html document and email to others for review

Degree Program Level: BS in HTS

Issues Encountered

Who is really the customer?Multiple operating systems and levels of securityIssues with plug-in software, supported applicationsTraining needsOperational roles vary across unitsVarying levels of administrative enthusiasm and involvementTree vs. multidimensional structure

Other Issues Encountered

Multiple evaluations/multiple cultures problem: Many external reviews and

accountability demands, all with unique features: SACS, Board of Regents program review; specialized accreditation (especially ABET)

Culture of disciplines demands custom solutions (especially ABET)

Path Forward - 1

Process to move towards integrated, customizable, culturally acceptable solution Interviews with key players and users to

assess needs during Summer 2005 Task force assigned to develop conceptual

recommendations by November 2005 Technical working group will write

specifications by March 2006 Programming and testing completed by

August 2006 Next OATS version operational for November

2006 OATS

Path Forward - 2

Desirable features identified so far: Customized “shell” views selected at login for each

relevant specialized accreditation body (ABET, CAC, AACSB, NAAB, NASAD, etc.)

Ability to link to representative courses in curricula Ability to view OATS entries by objectives/outcomes,

by assessment methods used, and by courses in major

Ability to import IR data into unit summary Ability to import assessment data from GT online

assessment data query tool directly into updates Ability to link across disciplines for project or

program evaluations Ability to do all of the above by yesterday with no

faculty effort

Questions?

Contacts: Joseph Hoey,

joseph.hoey@oars.gatech.edu or telephone 404.894.0510

Jon Gordon, jon.gordon@oars.gatech.edu or telephone 404.385. 1419