READING THE TEA LEAVES:IDEA RESULTS AND STUDENT

LEARNING

GOALS AND OBJECTIVES Goals: 1. To help faculty respond critically to

student generated course evaluation; 2. To help faculty improve student learning.

Outcomes: Faculty participants will Better understand the relationship between chosen

objectives and the IDEA Diagnostic Form Report; Recognize the connection between objectives and

classroom teaching methods; Have access to resources that will help them achieve

greater student learning; Understand additional evaluation tools.

ALL INFORMATION IN THIS PRESENTATION IS CREDITED TO THE IDEA CENTER

[A]reflective analysis on the part of the instructor can be instrumental in promoting instructional improvement (Braskamp & Ory, 1994). In addition, the instructor is the only person who can supply certain kinds of information needed by those charged with making such evaluations, including information about course objectives; readings, assignments, and other learning activities; the creation of instructional materials or learning opportunities; procedures for appraising student achievement; results of, and course modifications based on, classroom research and other faculty efforts directed to improving instructional skills.

LEARNING GOALS

You will choose your IDEA Objectives after you have established your Course Learning Goals.

Your course learning goals must be tied to your Major, College, and Degree goals.

After those goals are set, you move to IDEA Objectives. The IDEA form measures student perception of your teaching specific to the goals identified for the course.

CHOOSING OBJECTIVES Most instructors will be unable to

address more than 3-5 objectives Select with these three questions:

Is this a significant part of the course? Do I do something specific to help the

students accomplish this objective? Does the student’s progress on this

objective affect his or her grade?

ESSENTIAL/IMPORTANT AND THE SCORE YOU GET

WHAT DO THE OBJECTIVES MEAN?IDEA’S NOTE ON SELECTING OBJECTIVES

1. Gaining factual knowledge (terminology, classifications, methods, trends)

2. Learning fundamental principles, generalizations, or theories

These objectives are mainly cognitive in nature as distinguished from affective objectives, which focus on feelings or attitudes. They are closely related. Both are concerned with the acquisition of information or knowledge.

Stresses learning at its most basic level.

APPLYING WHAT HAS BEEN LEARNED TO SOLVE PROBLEMS, MAKE DECISIONS, OR PERFORM

SPECIALIZED FUNCTIONS:3. Learning to apply course materials (to improve rational

thinking, problem solving and decisions) 4. Developing specific skills, competencies and points of view

needed by professionals in the field most closely related to this course

The first of these emphasizes applications of principles, theories, and concepts to solve a problem or arrive at a decision. Course materials are employed to develop this general intellectual skill. It is appropriate in courses where applications are intended to develop problem-solving skills--analysis, synthesis, and evaluation.

Only in courses where a primary goal is to develop a generalized skill in applying course materials to concerns or problems should Objective 3 be selected as "Essential" or "Important."

EXPRESSIVENESS6. Developing creative capacities (writing, inventing,

designing, performing in art, music drama, etc.) 8. Developing skills in expressing oneself orally or in writing Creativity connotes originality, imagination, and

expressiveness. While it is often associated with the fine arts and literature, it is often relevant to aspects of science, engineering, and other fields where design, research, and innovation are required.

For this objective to be classified as “Essential” or “Important,” it should be a significant emphasis in the course and there should be specific instruction or assignments designed to promote its development.

INTELLECTUAL DEVELOPMENT7. Gaining a broader understanding and appreciation of

intellectual-cultural activity (music, science, literature, etc.) 10. Developing a clearer understanding of, and commitment

to, personal values. 11. Learning to analyze and critically evaluate ideas,

arguments, and points of view. This objective should be selected as important or essential

if the focus of the course is on broadening intellectual background and increasing the breadth of students’ interests and appreciations. Such courses are usually directed to non-majors, although they may serve as introductory courses for majors as well.

LIFELONG LEARNING9. Learning how to find and use resources for

answering questions or solving problems. 12. Acquiring an interest in learning more by

asking questions and seeking answers. This objective is especially appropriate for

courses in which students have already acquired sufficient background information and skill to permit them to identify the trends or unresolved problems that are likely to form the basis for future advances in the field.

TEAM SKILLS5. Acquiring skills in working with others as a member

of a team. Instructors who emphasize the development of team

skills will note the overlap with several other objectives included on the IDEA list, including Objective 8 Communication Skills, Objective 3 Applications for Problem Solving, and Objective 6 Creative Capacities. If the expectation is that all of these skills will be developed, together with skills involving effective interpersonal relationships, then this objective should be identified as “Important” or “Essential.”

LET’S CHOOSE SOME OBJECTIVES

DURING THE SEMESTER Post Objectives to your syllabus; Add objectives on each assignment and

exam; Emphasize the connection between the

class discussion and the objectives and learning outcomes;

PART 2: THE RESULTS

AN INTERPRETIVE GUIDECLICK FOR IDEA NOTES

An overall index of teaching effectiveness (PRO=Progress on Relevant Objectives) combines ratings of progress on the objectives identified by the instructor as Important (weighted “1”) or Essential (weighted “2”) The IDEA Center regards this as its single best estimate of teaching effectiveness.

Two additional overall measures of teaching effectiveness are shown on the report. These are the average ratings of two items using a 5-point scale

1. Overall, I rate this instructor an excellent teacher. 2. Overall, I rate this course as excellent. As an index of teaching effectiveness, the average of these two

ratings is commonly regarded as about equal in value to the “Progress on Relevant Objectives” index described above.

IDEA DIAGNOSTIC FORM REPORT:PAGE 1

Your report compares your average ratings to results for three different groups of classes. The first comparison group is with all classes in the standard IDEA database, and is always reported. The other two are reported only if enough classes were available to provide a stable basis for comparison. These consist of (1) all classes in the same discipline as the class in question and (2) all classes at your institution.

Pay attention to the 5 point scale numbers. The converted numbers are useful only insofar as they measure your results against other faculty. They are not necessarily a measure of progress on relevant objectives.

ENGLISH 2325: FALL 2009

DIAGNOSTIC REPORT--PAGE 2 The first portion of Page 2 lists the 12 objectives included on

the IDEA form and summarizes student ratings on those you selected as either Important or Essential. The main purpose is to help you focus your improvement efforts.

Note the % does not include student’s who respond with a 3. Students described the class by comparing it to other classes

they have taken in terms of (1) amount of reading, (2) amount of work in non-reading assignments, and (3) difficulty. Average ratings are compared with “All classes” in the IDEA database.

Students described their motivation by making self-ratings on the three items listed at the bottom of Page 2. These characteristics have been found to impact student ratings of progress.

DIAGNOSTIC REPORT—PAGE 2

DIAGNOSTIC REPORT--PAGE 3 The main purpose of instruction is to facilitate progress on objectives that

the instructor selects as Important or Essential. The IDEA Center has conducted many studies that relate ratings on each of

these “methods” to ratings of student progress on the 12 learning objectives. Through these studies, 7-10 methods that are most closely related to progress on each of the 12 objectives for classes of different sizes have been identified. Although there is some overlap, there are distinct differences in the methods that facilitate progress on the 12 objectives; there are also some differences among class sizes. The objectives for which a given method is especially helpful in promoting learning are identified in the column titled ”Relevant to Objectives.” The last column proposes an action for you to take, depending on the relevance of the item and how students rated it. If the rating for a relevant item was well above the IDEA average, it is described as a “Strength to retain”; if the rating was well below average, you are advised to “Consider increasing use”; and if it was in the average range, it is suggested that you “Retain current use or consider increasing.”

DIAGNOSTIC REPORT--PAGE 3

CONNECT PAGE 3 TO PAGE 2 Retain current use or consider

increasing. Why? Only 59% of the students marked

that I “inspired” them to set and achieve goals. If I focus on improving that element, I will improve their progress on all selected relevant objectives.

IDEA recommends: IDEA Item #15

EXAMPLE 2 To improve their progress on Objective

27 and 31, I need to focus some attention on asking students to share ideas.

IDEA Recommends: IDEA Item #16 What I find, based on IDEA data, is the

two issues are linked.

DIAGNOSTIC REPORT—PAGE 4 IDEA tells us that page 4 is for “those who

want to confirm scores reported on pages 1-3. We can also use page 4 to determine if our

Objectives correlate to the student experience.

I’m surprised my students focused on Developing creative capacities. It’s not marked as an Objective, but it is worth my time to examine the course to see why they saw it as important.

DIAGNOSTIC REPORT--PAGE 4

HOW TO IMPROVE?

IDEA NOTES ON LEARNING (CLICK TO LINK TO IDEA NOTES)

IDEA NOTES ON INSTRUCTION(CLICK TO LINK TO IDEA NOTES)

ADDITIONAL EVALUATION TOOLS Mid-semester evaluations (coming soon)

Find out in March so you prevent it by May; Short Answer Response Questions

Ask your students specific questions related to course organization, content, texts, etc.

Small group analysis Colleague or CITR Director can visit a class

to discuss progress on objectives and learning outcomes.

THE GOAL IS TO RESPOND CRITICALLY TO THE NUMBERS, NOT USE THE NUMBERS TO BE CRITICAL