PICA Response RatesMark Troy – Data and Research Services

– metroy@tamu.edu

2

How low are they? How concerned should faculty and

administration be about low rates? Can response rates be improved?

Three issues about response rates in online evaluations

3

What are the response rates?

4

PICA average ◦ 50%-60%

Paper average ◦ 70%-80%

100% response rate◦ Less than 10% of courses, whether evaluated with

paper forms or PICA achieve 100% responses Extremely low response rate

◦ Less than 10% of courses, whether evaluated with paper forms or PICA have response rates lower than 10%

Response rates: paper vs. PICA

5

PICA Response Rate by Department

RPTS

ECEN

MEEN

GEOG

BAEN

ANTH

HLKN

CLEN

CSCE

NUENBM

ENEA

HRALE

CTLA

CWFSC BU

SHMART

MARA

MARSMAR

B0

10

20

30

40

50

60

70

80

90

PICA Mean Response Rate, Spring 2012Mean response rates for departments with 10 or more courses evaluated in PICATAMU departments are in red. (Total courses = 2597)TAMUG departments are in green. (Total courses = 420)The mean response rate for all courses evaluated in PICA, Spring 2012 = 50% (red line)

6

How concerned should faculty and administration be about low response rates?

7

Non-response bias◦ The bias that arises when students who do not

respond have different course experiences than those who do respond.

Random effects◦ If the response rate is low, the mean is

susceptible to the influence of extreme scores, either high or low.

The problems with low response rate.

8

Only those who really like you and those who really hate you will do the evaluations

You will get responses only from the students who like you.

Only the students who hate you will bother to respond.

The students who really hate you will choose not to play the game at all.

Common expressions of non-response bias

9

If only those who like you and those who hate you respond, the means might not change (assuming equal numbers of both groups), but the shape of the distribution and the variance of the scores would change.

If only those who like you respond, the mean should increase and the distribution and variance would change.

If only those who hate you respond, the mean should decrease and the distribution and variance would change.

The effects of non-response bias

10

Are there consistent mean differences between paper and online ratings?

No significant mean differences were observed.

An upward trend is visible in the chart, but that began before the switch from paper to PICA.

11

Are there consistent mean differences between paper and online ratings?

07a 07c 08a 08c 09a 09c0

1

2

3

4

5

6

MeanMean + 1SDMean - 1SD

Biology Department

Paper PICA

No mean differences were observed.

The standard deviations did not change significantly after the switch to PICA.

12

Are there distributional differences? Same department, consecutive semesters.

A B C D E0

10

20

30

40

50

60

70

PICA (43%)Paper (81%)

Overall, this was a good course.

Strongly agree Agree Undecided Disagree Strongly disagree

Percent of students responding

13

TAMUG Response Distribution

5 4 3 2 10.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Fall 11 (85%)Spring 12 (35%)

Fall 2011- paperSpring 2012 - PICA

14

A comparison of courses taught by the same instructor in consecutive semesters showed that half of the ratings went up after the switch to PICA and half went down.

Overall, there was no significant difference in means of paper evaluations compared to PICA evaluations.

Significant mean differences were observed in 3 out of 70 courses (which would be expected by chance alone.)

The three significant differences were observed in courses with less than 30% response rate in PICA.

What differences will instructors be able to see in their ratings after switching to PICA?

15

The correlation between response rate and class size for paper administration = -.161 (Fall 2009)

The correlation between response rate and class size for PICA administration = -.056 (Fall 2009)

Conclusion: Although the relationship between class size and response rate is weak, there is a greater tendency for response rates to decrease as class size increases with paper administration than with PICA.

Class Size and Response Rates

16

Most differences between means can be attributed to chance variation and not to the method of evaluation.

Response rate does not appear to have an impact on means.

EXCEPTION: Extreme scores, whether high or low, exert greater influence on the mean when the number of responses is low.◦ Equally true for 100% response rate from a class

of ten as for 10% response rate from a class of 100.

Random effects

17

1. Non-response bias is a systematic effect that, if present, would appear as a difference in mean ratings, or as an increase in the variance of the ratings, and as a change in the distributions of the scores. Such differences are not seen. (Although that does not mean the non-response bias doesn’t exist.)

2. Random effects. In any evaluation, some ratings will be higher than others and some will be lower. The more responses, the less influence the extremes, either high or low, will have on the mean. A low response rate allows an unusually high or low rating to pull the average in its direction, therefore higher response rates are better than lower response rates. These random effects are often seen with low response rates.

Summary, the Effect of Low Response Rate

18

YES The average response rate per department

tends to increase over time as faculty and students attain familiarity with the system.

There are some strategies faculty can use to increase response rate.

Can response rates be increased?

19

Motivation◦ Students need to be motivated to do online

evaluations◦ Students do not need to be motivated to do paper

evaluations

Attendance◦ Students need to be in class to do paper

evaluations◦ Students do not need to be in class to do online

evaluations

The big differences between online and paper evaluations

20

Research on student evaluations in higher education shows:◦ Students’ most desired outcome from evaluations is

improvement in teaching.◦ Students’ second desired outcome is improvement in

course content.◦ Students’ motivation to submit evaluations is severely

impacted by their expectation that the faculty and administration pay attention to the evaluations.

The bottom line is that students submit evaluations if they believe their opinions will be valued and considered.

Students’ motivations to do evaluations

Getting Students to Respond: What works

21

Survey conducted of a random sample of students in classes that used PICA, Fall 2008.95 students submitted an appraisal online (Response), 94 students did not submit an appraisal although requested to do so (Non-response).

22

Receiving an invitation from MARS to do the evaluation has no impact on the response rate.

By a factor of 3 to 1, students are more likely to submit an evaluation:◦ If the request comes from their instructor.◦ If their instructor discusses the importance of the

evaluation.◦ If their instructor tells how the evaluations have been

used or will be used to improve the course. Incentives are less effective than discussing the

importance and the use of the evaluations.

Observations

23

Why students fail to submit online evaluations

A survey of students who had not submitted evaluations online in PICA, though requested to do so found the following reasons for not submitting.◦ Forgot — 48%◦ Missed the deadline — 26%◦ No other reason was given by more than 10%

The bottom line is that frequent reminders are important.

24

Texas A&M faculty have an opportunity to conduct mid-term evaluations (Actually fifth-week evaluations) in PICA

The purpose of the mid-terms is to provide formative information for improving the course at a point that changes are possible.

An examination of response rates at the end of the term shows that courses for which there was a mid-term evaluation have on average a 10% higher response rate than other courses on end-of-term evaluations.

The likely explanation for the effect on response rate is that faculty who do mid-terms can demonstrate to students that their opinions are considered.

Mid-term evaluations

25

Incentives appear to be not as effective a motivator as the intrinsic motivation to help improve teaching and the course.

A token incentive, however, can reinforce intrinsic motivation by communicating the importance of the evaluation.

Incentives

26

The Faculty Senate (Resolution FS 27.122) opposes granting academic credit for evaluations because such credit is not related to any learning outcomes, so it could have the effect that two students who learned the same amount would receive different grades.

Some faculty offer a token incentive to the entire class if some threshold of response rate is reached, thereby communicating the importance of the evaluation, but not disadvantaging a student who chooses not to do an evaluation.

Incentives, continued

27

Use mid-term evaluations to help students

see how to give feedback and what feedback is useful. Show students that their feedback

is acted upon in a positive way.

Discuss the importance of the evaluation and the use of the results in improving the

course and your teaching.

Remind students

frequently. Reminders are most effective

when they come from the

faculty.

3 Steps To Increasing Response Rate