Developing a feedback Developing a feedback questionnaire:questionnaire:

Principles and stepsPrinciples and steps

Workshop for NHS staff 28 Dec 1999 (Tuesday)

Kam-Por Kwan, EDU2766 6287

etkpkwan@polyu.edu.hk

2

Workshop outlineWorkshop outline

Why use a feedback questionnaire?

How to develop a feedback questionnaire that give useful and truthful information?

– What are the common problems?

– How to write good evaluation items?

– How to determine if the questionnaire is valid and reliable?

How to interpret student feedback in a meaningful way?

Developing a student feedback questionnaire on clinical experience

3

Why use a feedback Why use a feedback questionnaire?questionnaire? Economical to administer to the whole group, both in

terms of time and effort Allow anonymity of responses Allow respondents to control own pace of response

but

• Less chance to probe for clarification

• Emphasis on evaluator’s rather than respondent’s perspectives

• Reliability, validity, and usefulness depends on items included

4

Common problemsCommon problems

Feedback questionnaires often fail to provides true and useful information because:– the items are constructed in an ad hoc basis without

any theoretical framework behind

– they ask about things that the (student) raters cannot validly comment on

– the items are ambiguous to (student) raters and/or difficult to understand / interpret

– the interpretations of the items are not clear

– the items are too standardised to be useful

– respondents are not motivated to complete it seriously

5

A systematic approachA systematic approach

7 steps to developing a questionnaire:– determining the focus of the evaluation

– identifying all underlying dimensions and sub-dimensions involved

– drafting questionnaire items

– designing questionnaire: instructions and sequencing, etc.

– pilot testing the questionnaire

– revising questionnaire and items

– implementing the questionnaire

6

Clinical

supervision

Clear guidelines

Feedback

Support

Learning

Identifying focus and dimensionsIdentifying focus and dimensions

Providing useful comments on what and how to improve

Providing regular feedback onstudents’ clinical performance

7

Examining the dimensionsExamining the dimensions

Task 1 – Examine the draft questionnaire and identify

for each item:• the underlying dimension that it pertains to

measure, and• what kind of variable (presage, process, or

product) is being measured.

– What other dimensions or variables do you think should be included in the questionnaire?

8

Problematic itemsProblematic items Task 2: What’s wrong with the items?

In groups, discuss the problems of including the following items in a student feedback questionnaire.

– The teacher seemed to have an up-to-date knowledge of evidence-based nursing practice.

– The teacher worked hard to demonstrate clearly to me the proper skills of history-taking.

– My progress was a major concern of the teacher.

9

More problematic itemsMore problematic items

– I was provided with informed and constructive feedback on my performance by the teacher immediately after my clinical practice.

– I found it difficult to apply the theory I learned at university to my clinical practice.

– Every student was encouraged to participate in class discussions.

– The teacher did not discourage me from using techniques that are not evidence-based.

– Appropriate computer technology and AV aids were used by the teacher to facilitate learning.

10

Principles of item writing (1)Principles of item writing (1)

Use simple English and simple sentence structure

Avoid questions that the intended respondents do not have the knowledge to comment on

Avoid ambiguous questions or wordings that may have alternative interpretations

Avoid double-barreled questions (items containing more than one ideas)

11

Principles of item writing (2)Principles of item writing (2)

Avoid unnecessary jargons that may not be understood by the intended respondents

Avoid words like “all the time”, “never”, “every”, ...

Avoid double negatives

Avoid questions with unwarranted underlying assumptions

12

Using open-ended questions Using open-ended questions

Limitations of ratings:– reflect evaluator’s rather than respondents’

perspectives– suggest whether improvements are needed, but

not why or how– give a false sense of objectivity and precision

Open-ended questions– allow respondents’ perspectives to emerge– offer chances for respondents to clarify personal

meanings and suggest changes

13

Optional questionsOptional questions Standardised questions:

– allow comparisons across units or teachers BUT

– cannot cater for individual needs

Optional questions: – allow users to collect information on aspects

specific to the individual units or teachers– useful for improvement purposes

14

Revising the draft questionnaireRevising the draft questionnaire

Task 3In groups, suggest how the draft questionnaire might be further modified to make it more useful and valid. You might consider:

– adding new items /deleting redundant items

– rewording the items as needed to make their meaning clearer and less ambiguous

– inserting open-ended questions

– the possibility of allowing optional questions to be included

15

Good designGood design

as short and sharp as possible (a few short questionnaires at different points may be better than a long one at the end)

appeals to the intended respondents with clear instruction questions arranged in good psychological order,

from general to more specific attractive and neat in appearance clearly duplicated / printed easy to code and interpret

16

Validity and Reliability (1)Validity and Reliability (1)

Validity

– Does the questionnaire measure what it is supposed to measure?

– Do the items together measure the most significant aspects of the evaluation question?

– Improving validity by:

• judgment of a panel of experts

• pilot testing on a sample of intended respondents

• relating to theory of teaching

17

Validity and Reliability (2)Validity and Reliability (2)

Reliability– Does the questionnaire give a consistent

results of it is measuring?– Do the items yield results that agree with

each other, and are they consistent over time?

– Improving reliability by establishing the:• internal reliability of the instrument, scales, and

sub-scales• test-retest reliability

18

Student feedback: what research Student feedback: what research sayssays Quite reliable and consistent Reasonably valid Relatively uncontaminated by variables seen as

sources of potential bias Useful for a number of purposes

• An ‘imperfect’ measure of teaching• Must be interpreted in contexts• Useful insofar as one source of information• Can be abused if not interpreted properly

BUT

19

Nature of student feedbackNature of student feedback

Subjective perceptions

Based on what students have directly experienced

Influenced by their own characteristics such as prior knowledge, motivation, interest, etc.

Reflected students’ implicit theory of teaching and learning

20

Some pitfalls in interpretationSome pitfalls in interpretation

Treating student feedback as if it were a totally objective, precise, and truthful indicator of teaching

Over-interpreting small differences in ratings

Comparing ratings across units or teachers without considering the context

Ranking units/teachers by their total scores

21

Interpretation guidelinesInterpretation guidelines

Avoid over-interpreting small differences: only ‘crude’ judgements can be made

Focus on the relative strengths and weaknesses as reflect in the profile of ratings rather than the total scores

Interpret feedback in context: need to take into consideration the features of the centre and the students

Consider ratings from different classes, and over a number of years

Need to check student feedback against other sources of evidence

22

Some final wordsSome final words

Student feedback is a useful source of information, not a verdict

Student feedback cannot replace professional judgment of the teacher

Teacher’s self reflection on the feedback collected is the key to improvement