AMSc Research Methods Research approach IV: Experimental [1] Jane Reid [email protected].

AMSc Research MethodsResearch approach IV: Experimental [1]

Jane Reid

[email protected]

Jane Reid, AMSc RM, QMUL, 4/12/01

2

Overview of lectures

• Tuesday 4th December [Jane]– Lecture on experimental design

– Introduction to experimental design exercise

• Thursday 6th December [Jane]– Experimental design presentations

• Tuesday 11th December [Marie-Luce]– Lecture on statistical testing

– Introduction to statistical testing exercise

• Thursday 13th December [Marie-Luce]– Statistical testing presentations


3

Background

• Aim: to develop and/or test theories about human characteristics and behaviour

• Origins in experimental psychology

• Experimental work can be carried out at different stages of the system development process, e.g.– Design, i.e. to provide requirements capture information

– Implementation, i.e. for formative evaluation of prototypes

– Evaluation stage, i.e. for final evaluation, in terms of functionality or usability, of finished system


4

Scenario

• Imagine you wish to test 2 different schemes for helping young children to learn new vocabulary

• The 2 schemes are:– Cards with the individual words written on them (“W” cards”)

– Cards with the individual words and an illustrative picture on them (“W+P” cards)

• You want to examine how effective these cards are at different recall times of 2 minutes, 10 minutes and 30 minutes


5

Experimental hypotheses [1]

• A theory developed about a relationship between 2 or more factors is an experimental hypothesis

• Hypotheses are developed in pairings:– A hypothesis that predicts the presence of an effect or association

is an alternative hypothesis• E.g. The type of vocabulary scheme employed will have an effect on

the number of words recalled

– A hypothesis that predicts the absence of an effect or association is a null hypothesis

• E.g. The type of vocabulary scheme employed will have no effect on the number of words recalled


6

Experimental hypotheses [2]

• A hypothesis can predict an effect or association:– In one direction - a one-tailed hypothesis

• E.g. The W+P vocabulary scheme will result in more words being recalled than the W scheme (W+P > W)

– In either direction - a two-tailed hypothesis• E.g. The type of vocabulary scheme employed will result in a

different number of words being recalled (W+P > W or W > W+P, i.e. W <> W+P)


7

Hypothesis testing [1]

• An experiment is designed to answer one (or more) experimental hypothesis(es)

• A hypothesis cannot be proved; it can only be upheld on the basis of the evidence presented

• Dangers of hypothesis testing:– Type I error

• The null hypothesis is incorrectly rejected, i.e. the alternative hypothesis is incorrectly upheld

– E.g. finding that W+P > W, when really W = W+P


8

Hypothesis testing [2]

– Type II error• The null hypothesis is incorrectly upheld, i.e. the alternative

hypothesis is incorrectly rejected– E.g. finding that W = W+P when really W > W+P


9

Variables [1]

• An experimenter attempts to eliminate unwanted effects from confounding variables, i.e. factors other than those being investigated– E.g. age, family background, educational background, gender...

• Independent variables– Can sometimes be manipulated directly by the experimenter

• E.g. vocabulary scheme

– Can sometimes only be manipulated indirectly by assignment of participants

• E.g. gender


10

Variables [2]

– Each manipulation of an independent variable is an experimental condition

• E.g. W and W+P vocabulary schemes

• Dependent variables– These are the variables thought to be affected by changes in the

independent variables• E.g. number of words recalled after a set time

– The effect on the dependent variables is measured during the experiment


11

Experimental design [1]

• The basis for the experiment

• Differential designs:– Examination of the difference in effect of 2 or more experimental

conditions (of an independent variable) on a dependent variable• E.g. The effect of choosing the W or W+P vocabulary scheme on the

number of words recalled

• Correlational designs:– Examination of a possible association between 2 variables

• E.g. Any association between the ability to learn new vocabulary and the ability to spell


12

Experimental design [2]

– Not possible to form a hypothesis about which variables have an effect on which other variables, so:

• No concept of independent/dependent variables

• A cause-and-effect relationship does not always exist between the variables anyway


13

Designs with one independent variable [1]

• Using a control condition:– The 2 experimental conditions are the control condition and the

independent variable• E.g. a standard, existing vocabulary scheme using no cards

– The effect of the independent variable is measured against the baseline of the control condition results

• Using 2 experimental conditions:– The effects of the 2 conditions are measured against each other

• E.g. W and W+P vocabulary schemes


14

Designs with one independent variable [2]

• Using 3 or more experimental conditions:– Possible to examine trends across 3 or more conditions

• E.g. The effectiveness of a vocabulary scheme after different amounts of recall time: 2 minutes, 10 minutes, 30 minutes


15

Designs with 2 or more independent variables

• Each independent variable may have a separate effect - a main effect– E.g. effect of vocabulary scheme choice, effect of recall time

• There may also be an interaction effect due to the combination of 2 or more independent variables– E.g. One vocabulary scheme (W) may lead to better recall after a

short period of time (2 minutes) while the other (W+P) may lead to better recall after a longer period of time (10 minutes)

• Extension to 3 or more independent variables is possible, but such designs may quickly become complex– E.g. Include the effect of gender in the experiment


16

Participants [1]

• When different participants are assigned to each experimental condition, this is an unrelated or between-subjects design– E.g. Different children assigned to each vocabulary scheme

– Advantage: 2 closely related experimental conditions can be tested with different participants

– Disadvantage: possible bias in results due to unwanted effects from other variables


17

Participants [2]

• When the same participants are assigned to each experimental condition, this is a related or within-subjects design– E.g. The same children perform under all 3 recall conditions

– Advantage: no confounding variables; possible to study an effect over time

– Disadvantage: care needed in experimental design to avoid unwanted learning effects, e.g. by using counterbalancing


18

Participants [3]

• When different participants are assigned to each experimental condition, but matched on important characteristics, this is a matched subjects design, and is treated as a related subjects design– E.g. Children assigned to different groups are matched on age,

gender...

– Advantage: attempt to combine the advantages of the related and unrelated subjects designs

– Disadvantage: very difficult (impossible?) to identify and match on all significant variables


19

Data measurement

• For statistical analysis, data must be:– Quantitative

– Objective

• Levels of measurement:– Nominal - categories of data

– Ordinal - rank ordered data

– Interval - numerical data, measured on a continuous scale, with equal intervals between measurement points

– Ratio - interval data with an absolute zero


20

Measures

• For naturally quantitative data:– Results in interval or ratio data

– E.g. time, number of errors

• For qualitative data that can be measured in a quantitative way (or classified as such by the experimenter)– Results in ordinal data

– E.g. confidence, satisfaction

• For qualitative data– Results in nominal data

– E.g. preference for one interface over another


21

Instruments [1]

• Questionnaires:– Structured: closed (fixed) questions and response categories

– Semi-structured: some closed questions and some open (flexible) questions

– Unstructured: open questions

– Can be administered before or after the experiment (pre- and post- questionnaires)

– Advantage: experimenter does not need to be present

– Disadvantage: no interaction, so no discussion - unexpected data may be missed


22

Instruments [2]

• Interviews:– Similar possibilities to questionnaires– Advantage: interaction, so unexpected information may be captured– Disadvantage: participants may be less willing to express themselves

honestly

• Observation:– Experimenter observes the participant during the experiment, and

may video- or audio-tape the experiment– Advantage: can pick up on transitory events, e.g. hesitations– Disadvantage: Hawthorne effect, i.e. participants behave differently

when they are aware of being observed


23

Instruments [3]

• Think aloud protocols:– Participants are asked to “think aloud” during the experiment

– Advantage: can pick up on thoughts and feelings of the participant

– Disadvantage: an unnatural process, so may distract the participant from his task

• Automatic collection of data:– E.g. software logging


24

Data analysis

• Descriptive statistics– Describe the raw data in numerical or graphical form

• Inferential statistics– Statistical tests and significance testing

– Allow conclusions (inferences) to be drawn from results and generalisations to be made

BRING A CALCULATOR WITH YOU NEXT WEEK!

AMSc Research Methods Research approach IV: Experimental [1] Jane Reid [email protected].

Documents

Transcript of AMSc Research Methods Research approach IV: Experimental [1] Jane Reid [email protected].