Quantitative Research Methods

Session 1: What on earth are we trying to do here?Helen Petrie

Preamble

The course is not just about how to do statistical tests and which ones to choose

Will introduce a much broader approach to quantitative research methods

Why do we need this?

Examples of research disasters/problems Rats in mazes example: left or right

turns? Creating technology for a problem

people don’t have (very prevalent in assistive technology)

Explaining a difference that doesn’t exist? (worst case execution time analysis - WCET)

and an example I will draw on

Do people have difficulty navigating websites?

What characteristics of the navigational facilities alleviate these problems?

What inconsistencies in these facilities contribute to these difficulties?

A starting point …

What is the science in computer science about?

What does science mean to you?

Write down what makes computer science a science

What should make it a science… The use of the scientific method

“the collection of data through observation and experimentation, and the formulation and testing of hypotheses” (Websters Dictionary)

Is this generally what we are doing in computer science?

Why has the scientific method got to do with Quantitative Research Methods?If you are interested in QNT, then you must be

interested in undertaking something related to the scientific method (SM), although maybe you don’t realize this

Considering the whole philosophy and practicality of the SM should help you to do it more accurately, thoroughly and elegantly

First pass at defining the “scientific method”“Science is a structure built

upon facts” (Davies, 1968)

Empiricism Observe (lots of) facts -->

build a theory to explain them

Galileo often cited here - dropping objects off Leaning Tower of Pisa (repeated on the moon)

What are the “facts”???Is this man about to step onto the

surface of the moon or a secret NASA film set?

Are the shadows correct, where are the stars in the sky, did the flag flutter, where is the blast crater, etc etc

The observations/facts can be hotly disputed

Excellent discussion on wikipedia of the moon landing conspiracy including application of the “scientific method” to the debate

The “facts” depend on many thingsCould get into a whole

discussion of whether you see red the way I see red

Apart from people who have colour vision deficiencies (and there are a lot of them), as long as we agree to call a particular range of the spectrum red, let’s not stress this one

Facts for experts, facts for novices More interestingly,

what an expert /scientist sees may differ from what a novice sees

e.g. X-rays, K+ mesons, finger print whorls etc etc

Scientific method v2What you observe and how you describe it is

“shaped” (I won’t say determined) by your theoretical framework

So: Observe facts within a framework -> further

develop theory via induction -> make predictions via deduction

Small exampleA number of descriptive/relational studies show

that people have difficulty navigating websites when the navigational bars are inconsistent in their location through a website

(by induction)People need consistency in navigational

mechanisms(by deduction)People will have more difficulty and find a

website less acceptable if the navigation is inconsistent

Logical induction(not the same as mathematical induction)

Particular -> general/universal

All lectures I have attended are boring.Therefore all lectures are boring.

Problem of generalisability (highly important to the SM)I have only attended lectures by some lecturers (a sample out

of the population of lecturers) so my logic may be flawed

Flawed logicNotice that I deliberately did that when I gave

my example on navigation

Studies show that people have difficulty navigating websites when the navigation bars are inconsistent in their LOCATION

Therefore people need consistency in NAVIGATIONAL MECHANISMS

Maybe it’s only LOCATION that’s important

Strong theoriesHowever, I’m making a stronger “theory” here, that is

easier to falsify

First let’s look at the next step of the example:People will have more difficulty and find a website less

acceptable if the navigation is inconsistent

Deduction: general to specificNitty-gritty of the SM, making a good, falsifiable test of this Turning a theoretical hypothesis into a testable hypothesis

Three types of empirical researchDescriptive studies:

– carefully mapping out the situation (in effect, describing the “facts”)

– Observing behaviour, ethnographic research– Generally not enough of this in the social sciences

(because they are so busy testing their theories), so we lack information about how people behave (Carroll’s “psychology of tasks”)

– Are we developing software that people don’t need?

Three types of studies

Relational/correlational studies– Looking for relationships between things,

even if we don’t have a theory to explain them

– “fishing expedition” research - looking for what affects what, trying to find the components for a theory

Three types of research

What Rosnow and Rosenthal (gurus of research methods in psychology) call experimental research

but I’d rather call causal research - as it’s not always really experimental

where you try and pin down the nature of the relationships, the theory behind the observations/facts

Test a hypothesisUsually requires a series of studies, not just one

experiment

Creating a testable hypothesisWe tend to start from a general, vague questionNeed to turn this into something specific and

appropriateOften have two things we need to specify:

– Independent variable (the aspect of the environment that we are interested in)

– Dependent variable (the behaviour that we are interested in)

(variable = something that changes, takes different values, that we can alter or measure)

Operational definitions

Called operationalizing the hypothesis - turning the vague/theoretical concepts into operational definitions

Independent variable - the nature of the navigation on the website

Dependent variable - the difficulty that people have

Operational definitions IINot necessarily one particular operationalization of a

variable that is the “best”May well need multi-operationalism (i.e. different

operational definitions for variables) in the same, or different studies, before one is confident that one has understood a particular phenomenon

In HCI and related areas I think we do not do this enough - one study, one measure and we move on; in psychology one finds many studies on the same phenomenon with slight variations published

Non-psychologists see this as obsession, but it’s good science

Operational definitionsNavigational consistency

changes in navigational bars and elements of those bars: location, font colour, background colour, font type, exact wording, background decoration, groupingchanges in in-text navigation: initial colour, underlining, visited colour …

So in this one very small aspect of web design, there are many variables

One of the problems we have is isolating exactly what is causing the problem (true experimental design helps here - tomorrow)

Operational definitionsDifficulty that people have

Objective measures - time taken to complete tasks, errors made

But need to consider two aspects:– will the size of the difference be “noticeable”?

[equipment + power calculations] – Might I get ceiling or floor effects (i.e. everyone can do

the task error free/everyone finds something incredibly difficult)

Subjective measures - ask people to rate how “acceptable” a website is (and what exactly are you going to ask people to rate?)

Operational hypothesisPeople will take longer to complete tasks, make

more errors, and give lower ratings of acceptability on a website with a navigation bar that varies in its location from screen to screen in comparison to one in which the navigation bar appears in a consistent position on all screens

[I have multi-operationalized the dependent variable, but have a narrow, single operational definition for the independent variable - tomorrow you will see why]

H0 vs H1

I have stated the alternate hypothesis - that there will be a difference (known as H1)

I have stated it as a directional alternate hypothesis - that I’m predicting that one condition (level of the independent variable, arrangement of the world) will produced higher task times and errors, lower acceptability ratings

Sometimes one is predicting a difference, but cannot predict which direction it will take (a non-directional alternate hypothesis) - this makes a lot of difference in the statistical tests one conducts, and I’m sure Paul will take that up in his part of the course

H0 vs H1

The null hypothesis is the prediction that there will not be any difference - that navigational consistency will not have any effect on times/errors/acceptability ratings

In doing your statistical tests, you are actually trying to reject the null hypothesis

Fallacy of rejecting H0 If you do reject H0, you still might not have identified

exactly what in the situation that is causing the difference

A problem much discussed in research methods, the fallacy of rejecting H0

Paul Meehl, one of my heros, argued that unless you do a very tight experiment, your chances of falling into this fallacy is about 50% - so you might as well toss a coin

Penguin research video

Do you need an operational definitions/hypotheses?A question that I’m often asked by students - do I

need hypotheses for my research?Depends a lot on whether you are doing

descriptive/relational/causal researchIf causal - absolutelyIf the others, it certainly helps to set out what are

your variables (theoretical/operational definitions), the phenomenon/question/hypothesis you are investigating

Might not be able to formalize it to a precise H1

Allows you to make a simulation of what you will findReally useful to mock up the data you will

produce in a study, the levels of the independent and dependent variables, the numbers etc

Will it be statistically analysable (may need to consult a statistician, but much easier for them to advise you)

Will it really answer your hypothesis/question?

ExamplesRats: just whether they turned left or right did not

produce the right kind of data that discriminated enough between the conditions to answer the question posed (this was obvious to a person with some statistical training)

Navigation: is the question really about the best location for the bar or whether the bar is consistently in the same position (perhaps you need to answer the first before the second - a very common outcome of planning variables and hypotheses

Reading for this session

Chalmers, A.F. (1999). What is this thing called science? 3rd Edition. Open University Press.

Rosnow, R.L. and Rosenthal, R. (2005). Beginning behavioral research: a conceptual primer. 5th Edition. Pearson Prentice Hall.

Rosenthal, R. and Rosnow, R.L. (1991). Essentials of behavioral research. 2nd Edition. McGraw Hill.