Variables and measurements
description
Transcript of Variables and measurements
Variables and measurements
Today´s programme
Why do experimental research? Variables and measurement Different types of research methods Introduction to the scientific method Planning experimental work Experimental design
Exercises: Introduction to data entering in SPSS Team questions session Introduction to the mouse experiment – week 5
start
Practical information
Exercises in room: 4A58
Starts at 13.00 – ends at 15.00 (you can stay longer if you wish)
Handouts for exercises on the course website:
http://experimentdesign.wordpress.com
Science 101
We perform research in order to answer questions Do the users understand our menu
structure? Does our design put the user in a
pleasant mood? Can our customers use the product? Does giving elderly electric shocks when
they whine about today´s youth cause them to stop?
Etc. etc.
Answering questions
Two general ways in which to answer questions Observe what happens naturally in the
world▪ Correlational and observational methods
Manipulate an aspect of the environment and observe what happens▪ Experimental methods
Comparing methods
Correlational and experimental methods have much in common: Empirical: Gather evidence via observation
and measurement Measurement: Measures something Replicability: Results can be replicated by
others Objectivity: We seek an answer to the
question that is objective and unbiased
Difference: Experimental methods manipulate variables
Variables I
Scientists are interested in how variables change, and what causes the change
Anything that we can measure and which changes, is called a variable
”Why do people like the color red?” Variable: Preference of the color red
Variables can take many forms, i.e. Numbers, abstract values, etc.
Measurement
Measuring is important for comparing results between studies/projects
Different measures provide different quality of data
Nominal (categorical) dataOrdinal data Interval dataRatio data
Non-parametric
Parametric
Measurement
Nominal data (categorical, frequency data)
When numbers are used as names No relationship between the size of the
number and what is being measured Two things with same number are
equivalent Two things with different numbers are
different
Measurement
E.g. Numbers on the shirts of soccer players
Nominal data are only used for frequencies How many times ”3” occurs in a sample How often player 3 scores compared to player 1
Measurement
Ordinal data
Provides information about the ordering of the data
Does not tell us about the relative differences between values
Measurement
For example: The order of people who complete a race – from the winner to the last to cross the finish line.
Typical scale for questionnaire data
Measurement
Interval data
When measurements are made on a scale with equal intervals between points on the scale, but the scale has no true zero point.
Measurement
Examples:
Celsius temperature scale: 100 is water's boiling point; 0 is an arbitrary zero-point (when water freezes), not a true absence of temperature.
Equal intervals represent equal amounts, but ratio statements are meaningless - e.g., 60 deg C is not twice as hot as 30 deg!
-4 -3 -2 -1 0 1 2 3 4
1 2 3 4 5 6 7 8 9
Measurement
Ratio data
When measurements are made on a scale with equal intervals between points on the scale, and the scale has a true zero point.
e.g. height, weight, time, distance.
Measurements of relevance include: Reaction times, numbers correct answered, error scores in usability tests.
Variables II
Variables can take many forms
Continous variable Aggression – from calm to extremely violent
Discrete variables: No underlying continuum exists Either pregnant or not You cannot be ”a bit pregnant”
Difference can be fuzzy, and some continuous variables can be measured in discrete terms▪ Measuring reaction times to the nearest millisecond
Experimental vs. Correlational research
Correlational research: We observe what happens
Experimental research: We maniulate something and observe what happens
Correlational research is unbiased by the researcher
So why do experimental research?
Causality
Research questions often imply a causal link between variables Does giving the teacher red apples increase a
student´s grade?
Many research questions can be broken down into a proposed cause, and a proposed outcome The cause (apples) and the outcome (grade)
are variables The key is to figure out how the proposed cause
and the outcome relate to each other:
Causal relationship
Causality
Some problems with causality: (David Hume) We must be aware of confounding
variables (another variable than the one measured causes the effect)
Direction of causality: Is the cause the effect of the outcome? Or the other way around?
Need to isolate the causal variables (John Stuart Mill)▪ Solution: Compare two controlled situations:
one where the cause is present and one where it is absent
Causality
Karl Popper Distinguished between scientific- and
non scientific statements Scientific statements can be verified
with reference to empirical testing▪ ”beating children is morally wrong” – non-
scientific▪ ”On Earth, gravitational forces pulls objects
with mass towards the center of the planet” – scientific
He also argued that even testable theories may not be true – could just not be disproved yet
Testing theories
Summary: To test a theory we must:
1) Rule out other explanations of the supposed cause A) Control the conditions of experiment B) Minimize risk of random/unknown factors
influencing result C) Randomize the procedure
2) Gain confidence that one theory is correct and another is not
How do we do this in practice?
Testing theories
1) Ruling out other explanations:A) We need to control the conditions
of the experiment To verify if eating candy makes you fat,
we need one experimental setup where candy is present, one where it is not
The condition where the cause is absent is known as a control condition or baseline
Testing theories
In the simplest situation, the cause is either present or absent
There can also be multiple levels – (0 pieces of candy per day, 2, 4, 7, 10, 10000 etc.)
The variable being manipulated is the independent variable – it depends only on the experimenter Outcome variable
The variable not manipulated is the dependent variable – it´s value depend on the value of other variables in the experiment Causal variable
Testing theories
B) Minimize risk of random factors
We should compare situations that are identical in all respects apart from the proposed cause (causal/dependent variable)
All random factors should be held constant Everyone should eat the same candy Eat it at the same time of day Be the same gender Etc.
Testing theories
C) Randomizing procedure
We can rule out many random influences by randomizing parts of the experimental study E.g. randomly alllocating participants to
experimental and control groups – this spreads attributes randomly
I.e.: Do not permit any systematic bias to enter the experiment
Testing theories
2) Comparing theories
So far we have: Experimental conditions that control for
confounding factors We have isolated causal factors We have randomized our procedure
Now we need an objective way of comparing one condition with another: Math
Testing theories
In science, we draw inferences based on the confidence about a given set of results
i.e.: The difference between the experimental group (cause is present) and the control group (cause is absent) must be ”large” so as not to have occured by chance
This is where the statistics come in – to let us calculate the magnitude of the difference, and the chance of the result recuring randomly
Testing theories - Summary
Experimental research seek to isolate cause and effect by manipulating the proposed causal variable/-s
Correlational research does not always permit isolation of causal variables or controloing for confounding variables ....
Summary II
Correlational methods merely identify relationships: they cannot establish cause and effect.
A correlation between two variables is inherently ambiguous:
X might cause Y Y might cause X X and Y might both be caused by a third
variable or set of variables.
Summary III The experimental method is the best way of identifying causal
relationships. Example:
X causes Y (METEOR CAUSES NO DINOSAURS) if:
X occurs while Y is present (DINOSAURS MUST BE PRESENT BEFORE METEOR);
Y happens in the presence of X (DINOSAURS CROAK WHEN METEOR HITS);
Y does not happen in absence of X (NO METEOR, DINOSAURS (and no mammals...).Meteor:
No meteor:
Experiments enable us to eliminate alternative explanations:
To establish causality, we use experimental situations that differ systematically only on one variable (the independent variable) ...
... and measure the effects of this on an outcome variable (the dependent variable).
Summary IV
Questions
A study of student boredom examined factors involved in boredom during lectures. 50 students were involved. Half attended a very boring 2-hour lecture, the other half sat outside in the sun for 2 hours. After each period, the students were asked to rate their boredom on a scale from 1-10
How was outcome measured? What levels were used? What was the control group Were confounding variables controlled for?
Answers
How was outcome measured? Ordinal data (ranked, no zero point, not equal distance
between points) What levels were used?
Scale of 1-10 What was the control group
The students in the sun Were confounding variables controlled for?
Arguable – the environment of the control group was different, this problem if intent to compare between ”boring” and ”non-boring” lecture
As is now, the experiment only shows effect ”sun” vs. ”boring lecture”
If ”non-boring” lecture included – how do we make sure it is not boring?
The scientific method
Science
Science: Any systematic knowledge or practice.
Science generally refers to a way of acquiring knowledge through the scientific method, as well as the organized body of knowledge gained through such research.
Adheres to positivist philosophy: Only authentic knowledge is scientific knowledge
Science = Logic + Observation
Science
Three types of science: Natural science: The study of natural
phenomena Social science: The study of human behavior
and societies Formal science: Mathematics – uses a priori
rather than empirical methods, includes statistics and logic
Two first are empirical sciences, third a mixture, however all feed into each other
▪ A priori = deductive knowledge (independent of experience)
▪ A posteriori = Inductive knowledge (dependent on experience)
Science
Experimental science: Another term for empirical sciences
Applied science: Application of scientific research to specific human needs
The two are often combined
Empirical Sciences
Empirical sciences Knowledge obtained from observable
phenomena Reproduceable: Phenomena must be
reproduceable under experimental conditions by other scientists, in order to validated.
Careful, objective and systematic study of an area of knowledge
Must follow the scientific method
The scientific method
The scientific method A body of techniques for investigating
phenomena, and acquiring knowledge
Collection of data through observation and experimentation, and the formulation and testing of hypotheses
Evidence must be observable, empirical and measureable, subject to principles of reasoning
The scientific method
Empirical research must follow: Define the question Gather information and resources (observe) Form hypothesis Perform experiment and collect data Analyze data Interpret data and draw conclusions that serve as a
starting point for new hypothesis Redo entire cycle if necessary Publish results Retest (frequently done by other scientists)
Alternative: Explorative approach – similar requirements on
objectivity and reasoning, but forgoes hypothesis forming.
The ACTUAL scientific method
Hypothesis
A hypothesis defines an expected relationship between variables, which can be empirically tested.
For example: Eliminating the minimap in StarCraft will
increase player engagement Flash animations make a website more
attractive to blind white mice
Quantitative vs. Qualitative Empirical research methods come in two
forms:
Quantitative methods: Collect numerical data, strictly objective, analyzed using statistical methods
Qualitative methods: Collect data in the form of text, images, sounds etc. Drawn from observations, interviews,
documentary evidence etc., analyzed using qualitative data analysis methods (e.g. content coding)
Data and analysis can be subjective: Relies on researcher experience
Selecting methods
Qualitative: More appropriate in early stages of research
(exploratory research) and for theory building Qualitative methods applies well in real world
setting, but lack validity and control Problem with subjective interpretation of the
data
Examples▪ Case study: Observations carried out in a real world
setting▪ Action research: Applying a research idea in practice,
evaluate results, modify idea (cross btw. experiment and case study)
Selecting methods
Quantitative: Appropriate when theory is well developed. Theory testing and refinement
Examples:▪ Experiment: Apply treatment, measure results: This is
the only method that can demonstrate causal relationship between variables. Associated with the scientific method
▪ Survey: Asking rated questions in an interview▪ Historical data: Patterns in WOW auction house
spendings
Most quality research include both types of methods
Selecting methods
Method selection is critical to success of any project
Selection must be driven by state of knowledge
All hypotheses should be tested using two independent venues of data (enables crosss-correlation or data-triangulation)
Planning experiments
Planning experiments
Proper preparation is vital to all research And eliminates nasty surprises
Preparation 101: Ask relevant questions: What should I research? What has been done already? How should I research it? Can my experimental design be
meaningfully analyzed? Is my measure valid? What am I expecting to find?
What should I research?
Well, that is kinda up to you, but: The process of finding out can be
extensive Process of going from an initial
interest to defining a specific research question:
Perception of colors?
Reading textbook + browse the net
Reading science journals
Finding key literature on the topic
Formulating research question
Color perception on monitors
Does cultural background impact on the emotional impact of the color red on monitors?
Emotional impact of perceiving colors
Emotional impact as a function of cultural background
What should I research?
In the industry, sometimes your boss will tell/ask you: Test this product! Do users like what we do? Is this material better for ...?
The above statements are not research questions – they are not specific enough, i.e. not experimentally testable
Therefore, need to go through same process of knowledge gathering and refining the questions being investigated.
Often you end up with multiple research questions
What has been done already?
An important early step is to figure out if someone else has already done what you plan to do
If there is other relevant research out there
Many useful library databases of scientific literature, e.g.: Web of science ACM digital library PsyInfo (psychological abstracts)
What has been done already?
Another source is specific specialist journals in the area and conference proceedings.
For example, Human-Computer Interaction: Cyberpsychology and Behavior (journal) International Journal of Human-
Computer Interaction (journal) Computer-Human Interaction Conference
(CHI) proceedings (ACM publishers) INTERACT conference (ACM publishers)
What has been done already?
Especially useful when investigating a new area (and you will be doing this a lot!) is review articles
These are articles where the authors evaluate a lot of literature in an area and try to sum it up, or perform meta-analysis on the data published in an area
Review articles provide overviews of what has been done in an area – good place to start
How do I research my question? Choosing a dependent variable: Deciding
what to measure The outcome measured (the dependent
variable) should be an index of the construct of interest▪ E.g. Numbers of dinosaurs killed after meteor
strikes of variable size
How do I research my question? Define what the cause and effect are in
the research question (causal relationship): Cause: Meteor, effect: death of dinosaurs
Isolating dependent variable (DV) and independent variables (IV) is usually straight-forward
Figuring out how to manipulate IV and measure DV is not so easy ... (okay, it can be REALLY difficult)
How should the independent variable be manipulated? No easy answer
Rule of thumb: Manipulate in such a way as to compare a condition in which the cause is present with a condition in which the cause is absent
Important to isolate the dependent and independent variable – no confounding variables
▪ E.g.: mad sharks also killing dinosaurs!
How should the independent variable be manipulated?
Ensure comparable experiment conditions for all participants/tests E.g. use the same dinosaurs, same meteor
type, same Earth etc.
How should the independent variable be manipulated?
Levels of manipulation: basic level is control compared to experiment, but there can be more e.g. 5 different meteor sizes
What is the expected effect?
Measuring
The ways in which the dependent variable is measured has important ramifications, e.g.:
Number of dinosaurs killed total▪ Total count but no idea about where they
were when killed
Number of dinosaurs killed per square km.▪ Averaged total count, but some geographic
information
How to analyse the data?
Data analysis must be considered when planning the experiments Otherwise data may not be meaningful to
our purpose
The analysis method will depend on the data measure (nominal, ordinal, interval, ratio)
Rule of thumb: Get ratio data, and use parametric statistics (most methods available)
Is my measure valid?
Does your experiment measure what it should? This is called validity
Terminally important in self-report studies Where people tell us what they think/feel e.g. ”rate on a scale from 1-5 how fun this
game is”
We will get back to this in detail later in the semester when
discussing how to construct surveys!
Is my measure reliable?
Reliability is the ability of the measurement method to produce the same result under the same circumstances replicability criterion
Trust me!
Is my measure reliable?
Reliability is like validity foundational for all experimental work
Difficult ideal for questionnaire-based work – people differ so results will vary between groups of participants Ways of overcoming this, e.g. split-half
method
More on this later also ...
Measurement error
Measurement error is the difference between the scores we get on our measurement scale and the level of the construct being measured
Example: A weigth is precise to 0.5 kg (+/- 0.25 kg)
Fisherman measuring length of a live catch: +/- 10 cm (bidirectional error)
Fisherman reporting length of a catch: +200 cm (unidirectional error)
Summary
When wanting to do experimental study, planning is essential. Key steps include:
Define the research question What variables to manipulate What is the independent and dependent
variable Type of measure How to construct measures Reliability, validity
Questions - groups
Men and women shop for clothing differently
Define: Independent variable?Dependent variable?How to measure? (qual. vs.
quant./measure type)How set up experiment? Is measure valid? Is measure reliable?What is the measurement error?
Questions - groups
Large buttons on websites make navigation easier
Define: Independent variable? Dependent variable? How to measure? (qual. vs.
quant./measure type) How set up experiment? Is measure valid? Is measure reliable? What is the measurement error?