Experimental methods in empirical research The Rolls-Royce of scientific research? Daniel Gile
description
Transcript of Experimental methods in empirical research The Rolls-Royce of scientific research? Daniel Gile
D.Gile Experimental res 1
Experimental methods in empirical researchThe Rolls-Royce of scientific research?
Daniel Gile
www.cirinandgile.com
D.Gile Experimental res 2
What is good scientific research?
(In one easy lesson – satisfied or no money back)
Many possible definitions, but even the most demanding researchers would probably accept the following:
Good scientific research consists in exploring reality rigorously,
cautiously, in full awareness of human limitations inherent to such exploration, in compliance with social, editorial and
institutional norms of science
taking care to avoid and/or limit and/or measure the probability and/or magnitude of possible errors
D.Gile Experimental res 3
Is a specific research method or paradigm part of that definition of good scientific research?
D.Gile Experimental res 4
Is experimental research a prerequisite for
scientific research?
Japanese psychologist
Bears in the mountain
Tuna fish migration
Astronomy
Paleontology…
D.Gile Experimental res 5
Theoretical vs. Empirical methods
Theoretical research → ideas:
Analysis of facts
Analysis of theories
Empirical research is centered → data:
Collection
Analysis
(Most often for the sake of developing or testing theories)
D.Gile Experimental res 6
Naturalistic vs. Experimental research
Naturalistic research:
collecting data
about phenomena as they occur ‘naturally’
Experimental research:
Creating ‘controlled’ environments
to collect data about phenomena in these environments
D.Gile Experimental res 7
Why experimental research?
Phenomena which occur naturally are influenced by many factorsso
It is difficult to trace the influence of one particular factor
For example :
The success of a learning processMay depend on a teaching method
(in which the investigator is interested)
But could also be influenced
By the teacher’s personalityBy environmental conditionsBy the student’s motivationBy the time available to study…
D.Gile Experimental res 8
Why experimental research? The principles (1)
Something happens over time (from t1 to t2) to the object of study O, which turns into O’
We want to know why and how
We think factor F1is the cause of this change
But perhaps it is F2? Or F3?
How do we find out?
t1 t2
F1
F2
F3
O O’
D.Gile Experimental res 9
Why experimental research? The principles (2)
We think factor F1 is the cause of this changeBut perhaps it is F2? Or F3?
One possibility is to create an environmentWhere O is shielded from F2 and F3
So if O changes, it can only be due to F1This is one type of experimental setup
t1 t2
F1
F2
F3
O O’
D.Gile Experimental res 10
Why experimental research? The principles (3)
Another possibility is to keep F2 and F3 constantand suppress (or change) F1
If nothing happens to O, then it must be F1 which was causing the change
F1
F2
F3
O O’
?F2
F3
O
D.Gile Experimental res 11
Why experimental research? The principle (4)
In all these (and other) scenarios, We create a ‘controlled’ environment
in which we shield the object from outside factors or ‘control’ these factors
So as to be able to isolate the influence of whatever factor(a ‘variable’) in which we are interested
from the influence of ‘confounding’ factors
F1O O’
D.Gile Experimental res 12
Experimental research - An example
Is translator training method A an efficient booster of translation skills?
If we ‘just’ compare the work of 40 translators, 20 of whom have been trained with A and 20 without it,
Differences in translation performance may be due to A, but also to:
- characteristics of the texts, - age, - experience, - languages involved, etc.
If we experiment with the same textand two groups with similar characteristics
(age, experience, languages etc.)
If differences found, probably due to A
D.Gile Experimental res 13
Experiments for hypothesis-testing
This is a prototypical model of hypothesis-testing experiment
The behavior of 2 or more groups is observed
Each is made to be similar to the others except for the value of the variable to be examined
Comparisons of output variables (‘dependent variables’) are made
with a view to determine whether differences are likely to be due to chance or not
If not, they are called ‘significant’.
D.Gile Experimental res 14
‘What if’ experiments
Hypothesis-testing experiments are the best knownBut not the only ones
Exploratory experiments are also used:What if…?
(No special hypothesis)
What if I gave a translator this type of reference document?What if I made a translator work in a team with an expert without
any knowledge of the source language?What if I doubled his/her pay when the customer is happy with
the product?
What if…
D.Gile Experimental res 15
‘What if’ examples
Caveman – biological clock
Isolated Community of volunteers
Big Brother
Agriculture
Children in a room
Sometimes with hypotheses or theories (formalized)
Sometimes without
D.Gile Experimental res 16
Advantages of experiments
Eliminate the ‘confounding’ influence of ‘parasitic’ variables
Often allow to measure some indicative value more accurately than in naturalistic studies
(precise corrections made during translation, pauses etc.)
Allow to create interesting situations which do not occur or occur very seldom in reality
and therefore could not be studied naturalistically
Make it possible to study situations with small investments as opposed to huge investments
which studying the same phenomena with naturalistic methods would entail
D.Gile Experimental res 17
Why do experiments have such a high status in science?
- Because of the advantages outlined in the previous slide
- Because often used to test theoriesand therefore take place at an advanced phase of the research
process on a given phenomenon
- Because they often involve sophisticated thinking and design
- Because they most often involve statistics, which is a highly sophisticated tool
BUT DOES ALL THIS MAKE THEM MORE ‘SCIENTIFIC’THAN NATURALISTIC STUDIES?
D.Gile Experimental res 18
Limitations of experiments: ecological validity
Most often cited limitation of experiments in the social sciences has to do with ecological validity:
Is the task and are the measurements a valid model for real life?
For example: A translator who translates in a particular room under the
observation of cameras or an experimenter or on a specific computer with a specific type of software or under specific
time constraints or with specific instructions as to the use of data or with the instruction to think aloud while translating
…Does s/he perform the same translation task as in real life?
If not or if the answer is uncertain, can conclusions on real life be drawn from the experiments with the caution prescribed by
science?
D.Gile Experimental res 19
Limitations: neglected confounding variables
In experiments, confounding variables are controlled.But are all of them controlled?
- Potentially relevant of which the investigator is not aware?
- Variables which cannot be controlled because there is no way to control them or controlling them is too expensive/impractical?(family traditions, some genetic factors, some personality traits,
economic situation and history, history of social interaction…)
- Variables which cannot be controlled without reducing the sample to an insignificant size?
(only subjects aged 20 to 25, right-handed, with a given personality profile, with parents from a particular background and having a specific economic status, who attended particular schools with particular curricula, who read at least 20 books a year…)
D.Gile Experimental res 20
Neglected confounding variables
If we control F2 and F3 but not F4 and F5
Can we safely assume that the effect measured
in the experiment is only due to F1?
The answer is clearly No
t1 t2
F5
F4
F1
F2
F3
O O’
D.Gile Experimental res 21
Limitations: experiments as a case study
By ‘controlling’ relevant variables and parameters,For instance all subjects must be of the same age
Or right-handed, or work on one particular source textOr have 5 to 10 years of professional experience
Experimenters deliberately exclude some naturally occurring variability
And it is difficult to say with the appropriate caution prescribed by science
Whether the results would also be replicated with other values
If there are many replications of the experiments with different values (and without bias), fine
But if there aren’t?
D.Gile Experimental res 22
Limitations: statistical significance (1)
Inferential statistics (statistical tests) are used to determine whether differences are ‘significant’ or not
Variables in a population have certain values and a ‘distribution’(How tall, how many times go to church, how much money
spend on Belgian chocolate…)
In order to check whether some factor makes a difference(more training, more work, better food…)
Check whether the population with the ‘extra’ has the same distribution of values than the population without the ‘extra’
(or with a certain value in that variable and another value)
D.Gile Experimental res 23
Limitations: statistical significance (2)
How will a statistical test check this?
You will measure the values of a relevant output variable(for instance some metric of translation quality)
On samples from the two populations(the two conditions)
You will find some difference between the mean values
The test will tell you how likely it is that the difference is compatible with the idea that the two populations have the
same distribution for the relevant variable
If it tells you this is unlikely, you will say the difference is ‘significant’
D.Gile Experimental res 24
Limitations: statistical significance (3)
Actually, it will determine, on the basis of measurements made on the sample means
A certain distribution
And the proportion of values which lie between a lower and an upper limit if the two populations are not different
If your value is below a threshold or above another threshold,It will tell you that it corresponds to only 5% or 1% of the cases
still compatible with the idea that the two populations are the same
“significant at .05”“significant at .01”
D.Gile Experimental res 25
Limitations: statistical significance (4)
)
D.Gile Experimental res 26
Limitations: statistical significance (5)
‘Significance’ is determined at a certain level: the experimenter allows him/herself a certain probability of ‘false
positive’ results (of wrongly concluding that the difference is not produced by chance)
Significance at 5% (.05) means that the experimenter allows him/herself to draw the wrong conclusion that the difference is
‘real’ once every 20 times
If you were to decide that something is true or not, would you say that it is true if you have one chance out of 20 to be wrong?
(Significance at 1% means that you could be wrong once in 100 times)
D.Gile Experimental res 27
Limitations: statistical significance (6)
Statistical significance tests rely on a number which is calculated from values obtained from the samples being examined,
with a threshold For instance, if you get 5,62 the difference is significant, if you get
5,63 it isn’t.How do you like this transition from ‘yes’ to ‘no’?
Does it make sense to you?
Statistical significance says the difference is likely to be due (or not to be due) to chance.
It does not say how large the difference is on average.In TS, how useful is knowing that there is some difference, but without
knowing how large it is?
D.Gile Experimental res 28
So are experiments not a good research paradigm?
Experimental research can be very useful:
It is potentially powerful in eliminating (some) confounding variables
It does allow observation of situations which would not occur naturally
(but in the social sciences, what about ecological validity?)
It does allow precise measurements which would be difficult or impossible to carry out in real life
It can take on board variability, measure it and overcome it
But it has its limitations, and works best under certain conditions
D.Gile Experimental res 29
Conditions for powerful experimental research
Proper design, and in particular proper sampling and piloting
Cautious inferencing from the results
Correct statistics (test that conditions for the use of tests are met)
Many replications with various values of the independent variable
(so that you really cover a range of values and move away from the case-study scenario)
How often are such conditions met in your field?
D.Gile Experimental res 30
If conditions are not met
Still useful, but for ideas and tentative results
Experimental results are by no means the final ‘scientific criterion’ for a decision
In particular, ‘significance’ is just a tentative direction
Naturalistic findings can be more powerful
Especially with the use of corporaBecause they allow the use of much authentic data
Can even test hypotheses, more powerfully than experiments!
D.Gile Experimental res 31
Conclusion
Experiments are a tool
A powerful tool under certain conditions(and in particular in cognitive psychology)
But only a tool
Not very powerful under other conditions
NOT THE ULTIMATE IN SCIENCE
Don’t let scientific status symbols fool you
Science is in the human mindWhen you explore reality with a rigorous, skeptical, cautious,
systematic mindset,You are ‘scientific’ whatever the tool