Scientific Research Robert O. Briggs Delft University of Technology University of Arizona...
-
Upload
philippa-jordan -
Category
Documents
-
view
220 -
download
0
Transcript of Scientific Research Robert O. Briggs Delft University of Technology University of Arizona...
Scientific Research
Robert O. BriggsDelft University of Technology
University of [email protected]
Tucson, AZ 85721
Today’s program
Introductions Epistemology The Philosophy of Science The Scientific Approach
Which is Bob’sExciting Secret Identity?
Bob Briggs Facilitated the surrender of Napoleon
at Waterloo Invented the Internet Sang with Elvis Presley in concert
Photographic Evidence
Elvis
Bob
You ain’ nuttin’ but a hound dog!
You ain’ nuttin’ but a hound dog!
You ain’ nuttin’ but a hound dog!
Three Ways to Think About Academia
The Philosophical The Pragmatic The Publishable
EpistemologyThe Philosophical View
The study of the nature of knowledge Presuppositions Foundations Extent Validity
Epistemology:The Philosophical View
A way of knowing A way of creating knowledge
Prevailing Epistemologies
Interpretivism Criticalism Positivism
Interpretivism Creating knowledge about… The inferences people draw from
and the meanings people ascribe to the words and actions of others.
Key Assumption “There is no objective reality”
Positivism (Science)
Creating knowledge about… Cause-and-effect Key Assumption
“There is an objective reality”
Criticalism
Creating knowledge about… The nature of and resolution
of deep social ills Key Assumption
Deep social ills exist
Epistemology Myths
Positivism and Interpretivism are mutually exclusive world views
Objective Reality vs. No Objective Reality?
What is Reality?
Epistemology Myths
Positivists skew studies to find the result they want
Interpretivists don’t believe in gravity
Epistemology Myths
Interpretivism is qualitative Positivism is quantitative
Epistemology Myths
An epistemology is something you “are” I’m an interpretivist I’m a positivist
Pragmatic Epistemology
A set of mental disciplines To keep us from drawing (and then
publishing) bone-headed conclusions
Pragmatic Interpretivism
A set of mental disciplines To keep us from drawing bone-
headed conclusions About the inferences and
meanings people ascribe to the words and actions of others
Pragmatic Positivism
A set of mental disciplines To keep us from drawing bone-
headed conclusions About patterns of cause-and-effect
Publishable Positivism
Report of a study on the causes of a phenomenon-of-interest that…
Provides convincing arguments that…
The conclusions may not be bone-headed
The Philosophy of Science
Positivist Assumptions Regular patterns of causation Independent from human mind “Knowable”
The Boundaries of Science
If it’s not about cause-and-effect… It’s not science Period.
Goals of Science Create causal models for
phenomena of interest Test the usefulness of those
models Use those models to increase the
likelihood people will survive and thrive.
Why should you care about
Positivist Science?
Why Should You care?
Good science will Make it more likely that people will survive and
thrive Make you work smart Get you published in the good journals
Bad Science will Harm others Waste effort, time, and money Embarrass you for years
The Phenomenon of Interest
In the world of cause-and-effect… The phenomenon-of-interest is the
EFFECT The EFFECT is what you seek to explain The EFFECT is what you seek to
improve The EFFECT is the outcome you
measure
The three most exciting words in science are, “Gee, that’s funny…”
- Issac Azimov
The Positivist Disciplines
Phenomenon-of-Interest Who Cares? Theory Hypotheses Research Methods Analysis
The First Discipline
Explicitly Define The Phenomenon of Interest
The First DisciplineDefine The Phenomenon of Interest
Explicitly In writing Refine the definition as your
understanding deepens Challenge your definition
continuously
Explicitly Define thePhenomenon of Interest
Satisfaction First definition:
The degree to which needs are fulfilled Measures
I am satisfied My needs are fulfilled I feel satisfied
Better definition An affective arousal with a positive valance in
response to the judgment that needs have been or will be satisfied
Measures I feel satisfied with… …gave me a feeling of satisfaction I feel good about…
Phenomenon Vs Domain
The phenomenon-of-interest is the OUTCOME you hope to improve measurably Productivity Creativity
The domain is the setting in which the outcome manifests Requirements Engineering Project Management
Phenomenon vs. DomainThe Pragmatic View
You study the phenomenon of interest …Don’t ever forget it
You sell the domain To funding agencies To reviewers To readers
The Second Discipline
Who Cares?!?
Who Cares?!?
Why is this phenomenon-of-interest is worthy of study?
Philosophical “Who cares?”
Science must increase the likelihood that people will survive and thrive
Society provides the scarce resources for scientific enquiry. You must be able to justify your use of them.
Pragmatic “Who Cares”
Your reviewer just had a “much better” paper rejected by the same journal
Publishable “Who Cares?”
1. The phenomenon of interest is worth studying1.1 People are more likely to survive
and thrive if we understand the cause of this phenomenon
1.2 The existing literature does not fully explain the causes of this phenomenon
Publishable “Who cares?”
You must define explicitly the phenomenon of interest in the first or second paragraph
It’s your anchor for all that follows
Good “Who Cares?” 1.1 Organizations exist to create value for
stakeholders Organizations operate under risk Mitigate risk, the organization may survive Internal risk assessments can mitigate risk Risk assessments must be run by groups If we can make risk assessment groups more
productive, we may increase that people will survive and thrive!
Productivity is…. This study examines the use of GSS to make
risk assessment groups more productive.
Bad “Who cares” 1.1 Organizations do risk assessments
frequently We studied collaborative risk
assessment workshops
Ugly “Who Cares” 1.1
We collected some data about risk assessment workshops
Good “Who Cares 1.2” Connolly et al (1992) showed that productivity
of brainstorming teams could be improved by making them anonymous.
However, Johnson and Stephens (2003) showed better productivity when brainstorming teams were identified
A causal theory of productivity might be useful for explaining these seemingly disparate results, and might allow the development of even better brainstorming techniques.
This paper offers such a theory
Bad “Who Cares” 1.2 Jones (1983) said nothing has been
done about productivity Smith (1978) called for more
research on productivity Johnson (1981) studied productivity
among factory workers I studied productivity among
brainstorming groups
Ugly “Who Cares” 1.2
I searched 3 on-line databases and browsed 6 web search engines and only found 2 articles on this topic.
Little is known about this topic Nobody has studied this topic yet.
Publishable PositivismThe Opening Argument
Section 1. Who Cares?!?Argument: This phenomenon is worth studying.
1.1 People will be better off if we understand this phenomenon
1.2 Current literature does not yet fully explain it
The Third Discipline
Theory
Theory
A causal model of the phenomenon-of-interest
Drives all subsequent activity Hypotheses Experimental design Measures Analysis Conclusions
Data have no meaning except with respect to
the theory from which they spring
Today’s Message:Today’s Message:
Goals of Science Create causal models for
phenomena of interest (Theory) Test the usefulness of the models
(Experiment) Use those models to increase the
likelihood people will survive and thrive. (Application)
Anything Missing?
TruthTruthTruthTruth
Anything Missing?
Positivist Perspective
Science = Useful Science = Useful
Science <> TrueScience <> True
A useful model is better than Truth
Useful Is Better Than True
Useful Is Better Than True
Name the PhenomenonName the Phenomenon
BobeziteBlock
Describe the PhenomenonDescribe the Phenomenon
BobeziteBobeziteBlockBlock
A
B
Explore the PhenomenonExplore the
Phenomenon
BobeziteBobeziteBlockBlock
A
B
BobeziteBobeziteBlockBlock
Explore the PhenomenonExplore the
Phenomenon
BobeziteBobeziteBlockBlockBobeziteBobezite
BlockBlockBobeziteBobeziteBlockBlockBobeziteBobezite
BlockBlock
A
B
BobeziteBobeziteBlockBlock
A
B
Describe the dynamics of the phenomenon
Describe the dynamics of the phenomenon
A Useful ModelA Useful Model
One Gear
TruthTruth
One Thousand Gears
When does the Model Become Useful?
When does the Model Become Useful?
When you want todo something newWhen you want todo something new
Therefore
For matters of cause-and-effectA useful model (Theory)
is better than Truth
An experiment, without a Theory is
meaningless
What is a Theory?
An excuse to not do anything meaningful?
Pie-in-the-sky disconnect from reality?
There is nothingmore useful
than a good theory
What is a theory?
Causal Model Internally Consistent Explains and/or predicts Proposes mechanisms of causation Testable
Structure of a Theory
Axioms Propositions
Axioms
Assumptions about the fundamental nature of the universe
Axioms are “received”
Example Axioms
Axiom 1: Human attention is limited Axiom 2: All action is purposeful for goal attainment
Axioms Are Received
Source is irrelevant Feynman’s Inspiration
Propositions
Functional Statements of cause-and-effect that must be logically true if the axioms are true
Propositions are...
Causal Composed of constructs Without empirical content
Useful Propositions
Proposition 1: Productivity is a function of effortProposition 2: Effort is a function of goal congruenceProposition 3: Effort is an inverse function of distraction
ProductivityEffort
Distraction
Goal Congruence
+
-
+12
3
Mathematical Propositions
P = (E)Where
P = Productivity
G = Goal Congruence
E = -(D)Where
E = EffortD = Distraction
Problematic Propositions
WORK DESIGN & EXECUTION OUTCOMES
IMPLICIT INCENTIVES
EXPLICITINCENTIVES
SOCIAL ENVIRONMENT
TECHNICAL ENVIRONMENT
RESOURCE ENVIRONMENT
ORGANIZATIONAL STRUCTURE
ENVIRONMENT
DISTRIBUTED WORK
ARRANGEMENT
ORGANIZATIONAL LEVEL
GROUP LEVEL
INDIVIDUAL LEVEL
INCENTIVE STRATEGY
(e.g. Reward & Compensation)
WORK DESIGN & EXECUTION OUTCOMES
IMPLICIT INCENTIVES
EXPLICITINCENTIVES
SOCIAL ENVIRONMENT
TECHNICAL ENVIRONMENT
RESOURCE ENVIRONMENT
ORGANIZATIONAL STRUCTURE
ENVIRONMENT
DISTRIBUTED WORK
ARRANGEMENT
ORGANIZATIONAL LEVEL
GROUP LEVEL
INDIVIDUAL LEVEL
INCENTIVE STRATEGY
(e.g. Reward & Compensation)
Qualities of a Good Theory
Parsimony Explanation/Prediction Boundaries
Pragmatic Theory
You usually start with propositions and work backward to axioms
You usually start badly and get better
You use someone else’s theory whenever you can
Your technology is probably not in your theory
Pragmatic Theory
A good theory will get you to the moon and back safely on the first try
Good theory will do more to save you from drawing bone-headed conclusions than any other discipline of positivism
Publishable PositivismAlternative Wordings for Propositions
Y is a function of Z Z causes Y Z determine Y The more Z you do, the more Y you
get Z has a positive influence on Y
Publishable Positivism
Section 2. TheoryArgument: I understand what causes Z
If we assume X to be the case, then it must be that:
Proposition 1: Y is a function of Z.
The Fourth Discipline
Hypotheses
The Fourth DisciplineHypotheses Comparative statements
Some explicitly stated measurable outcome Compared across at least two treatments
Logically derived from propositions Tests the proposition Empirical content An answer to a research question
Example Hypothesis
H1: Brainstorming teams with access to an automated social-comparison-feedback graph will produce more unique ideas than teams with no automated graph
Example Hypothesis
H2: During brainstorming, the more we pound randomly on the walls, the fewer ideas a team will produce.
Problematic Hypotheses
H3: Groups using richer media will exhibit higher levels of cohesion initially
Problematic Hypotheses
H4: On negotiation tasks, face-to-face groups will outperform computer mediated groups, will experience less process difficulty, than computer-mediated groups, and will have more favorable reactions to their group task performance, interaction process, and communication medium
Publishable Positivism
Section 3. HypothesesArgument: This theory is testable
If, as Propositon 1 posits, Y is a function of Z, then it must be that:
H1. People using Technology-1 will score higher on the Y-test than do people using Technology-2.
The Fifth Discipline
Research Methods
An experiment without a theory is meaningless
Today’s Message:Today’s Message:
Experiment
Compare outcomes Different treatments Control other possible causes
Experimental InquiryExperimental Inquiry
TreatmentTreatment11
TreatmentTreatment11
TreatmentTreatment22
TreatmentTreatment22
IdenticalSubjectPools
IdenticalSubjectPools
ResultsResultsResultsResults
ResultsResultsResultsResults
}} CompareCompare
Investigative InquiryInvestigative Inquiry
PopulationPopulation11
PopulationPopulation11
PopulationPopulation22
PopulationPopulation22
ResultsResultsResultsResults
ResultsResultsResultsResults
}} CompareCompareOneTreat-ment
OneTreat-ment
Positive Results mean...
Manipulation caused difference Hypothesis has support Theory has support
Negative Results Mean
Experiment Flawed? Hypothesis Flawed? Propositions Flawed? Axioms Broken?
The Only Scientific Truth
The Model is No Good
Publishable Positivism
Section 4. MethodsArgument: I found a reasonable way to test the hypotheses
4.1 My DV instantiates the phenomenon of interest4.2 My IV instantiates a causal construct4.3 My approach would reveal a difference if there were one4.4 There are few alternative explanations for any difference discovered
An Experiment without a theory is
meaningless
Phenomena: Phenomena: Large, Odd-Smelling BoxesLarge, Odd-Smelling Boxes
Scientific Instrument: Scientific Instrument: DrillDrill
Collecting Data without A TheoryCollecting Data without A Theory
Collecting Data Without A TheoryCollecting Data Without A Theory
Collecting Data Without A TheoryCollecting Data Without A Theory
Collecting Data With a TheoryCollecting Data With a Theory
Collecting Data With a TheoryCollecting Data With a Theory
Collecting Data With a TheoryCollecting Data With a Theory
Collecting Data With a TheoryCollecting Data With a Theory
Collecting Data With a TheoryCollecting Data With a Theory
A Physicist Uses the A Physicist Uses the Elephant TheoryElephant Theory
+
=Fission!
A Farmer Uses the Elephant A Farmer Uses the Elephant TheoryTheory
A Farmer Uses the TheoryA Farmer Uses the Theory
There is nothingmore useful
than A Good Theory
An Experiment without a theory is
meaningless
Data have no meaning except in reference to the
theory from which they spring.
Kinds of Causal Theories
Descriptive Predictive Explanatory
Descriptive Model:Descriptive Model:
•What factors impact the length of pins?•Pin-length factors:
- Social Tone (Parties)- Bob-Presence- ?
A
BobeziteBobeziteBlockBlock
B
BobeziteBobeziteBlockBlock
A
B
Predictive Causal Model:Predictive Causal Model:
How can we predict the length of B?The length of B is directlyproportional to the length of A
Predictive Causal Model:Predictive Causal Model:
What about the Hacksaw Experiments?
BobeziteBobeziteBlockBlock
A
B
A
B
Explanatory Model:Explanatory Model:
Why is the length of B proportional to the length of A?
A and B are linked by a gear.
Another View of Theory
Three out of four kinds of theories are dangerous
Levels of Theory
A - Fully Axiomatized B - Building or Broken C - Construct Theory D - Descriptive Theory
A - Level Theory
All Axioms in place Many propositions expressed Extensive, unequivocal empirical
support
A-Level Theory F = M * A
A good theory gets you to the moon
on your first try.
B-Level Theory
Some axioms in place Some propositions Little empirical support Danger - some unknown effects
C-Level: Construct Theory
Assert that a construct exists Find a way to measure it Danger: You always will find a way
to measure it
C-Level Studies
Communication Apprehension Instrument
Measure different groups Compare to other constructs
C-Level Studies
Locus of Control $3,000,000 study Disastrous result
D-Level: Descriptive Theory
Describe Characteristics Taxonomy, Framework Dangers:
Over Aggregation Infinite regression
TeamTeam
TaskTask
TechnologyTechnology
ContextContext
ProcessProcess OutcomeOutcome
An Input-Process-Output Model ofGroup Outcomes from GSS Use
From Nunamaker, et al. (1991)
Endlessly Divisible Constructs
Characteristics of the team Structure
Leadership Style Power differences Norms Intra group process
History Cohesiveness Heterogeneity Etc. Etc.
Infinite Regression
Conclusion: The phenomenon can’t be studied
Better Conclusion: I’m asking the wrong question
The Experiment
Points to Ponder You don’t have to measure cause, you only
have to manipulate it. You must measure every effect You must have a theoretical explanation for
every effect
Experimental ModelExperimental Model
TreatmentTreatment11
TreatmentTreatment11
TreatmentTreatment22
TreatmentTreatment22
IdenticalSubjectPools
IdenticalSubjectPools
ResultsResultsResultsResults
ResultsResultsResultsResults
}} CompareCompare
Investigative InquiryInvestigative Inquiry
PopulationPopulation11
PopulationPopulation11
PopulationPopulation22
PopulationPopulation22
ResultsResultsResultsResults
ResultsResultsResultsResults
}} CompareCompareOneTreat-ment
OneTreat-ment
Investigative InquiryInvestigative Inquiry
PTAPTAMembersMembers
PTAPTAMembersMembers
Non-PTANon-PTAMembersMembersNon-PTANon-PTAMembersMembers
Loved Loved ItIt
Loved Loved ItIt
So-SoSo-SoSo-SoSo-So
}} CompareCompareEatAt
Joe’s
EatAt
Joe’s
Investigative InquiryInvestigative Inquiry
Non-PTANon-PTAMembersMembersNon-PTANon-PTAMembersMembers
PTAPTAMembersMembers
PTAPTAMembersMembers
So-SoSo-SoSo-SoSo-So
LovedLovedItIt
LovedLovedItIt
}} CompareCompareEatAt
Joe’s
EatAt
Joe’s
•PTA Causes Change in Taste?•Joe was charismatic principal.
Experimental Logic
If every thing else is the same, the difference MUST be caused by my treatments.
Science and Technology
You do not study technology You study the effects to which
technology can be applied
Science and Technology
Every PRESCRIPTION implies an underlying model of cause-and-effect
The Dangers of “Match” and “Fit” Theories
“The quality of the building depends on the fit between the plan and the purpose”
The Dangers of Match and Fit Theories
Every Match or Fit theory implies one or more underlying models of cause-and-effect
But does not bother to articulate them
Experimental Design
Construct Validity Statistical Validity Internal Validity External Validity
Construct Validity Am I measuring the construct I think I’m
Measuring? Thermometer to measure time? Theory drives measures
Statistical Validity
Are statistics interpreted meaningfully
Theory Drives Statistics
Internal Validity
Did my treatment really cause the difference I observed?
Threats to Internal Validity
Unfavorable Comparison Group receiving “Poor” treatment
stops trying
Threats to Internal Validity
Between-group competition:Group receiving the “poor”
treatment makes extra effort to excel
Threats to Internal Validity
The Hawthorne Effect:Paying attention to people affects
their performance.
Control for Hawthorne Effect
Group1Group1
ControlControlGroupGroup
Treat-Treat-MentMent
Pay Attention to Both GroupsPay Attention to Both Groups
Threats to Internal Validity
Novelty Effect: New situations stimulate
performance. Control: Longitudinal Study
Threats to Internal Validity
Maturation: Perhaps the effect occurred simply
because the subjects got older.
Control for Maturation
Group1Group1
ControlControlGroupGroup
Treat-Treat-MentMent
Measure HereMeasure Here
Threats to Internal Validity
History:Something happens during the
experiment that causes the effect
Control for History
Group1Group1
ControlControlGroupGroup
Treat-Treat-MentMent
Measure HereMeasure Here
Threats to Internal Validity Reactive Measures:Somehow the initial measuring
process causes the effect
Control for Reactive Measures
Group1Group1
ControlControlGroupGroup
Treat-Treat-MentMent
Measure HereMeasure Here
Threats to Internal Validity
Calibration:differences caused by shifts in
instrument calibration over the course of the study.
Control for Calibration
Group1Group1
ControlControlGroupGroup
Treat-Treat-MentMent
Measure HereMeasure Here
Classic Books
Campbell & Stanley Cook & Campbell
External Validity
To what population do my results apply?
Generalizability
Theory Drives:
Hypothesis Measures Treatments Statistics
Scientific Method Discover Phenomenon Theorize Hypothesize Fastest Falsifications Experiment Conclude Apply
Selling Your Science: Getting Published Introduction: Who cares? Theory: Says Who? Hypotheses: Prove it! Design: Are you sure? Results: Did you get it? Discussion: So What? Conclusions: Theory Good?
Truth
Powerful theory will outperform powerful statistics every time!
Truth There is No Perfect Study You must pilot your study
Truth No Theory is made or broken by a single study
Remember
Experiments without theories are meaningless
Remember
Data Have No Meaning except in reference to the
theory from which they spring