Post on 02-Jun-2018
8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
1/11
Interviews, Surveys, and the Problem of Ecological ValidityAuthor(s): Aaron V. CicourelSource: The American Sociologist, Vol. 17, No. 1 (Feb., 1982), pp. 11-20Published by: American Sociological AssociationStable URL: http://www.jstor.org/stable/27702491.
Accessed: 28/10/2013 15:50
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at.http://www.jstor.org/page/info/about/policies/terms.jsp
.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
.
American Sociological Associationis collaborating with JSTOR to digitize, preserve and extend access to The
American Sociologist.
http://www.jstor.org
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/action/showPublisher?publisherCode=asahttp://www.jstor.org/stable/27702491?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/27702491?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=asa8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
2/11
The
Problem
of
Ecological
Validity
11
tions that
a
classic
can
serve.
We
may
believe
that students'
minds
are
expanded
by
reading
Durkheim
without
our
having
to believe Durkheim has many true gener
alizations
about the
causes
of suicide.
George
Herbert
Mead
can
symbolize
what
is
distinctive
in
symbolic
interactionism
even
if
we
cannot
quite
figure
out
how to
test
the
hypothesis
of
the
independence
of
the
F
from
the
me,
and
to
turn
it into
a
puzzle
for routine
science.
And
orte
can
enjoy
the taste
of
Marx's
famous
passage
in The 18th
Brumaire
about
French
peas
ants
forming
a
vast
mass,
without
that
beauty being undermined when we find
some
regions
of
modern
France
where the
peasants
vote
Communist.
What
is
destructive
about
admiration of
the
classics,
then,
is
the
halo
effect,
the
belief that
because
a
book
or
article is
useful
for
one
purpose,
it
must
have
all
the
virtues.
INTERVIEWS, SURVEYS, AND THE PROBLEM OF
ECOLOGICAL VALIDITY*
Aaron
V.
Cicourel
University
of
California,
San
Diego
The
American
Sociologist 1982,
Vol. 17
(February):
11-20
Despite
the
fact
that
virtually
all
social
science
data
are
derived
from
some
kind
of
discourse
or
textual
materials,
sociologists
have
devoted little
time
to
establishing explicit
theoretical
foundations
for
the
use
of
such
instruments
as
interviews
and
surveys.
A
key
problem
always
has been
the lack
of
clear theoretical
concepts
about the
interpretation
of
interview
and
survey
question and answer frames. We lack a theory of comprehension and communication that can
provide
a
foundation for
the
way
that
question-answer
systems
function,
and
the
way
respondents
understand
them. The
paper
briefly
describes
the
possible
relevance
of
linguistic
and
cognitive
processes
for
improving
our
understanding
of
interviews
and
surveys.
The
theoretical
foundations
of
interviews
and
surveys
also
must
address
the
way
that
artificial
circumstances
become
necessary
to
guarantee
adequate study
designs.
These
artificial
circumstances
often
violate
ecological
validity,
or
the
way
interviews
and
survey
questions
are
constructed,
understood,
and
answered,
as
contrasted
with
the
way
that
field
notes
and
tape-recordings
of
natural
settings
are
used
to
address the
same
or
comparable
substantive
and
theoretical
issues.
Social scientists
have relied
on
inter
views
for
a
long
time.
There
is
little
reason
to doubt their value and utility formany
theoretical
and
practical
purposes.
There
exists
a
huge
literature
on
the
virtues
and
drawbacks
of interviews
that
use
open
ended
questions
and
surveys
that
use
close-ended
questions.
Yet there is
little
in
the
way
of
theory
that
would
link
inter
*
Presented
at
the
thematic section
Fact
or
Ar
tifact:
Are
Surveys
Worth
Anything?
held
at
the
1980
American
Sociological
Association
Meetings,
New
York,
August
27,
1980.
The other
speaker
was
Howard
Schuman,
taking
a
less critical view of
sur
vey
research. I am
grateful
to Michael
Cole,
Roy
D'Andrade,
and
Hugh
Mehan for
their
valuable
re
marks and
suggestions
on a
much
longer
first draft
of
the
paper.
[Address
correspondence
to:
Aaron
V.
Cicourel;
Department
of
Sociology; University
of
California,
San
Diego;
La
Jolla
CA
92037.]
views and
surveys
to
more
general
issues
of communication and
comprehension.
Those researchers who are convinced that
interviews and
surveys
are
basic
research
tools for the
sociologists
are
concerned
about
improvements
in
survey
design
and
use,
but
see
little
point
in
challenging
their
routine
use.
In this
paper
I
want to
suggest
a
few
cognitive
and
linguistic
issues
that
can
clarify
our
understanding
of
the
pro
cesses
and
mechanisms
underlying
the
use
of interviews
and
surveys.
I
also
want
to
suggest
some
theoretical
ideas
that
can
strengthen
the
ecological validity
of
inter
view and
survey
methods and
findings.
The
necessity
of
writing
a
brief
paper
does
not
permit
me
to
discuss old
issues
about
current
interview
and
survey prac
tices
that
I
hope
are
obvious
to
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
3/11
12
The American Sociologist
sociologists:
for
example,
the
way
that
preliminary
qualitative
interviews
nor
mally
precede
the construction
of inter
view schedules and surveys; and theway
that
pre-testing
with
in-depth
interviews
helps
to
create
questions
that
respondents
can
understand,
while
helping
to
create
answer
categories
that
reflect
the
thinking
of
respondents
and
not
simply
the
re
searcher.
Hence
I will
avoid
discussing
the
range
of
practices
that
are
necessary
to
insure
quality
control,
such
as
using
different
types
of
questions
on
the
same
topic,
exploring
the
significance
of
changes
in
wording,
and
other
procedures
too numerous to mention here.
Interviews
and
surveys
usually
occur
within
a
broader
context
of interaction
that
includes
complex
cognitive
and
lin
guistic
activities within
a
set
of
in
stitutionalized
and
emergent
socio
cultural
constraints.
The
questions
used
in
surveys
almost
always
are
framed
in
a
textual
format
that
displays
features
in
common
with
I.Q., aptitude,
and
reading
tests.
Virtually all social science
data
are
de
rived
from
some
kind of
discourse
or
tex
tual
materials
such
as
reports,
written
ac
counts
of
observations,
interviews,
audio
or
video
recordings
of
natural
settings,
historical
or
contemporary
documents,
minutes
of
meetings,
newspapers,
maga
zines,
and
the
like.
Questionnaires
mailed
to
respondents
presuppose
something
about
the
way
people
are
able
to
analyze
textual
materials.
For
example,
in
re
sponding
to
questions
in
a
reading
test,
the
respondent must utilize several sources of
knowledge
that
the
researcher
used
in
interviews
and
surveys,
therefore,
pre
sumes
a
theory
of
communication
and
comprehension
that
seldom
is
addressed
by
sociologists.
The remainder
of
the
paper
will
suggest
aspects
of communica
tion
and
comprehension
that
can
help
es
tablish
some
theoretical
foundations
for
interviews
and
survey
research.
Aspects ofmemory and comprehension
presupposed
in
surveys
and
interviews
Recent work
on
learning
and
reading
comprehension
(Bransford
et
al., n.d.)
reminds
us
that
our
research
instruments
stringently
control
the
information
re
sources
available
to the
subject
or
respon
dent.
In
tests and
questionnaires
normal
procedures presuppose an agreement
or
social
contract
between
researcher
and
respondent:
the
contract
does
not
permit
the
use
of
other
persons
(nor
the inter
viewer)
in
order
to
decide
on
the
meaning
of
the
question
and
the
appropriateness
of
a
response.
Normal
group
or
peer
sources
of
help
are
blocked.
The
test
or
question
naire
item
is
assumed
to
be
self
explanatory
or
self-contained.
These
con
ditions
contrast
with
the
possibility
of
consulting
a
friend
or
colleague
or
return
ing to a textual report, newspaper article,
book,
and
the
like,
during
or
after
an
ini
tial
reading
of
the
text.
The interview
and
survey
seek
to
re
strict
the
question
frame,
and
in
the
case
of
surveys,
the
answer
frame.
The
goal
is
to
restrict
the
question
in
such
a
way
as
to
anticipate
and
even
designate
(in
surveys)
the
range
of
responses
that
can
be
used.
The
aggregation
of
responses requires
a
few choices
that
either
are
formally
part
of
the questionnaire
or
are
imposed
on
open-ended
responses.
We
impose
infor
mation
processing
restrictions
on
the
re
spondent
because
they
enable
us
to
aggre
gate
and summarize
a
large
amount of
in
formation
in
a
fairly
succinct
way.
But
we
pay
a
price,
and
we
need
to
understand
the
costs
in
order
to
improve
the
reliability
and
validity
of
interview
and
survey
data.
We
need
a
better
understanding
of
the
role
of
memory
and
the
way
questions
are
comprehended.
Norman
(1973)
has
noted
several relationships between the organi
zation
of
memory
and
answers
to
ques
tions:
a
question
may
not
evoke
an
appro
priate
recall
if
it
is
phrased
differently
from
the
storage
format.
Norman
calls
this
the
paraphrase
problem
because
the
best
answer to
a
question
may
be
the
use
of
another
question
by
the
respondent
in order
to
clarify
what
is intended
by
the
original question.
Norman's
reference
to
memory
brings
up
the
problem
of
how
people
store
information
and combine
general
and
specific
sources of informa
tion
in
order
to
reveal
what
they
think
is
addressed
by
the
question
they
are
asked.
Norman
is
interested
in
the
pre-processing
that
occurs
before
an
answer to
a
question
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
4/11
The Problem
of
Ecological Validity
13
can
be
produced.
In
the
case
of
surveys
we are
faced with
the
paradox
that the
respondents
are
not
encouraged
to
pro
vide us with reasons or explanations about
their
answers,
yet
such
information
gives
us
clues
about
how the
question
was
un
derstood.
Norman
(1973)
states
that retrieval
of
information from
memory
requires
con
struction
by
the
respondent
because
of the
paraphrase problem.
It
is
difficult,
there
fore,
to
show
the
way
questions
and
an
swers are
articulated
by respondents
be
cause no
simple algorithm
can
be iden
tified that would enable
us
to
specify
a
sequence of instructions or steps or ac
tions that
directly
connects
questions
and
answers.
We
need,
therefore,
an
under
standing
of the
comprehension
process
and
the construction
of
responses
in
inter
views
and
surveys.
Aspects
of
language
presupposed
in
surveys
and interviews
I want to
suggest
a
parallel
between
as
pects
of modern
linguistic theory
and
sur
veys.
Language
can
refer
to
a
lexicon and
individual
words
as
a
set
of
carefully
con
structed
ideals that
can
be studied
and
described
independently
of
actual lan
guage
use
in
social
settings.
The
linguist's
syntax-based
theory
of
language provides
the ideal
structures
the
survey
researcher
needs
for
constructing
standardized
ques
tionnaires
whose forced-choice
responses
can
be
analyzed independently
of
the
way
persons
in
daily
life
actually
discuss
or
pose and answer questions of each other
within the
constraints of
daily
practices
in
socio-cultural
organizations.
The
linguist's
normative
theory
of lan
guage
describes idealized
prescriptive
and
proscriptive
rules
that
are
recipes
for de
ciding
what
are
socially acceptable
or
un
acceptable
sentences.
Survey questions
are
equally
normative because
they
occur
in
highly
restrictive
settings
that
have little
or
nothing
to
do
with
actual
discussions
or
practices
in
group
or
informal
organi
zational activities. But the
linguist's
and
survey
researcher's
idealized
language
structures and substantive
questions
about the
world of
opinions,
attitudes,
be
liefs,
and
moral
judgments
are an
integral
part
of
the
way
researchers and
the
public
governmental
agencies
conceptualize
and
interpret
the
world
around
them.
The paradox
we
face is that
our
surveys
and
interviews
only
indirectly
reflect
as
pects
of
the
daily
life
settings
of those
we
interrogate, yet
these
instruments
can
be
the
basis for the
development
of
policy
by
organizations
in
many
complex
nation
states.
If
we
recognize
that
questions
and
an
swers
are
speech
acts
(Austin,
1962;
Searle,
1969),
we can
make
some
sug
gestions
about
the
way
that
the
structure
of
language
can
improve
our
understand
ing of interviews and surveys. Speech act
theory
seeks
to
combine
the
analysis
of
the
propositional
content
of
an
utterance
with
its
illocutionary
force;
the intention
of
a
speaker
to
act on
the
world
by
the
use
of
a
promise,
assertion,
command,
and
the
like.
Speech
acts
enable
the researcher
to
establish
a
functional
meaning
for
an
ut
terance
by
the
way
they
are
classified
as
statements
about
the
world,
a
speaker's
act
on
the
world,
or
a
symbolic
represen
tation
of
an
event
in
the world.
A
meta
language
was
felt
to
be
necessary
for dis
cussion of
speech
acts.
The
way
in which
surveys
and
inter
views
are
conducted
presupposes
a
model
of
conversational
behavior that
has been
ignored
by
most
sociologists.
The notion
of
a
speech
act
model
has
been
extended
to
the
idea
of conversational
postulates
by
Grice
(1975),
and
derived
from his
more
general
notion
of
the
cooperative
principle.
The
principle
refers
to
a
kind of
directive to the speaker to formulate all
aspects
of his
or
her
utterance
in
a
way
that
will
permit
participants
of
a
conver
sation
to
facilitate
to
the
utmost
the
achievement
of
the
explicit
and
tacitly
agreed
upon
aims of
the conversation.
Grice
identifies
four
categories
that
are
designed
to
orient
the
speaker
to
be
as
informative
as
possible
but
not
more
in
formative
than
seems
necessary.
Nor
should
the
speaker
say
anything
believed
to
be
false
or
anything
that lacks
sufficient
evidence.
The
speaker
also is
to be rele
vant
and
to
be
brief
and
orderly
while
trying
not to be
ambiguous
or
obscure
in
his
or
her
use
of
expressions.
Notice
that
the
term
speaker
would
apply
to
both the
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
5/11
14
The American
Sociologist
interviewer and
respondent
in
the
case
of
surveys
and
interviews.
Another theoretical
issue
contained
in
Grice's work includes the notion of con
versation
implicature.
In
everyday
con
versation,
listeners
are
expected
to
make
inferences that
do
not
necessarily
follow
from the
premises
or
statements
given,
yet
the
statements
are
necessary
for the
com
prehension
of
the
discourse
or
text. The
notion
of
conversational
implicature
is
central
to
surveys
and
interviews
as
sub
sets
of
discourse
and textual
activities
be
cause
our
questions
require
that
the
re
spondent
go
beyond
the
information
given
in survey questions and presume that the
utterances
can
be
expanded
in
order
to
pursue
their
implications
and
derive
co
herence from what is said.
Speech
act
theory
can
help
us
under
stand
the
way
that
variations
in
the
textual
content
and
structure
of
interview
and
survey
questions guide
the
kinds of
in
terpretations
that will
be
made,
and how
these attributions of
meaning
will
influ
ence
the
construction of
a
response
or
the
selection of
an
option
in
a
set
of forced
choice
responses.
Some
aspects
of question-answer
systems
Question-answer
systems
deal with
a
sub-set of
speech
act
theory.
The
ques
tions
employed
are
for
themost
part
direct
attempts
to
elicit information.
In
everyday
English
we
often
use
indirect
speech
acts
and
our
questions
do
not
always
follow
an
interrogative
format.
Scheduled inter
views and surveys presume a respondent
who is
aware
of
the
general procedure.
A
specific
style
of
interrogation
is
used and
respondents
assume
a
response
stance
that
differs
markedly
from routine
every
day
conversation. But this
response
set
is
quite
similar
to
occasions
when
someone
is
being
interviewed for
a
job
or
is
taking
a
test.
The formal
aspects
of
conversation
out
lined
by
Grice
indirectly parallel
some
formal
properties
of
question-answer
systems
described
by
Harrah
(1973).
Whereas Grice
specifies
some
general
conditions
governing
all
conversations,
including
aspects
of the
reasoning
neces
sary
for
successful
exchanges,
Harrah
in
dicates
conditions
that
are
especially
rele
vant to
the
questioner.
In
Harrah's
(1973)
model
the
ques
tioner:
(a)
Is
presumed
to
know
what the
problem
is
about.
(b)
Knows
how
to
express
the
question
in
an
effective
manner.
(c)
Knows what
the
set
of
possible
alterna
tives
can
be.
(d)
Can
claim
that
one
of
the
alternatives
is
true.
(e)
Does
not
know
which alternative
he
wants
to
know.
(f)
Believes
the
respondent
can
help
him
if
particular
question
is
put
properly.
But
there
are
various
logics
of
questions,
notes
Harrah,
and
they
will
vary
according
to
the
social situation
in which
questions
are
used.
Thus
in
a
classroom
setting
a
teacher
puts
questions
to students
and
knows
the
answers
expected.
Harrah
de
scribes
some
of
the
conditions
of
a
Ph.D.
examining
committee
where
their
ques
tions
may
be
directed
as
much
to
each
other
as to
the
student.
In the
Ph.D.
ex
amination
the
knowledge
base of
the
re
spondent
is
presumed
to
be the
problema
tic
issue.
When
a
physician
asks
questions
of
a
patient,
the
knowledge
base of
the
respondent
may
be viewed
as a source
of
new
information.
But this
information
re
quires
particular
types
and
sequences
of
questions
and
answers.
The
patient's
an
swers
also
may
become
problematic
de
pending
on
the
kinds
of
attributions
made
to
the
patient
because
of
age,
mental
status,
social
position,
and
physical
con
dition and appearance, tomention a few
key
variables
that
affect
all
question
answer
systems.
Questionnaire
items
are
not
merely
in
dividual,
self-contained
texts,
but become
the
basis
for
inferring
macro-structures
that resemble
those
reported
by
re
searchers
working
on
text
comprehension
(Kintsch
and
van
Dijk,
forthcoming;
van
Dijk,
1972).
The
respondent
seeks
a
more
comprehensive
understanding
of
the
dif
ferent
questions
asked
despite
the
re
searchers
attempts
to randomize the pre
sentation
of
questions
that
are
linked
by
hypotheses
in the
research
design.
This
search
for
a
pattern
on
the
part
of
the
respondent
is
part
of
an
attempt
to
create
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
6/11
The
Problem
of
Ecological
Validity
15
an
explanation
that
goes
beyond
whatever
cryptic
information is
supplied by
the
interviewer
or
questionnaire.
The
respon
dent becomes an active participant in the
survey
or
interview
and
seeks
to
develop
his
or
her
own
hypotheses
about
what
is
going
on
and
what
intentions the
re
searcher
projects
by
the
kinds
of
ques
tions asked.
Discourse and
textual
analysis
as
comprehension
A
key
issue
in
the
study
of
comprehen
sion is
trying
to
estimate
what
the
respon
dent brings to the reading test, interview,
or
survey.
There
are
several
strategies
available for
studying comprehension.
In
unpublished
work
by
David
Rumelhart,
comprehension
by
subjects
is
measured
by
asking
them
to
read
a
story
line
by
line
while
indicating
what
is
happening
after
each
line.
An
unpublished
project
in
prog
ress
on
reading
comprehension by
Charles
Fillmore
and Paul
Kay
uses
reading
test
items
(that
strongly
resemble
survey
questions)
as
the basis for interrogating
the
child
about
his
or
her
understanding
of
the
text
of
each
test
question.
More
abstract
types
of textual
analysis
include
the identification of
topics
or
themes
and
their
continuity
over
a
large
textual domain
(Grimes,
1980).
A
large
and
growing
literature
exists
here
that
ad
dresses
textual
analysis
(Dressier,
1977;
Halliday,
1967).
I
will
not
attempt
to
re
view
current
work
on
the
analysis
of
texts
but
only
mention
the
broad
goal
of
iden
tifying those structures thatwould serve
as a
basis for
interpreting specific portions
and
general
aspects
of
a
text.
Particular
conventions
of
language
use,
such
as
fol
lowing
certain forms of
language
structure
when
writing
a
letter,
a
report,
in
structions
for
filling
out
the
necessary
pa
pers
for
a
bank
loan,
and
the
like,
generate
expectations
about how
topics
are
intro
duced,
developed,
and
terminated.
We
need to
know how interview
and
survey
questions
as
texts
are
interpreted by
the
researcher
and
respondents.
Current work
on
the
analysis
of discourse and textual
materials
can
help
us
develop
a
theoretical
foundation for
understanding
and
im
proving
interviews. The theoretical
foun
dations of
interviews and
surveys
must
include
the
way
that
artificial
circum
stances
necessary
to
guarantee
adequate
study designs can violate the ecological
validity
of
findings
vis-?-vis
what
takes
place
in
daily
life
settings.
Restating
the
ecological validity
issue
for
sociology
Social scientists
often
are
so
pre
occupied
with
creating
an
adequate
study
design
that
they
overlook
the
ecologi
cal
validity
problem:
Do
our
instruments
capture
the
daily
life
conditions,
opinions,
values, attitudes, and knowledge base of
those
we
study
as
expressed
in
their
natu
ral habitat?
Recent work
by
Cole
et
al.
(1978)
re
views the
history
and current
efforts
to
deal
with
the
problem
of
ecological
va
lidity.
Cole
et
al.
refer
to
the
revolutionary
impact
of
Wundt's
laboratory
psychology
for the
experimental
study
of mind in
arti
ficially
constructed and
simplified
envi
ronments.
Can
we
extend the
elegance
and control of laboratory research to field
settings?
In
sociology
it is
difficult
to
study
everyday settings
while
using
carefully
formulated
surveys.
Much of
survey
re
search
can
be viewed
as
the
application
of
rigorous techniques
to
data elicited
in
simplified
and
artificial social
envi
ronments.
In
sociology
and
psychology
ecological
validity
remains
a
minor issue
because
studies of
the social
organization
of
the
laboratory and the interview or survey
settings
often
are
relegated
to
minor
roles
when data
are
analyzed.
Psychologists
have demonstrated renewed interest
in
pursuing laboratory-derived problems
in
natural
settings
and
incorporating
every
day
tasks
into
laboratory settings.
In
sociology
this
would
mean
contrasting
the
way
interview
and
survey
questions
are
constructed,
understood,
and
answered,
with the
way
that field
notes
and
tape
recordings
of
natural
settings
are
used
to
address
the
same or
comparable
substan
tive and
theoretical
issues.
Psychologists
are
sensitive
to
problems
associated
with task
definition,
mental
overload,
and
possible
differences
in
the
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
7/11
16
The
American
Sociologist
way
remembering, thinking,
and
attending
to
activities
occur
in
laboratory
and
non
laboratory
settings.
But
they
do
not
study
the cultural definitions of everyday life
that
are
part
of the
laboratory
setting
and
that
are
invoked
necessarily
when
at
tempting
to conduct
controlled
research
in
non-laboratory settings.
Psychologists
are
not
sensitive
to
what
their
subjects
must
be
able
to
do
to
make
themselves
appear
as
normal
members
of
a
group
and the
larger
society
in
order
to
perform
ade
quately
in
an
experiment
or
in
an
every
day
setting.
The
subjects
and the
exper
imenter
both
must
rely
on
their
everyday
tacit knowledge in order to satisfy stated
and
unstated social conditions
that must
be
followed
if
the
research is
to
be
consid
ered
successful.
Sociologists
are
sensitive
to
the fact
that
many
problems
are
associated with the
way
questionnaires
are
administered,
coded,
and
organized
for
analysis.
But
they
are
insensitive
to
the
information
processing
problems
associated with
these
tasks.
Because
so
many
surveys
are
done
in the
same
culture in which the re
searchers
also
are
native,
and
because
we
gradually
have
socialized
our
respondents
to
be
fairly
docile
to
the
demands
of
sur
veys,
especially
since
everyday
life
cir
cumstances
often force them
to
submit
to
such
activities,
we
have little
knowledge
about
the
social
practices
of
survey
re
search
within
field
settings
and
within
re
search
centers
where
the
analysis
takes
place.
When
we
administer
surveys
in
other
cultures
we
incorporate
natives who
have been trained in the same method and
who
can
tacitly negotiate
the cultural
dif
ferences.
During
everyday
interaction
the
mem
bers
of
a
group
who
routinely
discuss
political,
economic,
and
social
events
are
sensitive
to
group
resources
of informa
tion
and
the
interpersonal
constraints that
are
imposed
on
exchanges,
and also
are
aware
of
the
knowledge
limitations of the
members
of the
group.
The
ecological
va
lidity
issue addresses
the
extent to
which
responses
to
interview
and
survey
ques
tions
reflect
or
represent
the
daily
actions
of
a
collectivity.
We
must
compare
the
way
collective discussions
about
topics
covered in
interviews
and
surveys
parallel
or
differ from
the
way
such themes
are
presented
in
the
formal
setting
created
by
research
goals.
A partial examination of the ecological
validity
issue
can
be found
in
recent work
by
Schuman
(1966),
and Schuman
and
Presser
(1977;
1977-1978; 1979;
forth
coming)
where
they
show that
changes
in
the
wording
of the
questions
often
lead
to
changes
in
response
patterns.
The work
by
Schuman
and
Presser also
contains
valuable
information
on
differences
in
the
use
of
open
and closed
questions.
The
authors
(forthcoming:9-10)
note
that
if
re
spondents
are
given
a
question
about
which they know nothing, many will an
swer
the
question
if
there
is
no
explicit
don't
know
option.
Many respondents
are
willing
to
admit
ignorance.
Respon
dents
also
are
said
to
make
an
educated
(though wrong) guess
about
a
topic
de
spite
their
being
uninformed
about
the
issue. The authors
note that
these
re
sponses
are
like
non-attitudes
in
the
sense
that
there
probably
was
no
prior
thought
about
the
attitude before
pre
sented by the interviewer.
The studies
by
Schuman
and Presser
seek
to
resolve
possible
problems
in
inter
viewing
and
surveys
as a
way
of enhanc
ing
their
reliability
and
validity.
These
studies,
though
limited
in
scope
and
depth,
are
valuable contributions
to
the
minimal work
that
has
been
done
by
re
searchers
working
within
social science
who
strongly
support
these methods
in
their
existing
form.
Many
serious
prob
lems of
reliability
and
validity
remain
be
cause of normal practices that are devoid
of
adequate
concern
for theoretical
foun
dations.
For
example,
we
restrict
our
col
lection of
information
to
a
few
categories
in
order
to
restrict
the
number
of
compari
sons
we
have
to
make. The
respondent's
knowledge
of the world
is
not
a
problema
tic
issue.
In
a
laboratory study
or
in
sur
veys
with
fixed-choice
questions, subjects
or
respondents
bring categorical
mech
anisms
with
them,
but
the
actual
pack
ages
of information
they employ
must
be
either recoded or tailored to the
particular
conditions
of
the
experiment
or
survey
question
and
forced
choices
provided.
The
range
of
speech
acts
becomes
se
verely
restricted
in
survey
research.
The
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
8/11
The
Problem
of
Ecological
Validity
17
respondent's
cultural
resources
available
for
searching
one's
memory
are
con
strained,
and
this
limits the
respondent's
ability tomake comparisons. The survey
question
introduces co-variation
by
the
wording
of
the
question.
The
following
examples
are
from
a
national
survey
by
E.
C.
Ladd,
Jr. and
S.
M.
Lipset
(as
cited
by
S.
Lang
(1978)).
The
questions
were
sent
to
college
and
university professors.
(1)
The
statements
below relate
to
teaching
and
student
performance.
Does each
correctly
reflect
your per
sonal
judgment?
(i)
The
students
with
whom
I
have
close
contact
are
seriously underprepared
in basic
skills?such
as
those
re
quired
for
written and
oral communi
cation.
(a)
Definitely
yes
(b)
Only partly
(c)
Definitely
no
(2)
Grade
inflation
is
a
serious academic
standards
problem
at
my
institution.
(a)
Definitely
yes
(b)
Only partly
(c)
Definitely
no
(3) American higher education should ex
pand
the
core
curriculum,
to
increase the
number
of
basic
courses
required
of all
undergraduates.
(a)
Definitely
yes
(b)
Only partly
(c)
Definitely
no
The
questions
assume
a
sample
of
re
spondents
familiar
with
the
content
of
the
items
(in
the
present
case
American
col
lege
and
university
professors)
as
well
as
with the
meaning
of
definitely
yes,
only
partly, and definitely no. The questions
and the
responses
reflect
a
co-variation
that
enables
a
reader
to
parse
the
question
easily.
The
answers
permit
easy
aggrega
tion
so
that
these
and
other
questions
can
be
cross-tabulated with
size
of
school,
age
of
respondent,
discipline
or
field of
study
and
the
academic
rank
of
the
respondent.
In
the
present
case
these
questions
were
mailed
out
to
the
informant.
There
was
no
way
for
the
respondent
to
obtain
informal
clarification
about
questions
from
the
interviewer. The
problem
solving
aspects
of
answering
a
fixed-choice
question
are
severely
limited.
There
were
no
open
ended
questions
to
make
it
possible
to
ob
serve
some
elements
of
a
problem-solving
strategy
at
work
(including
the
limitations
of the
respondent's
knowledge
base)
as
the
interviewee and
interviewer
negoti
ated the questions and answers. The
questions
on
the
basic
skills of
students,
grade
inflation,
and
the
idea
of
expanding
the
core
curriculum attribute
an
expertise
to
the
respondent
that
cannot
be
chal
lenged.
There
is
no
possibility
of
exploring
the individual
experiences
of
teachers
within
the
same
subject
area
much
less
across
disciplines.
Nor
can
we
distinguish
respondents
by
the level
of
college
or
uni
versity
classes
they
teach.
We
have
no
information
on
the
background
of their
students.
The
categories
created
by
the
re
searcher
must
be
negotiated
individually
by
each
respondent.
But the researcher's
categories
provide
ready-made
classes
and the
response
set
generates
automatic
criteria for
deciding
class
membership.
It
is difficult
to
interpret
the
meaning
of
these
personal
judgments
vis-?-vis
the
ex
periences
and
knowledge
base of
a
pre
sumed
group
of
experts.
The
expertise
is an automatic creation of identifying
a
population
known
as
being
college
and
university
teachers.
The
responses
we
obtain,
however,
remain
ambiguous
per
sonal
judgments.
The
ecological
validity
of
the
response
is
not
addressed.
Class
membership
and
interview
and
survey
categories
An
important
aspect
of
surveys
and
open-ended
interviews
is the
extent
to
which a concept or class membership is
presupposed
in
the
way
we
elicit
infor
mation
from
respondents.
Our
ability
to
perceive,
remember,
and talk
about
some
object
or
event
as an
instance
of
a
class
or
concept
we
are
presumed
to
know
is
fun
damental
to
the
way
we use
surveys
and
open-ended
questions.
Public
and
private
bureaucracies
so
cialize
their
employees
to
the
use
of
categories
that
can
subsume
a
variety
of
activities
under
identifiable
classes
and
thus confer a
stability
on the world that
enables
them
to
go
beyond
the
informa
tion
given.
We
negotiate
the
assignment
of
an
object
or
event
or
some
aspect
of in
formation
to
a
class
initially
on
the
basis of
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
9/11
18 The American
Sociologist
expectations
of its
perceptible
elements,
and
then
begin
to
infer
some
of the
ob
ject's
or
event's
nonperceptible
attributes.
This is like saying thatwe construct the
identification of
typical
features that
en
able
us
to
claim
class
membership
for
an
object
or
event,
or we
claim
that
certain
features
suggest
one
or
more
possible
classes that
trigger
a
search
for additional
elements
that
enable
us
to
choose
among
several
possible
candidates of
classes. In
our
perception
of
speech,
for
example,
we
may
be forced
to
imagine
or
recall
or
search
for
information
that
extends be
yond
conventional
or
dictionary interpre
tations of what is said because of the so
cial
setting.
Our
ability
to
make
the
non
perceptible
visible
and
hence
integral
to
the
invocation of
a
class
or
concept
is
a
basic
process necessary
for
all
social
in
teraction
and
bureaucratic
practices.
A recent
paper
by
Medin
and
Smith
(in
press)
distinguishes
between three views
of
concepts.
The
first
or
classical
view
requires
that
a
concept
have
common
properties
which
become
necessary
and
sufficient to define the concept. Every
member
of
a
class
can
be
specified by
the
properties they
all
must
possess
through
a
single
description
of
all
members.
Medin
and
Smith
note
that
attacks of this view
revolve
around
the
properties
of
descrip
tion that
must
be
true
of
all
members.
The
second
view
of
concepts
is called
the
prototype
or
poly
the
tic
position.
The
focus
of
this
view
is
on
the
way
instances
of
a
concept
can
vary
in
th?
degree
to
which
they
all
share
certain
properties.
This view says that the different instances
can
vary
in
the
extent to
which
they
will
embody
the
concept.
A
single
description
of
some
may
again
suffice,
but the
prop
erties
of the
description
are now
only
true
for
most
but
not
all
of
the
members.
Some
instances
of
the class
will
possess
more
of
the
critical
properties
than
will
other
in
stances.
Those
instances
possessing
more
critical
properties
are
said
to be
more
rep
resentative
of
the
concept
in
question.
The third
view
described
by
Medin
and
Smith is called an
exemplar
notion.
This
view
of
a
concept
states
that
no
single
representation
exists
for
an
entire
class
or
concept,
but
instead
only
specific
repre
sentations
of
the
class's
exemplars
occur.
The
example
given
by
Medin and
Smith
for
this
third view is
that
of
the
class
of
persons
who
might
be
called
suicidal.
The example of persons who might be
suicidal is used
by
Medin and Smith
to
compare
the three views
briefly.
They
note
that
the
classical
view failed when
used
by
clinicians because
no
necessary
and
sufficient
common
properties
could
be found
to
define
all
people
who have
suicidal
tendencies.
The
polythetic
view
fails, say
Medin
and
Smith,
because
it falls
short of
revealing
how
someone
decides
that
a
particular
person
is
suicidal.
Fol
lowing
a
suggestion
by
Twersky
and
Kahneman (1973), Medin and Smith con
tend
that
because
clinicians
are
not
likely
to
use a
single
description
of
all
persons
with
suicidal
tendencies,
they
might
in
stead
make the
decision
about
someone
being
suicidal
by comparing
the
individual
to
other
persons
known
to
be suicidal.
The
exemplar
view
would
result
in
the
class
of
people
with
suicidal
tendencies
being
rep
resented
by
separate
descriptions
for
various
people
known
to
be members
of
the class of suicidal persons.
Two
key questions
raised
by
Medin
and
Smith here
are
as
follows:
(1)
Is
it
possible
to
have
a
single
or
unitary
description
for
all
members
of the
class?
(2)
Can
we
say
that
all
of the
properties
specified
in
a
unitary description
are
true
of
all
members
of the class?
According
to
Medin
and Smith
the
classical
view
would
answer
both
ques
tions
with
an
affirmative
response,
while
the polythetic view would say yes to (1)
and
no
to
(2).
The
exemplar
view
would
say
no
to
(1)
and consider
(2)
to
be
irrelevant.
When
we use
fixed-choice
question
naire items
the
respondents
are
expected
to
be
able
to
recognize
the
classes
of ob
jects
stated in each
item
as
self
explanatory.
This
expectation
derives
from the
assumed
pretesting
of
each
questionnaire
item
prior
to
sending
out
or
utilizing
the final
questionnaire.
The
ex
tent to
which
the
respondent
possesses
the
necessary
knowledge
base
in
order to
answer
the
question
is
seldom
a
relevant
issue. The
possibility
that the
concepts
or
classes
presented
to
the
respondent
may
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
10/11
The
Problem
of
Ecological Validity
19
not
be
defined
clearly
in his
or
her
mind
is
also
seldom
an
issue.
In
both
cases
the
forced-choice
nature
of
the
survey
guar
antees an adequate response so long as
the
respondent
is
willing
to
check off
one
of
the
choices
available. The researcher's
versus
the
respondent's
conception
of
a
particular
class
or
concept
is
presumed
to
be
resolved
by
the
pretest
done
to
the
final
questionnaire.
The
issue of
possible
social
classes used is
not
directly
testable. What
is
testable is
the
way
different
respondents
can
be
distinguished
by
some
measure
of
social
class
as
determined
by
the
way
fixed-choice
questions
are
constructed,
answered, and coded.
Having
created
several
social
classes
by
one
set
of
ques
tions,
we
examine
other
questions
an
swered
differentially
and
can
attribute
the
differences
to
social
class
membership.
Surveys
are
presumed
to
be
hypothesis-driven
and
hence
a
way
of
testing
theory.
But
the
technical
aspects
of
the
instrument
make
it
difficult
to
clarify
theoretical
concepts
or
classes
said
to
have
motivated
the
use
of
the
survey.
Theoretical concepts are subservient to
the
mechanics
of
creating
and
imple
menting questions
and
their
coding.
The
discourse
and textual
processes
and
mechanisms that
provide
the
theoretical
basis
for
surveys
remain
unexamined,
as
are
the
social
constraints
and
practices
of
the
society's
social
stratification
system
that enable
the
researcher
to
utilize
such
an
esoteric
and
indirect instrument
to
learn
about
the
everyday
activities and
beliefs of the
members
of
a
group.
Conclusion
Part
of
the
paper
has
been critical
of
survey
research.
I
have
slighted
several
issues.
For
example,
conscientious
survey
researchers
seek
a
form of
quasi-experi
mentation
with
questionnaire
items
in
which
a
particular
question
frame
is
re
tained but
a
particular
word
(or
perhaps
phrase)
is
altered.
Important
differences
often
are
found
because
of
these
changes
(cf.
Schuman and
Presser,
1981),
even if it
is
not
always
clear what
sort
of
reasoning
we
should
attribute
to
the
respondents.
What
is
important
for
the
survey
re
searcher
is
the
patterning
that
occurs or
emerges
that
gives
us more
confidence
in
the
overall
survey.
When
the
same
ques
tions
are
used
across
different
groups
or
with the same group at different times,
and
similar,
or
the
same,
patterning
emerges,
then
the researcher
feels consid
erably
more
confident
that
the
question
naire is
reflecting something significant
about
the
respondents'
opinions,
atti
tudes,
or
beliefs.
Knowing
that
some
identifiable
group
was
more
in
favor
of
some
action
or
law
this
year
than last
year
is
part
of the formal
patterning
that is
sought.
There still
can
be
problems
here
when
particular
researchers
throw
out
items that do not seem to work, but the
general
idea
is
to
avoid
using question
naire
items
as
if
responses
on a
given
oc
casion
can
be
treated
in
some
absolute
way.
The
goal
is
to
look
at
some
group
or
category
relative
to
other
groups
or
categories
at
specific
periods
of
time,
and
not
how
many
respondents
endorse
a
given
item
at
a
particular
time.
The
survey
researcher
seeks
to
control
the
way
a
data
base is
generated
by
creat
ing restricted conditions under which in
formation is
to
be
elicited,
coded,
and
an
alyzed.
The
conditions
simplify
and
dis
tort
the
daily
life activities
of
those
groups
and
institutions
we
seek
to
understand
and
predict,
but
the
controls and
restricted
data
base
are
highly
valued
by
many
social
scientists
because
they
foster
a sense
of
scientific
rigor
in
our
research.
Another
source
of
control
in
survey
re
search
can
be
found
in
the
enormous
ad
vances
that
have
occurred
in
sampling
theory and the researcher's ability to sam
ple
different
respondents.
What is
more
difficult is
the
sampling
of behavior. In
the
case
of
voting
behavior
we
find
a
fairly
close
correspondence
between what
people
say
in
response
to
a
questionnaire
item and the
way
they actually
vote.
But
other
topics
do
not
fare
as
well,
and
some
not
well
at
all. We
are
not
clear about the
behavior
or
activities
the
survey ques
tionnaire items
are
said
to
index.
People
are
not
very
accurate in
describing
their
own behavior when asked to
respond
to
direct
questions.
The
primary
difficulty
remains the
absence
of
strong
theories.
Instead of
using
strong
theories
we
invari
ably rely
on
the detection
of
patterning
in
This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
11/11
20
The
American
Sociologist
survey
responses
in
order
to
guide
us
in
making
theoretical
explanations
after
the
fact.
Theory
seldom
guides
social
re
search explicitly; we depend
on
research
findings
to
decide
which
theoretical
con
cepts
seem
appropriate.
Sophisticated
survey
researchers
surely
can
find answers
or
replies
to
the
is
sues
I
have
raised
and
will
point
to
the
use
of
other
sources
of data
or
additional
checks
or
strategies
that
I
have
not
cov
ered in
the
paper.
W?
need
strong
theories
to
decide
whether
a
particular
method
and the data
it
yields
tells
us
something
worth
know
ing.We all are forced to deal with the
same
problem
of
interpretation
regardless
of whether
we use
surveys,
census
mate
rials,
vital
statistics,
extensive
interview
ing, participation
observation,
or
audio
or
video
tapes.
The
interpretation
issue is
seldom the focus of
survey
research,
much
less
any
other
type
of
research
in
sociology.
Notions like limited
capacity
processing,
comprehension
of discourse
and textual
materials,
and
language
use
in
socially constrained contexts,
remain
pe
ripheral
topics
in
sociology.
Yet
they
ad
dress the
interpretation
issue
in
several
explicit
ways.
Can
we
afford
the
con
venience of
ignoring
these
issues?
REFERENCES
Austin,
John
L.
1962
How
To Do
Things
With
Words. Cam
bridge,
MA:
Harvard
University.
Bransford,
John
D.,
Barry
S.
Stein,
Tommie S.
Shelton,
and
Richard A.
Owings
n.d.
Cognition
and
adaption:
The
importance
of
learning
to
learn.
Unpublished
manuscript.
Cole,
Michael,
Lois
Hood,
and
Ray
P.
McDermott
1978
Ecological
niche
picking:
Ecological
inva
lidity
as an
axiom
of
experimental cognitive
psychology.
Unpublished
manuscript.
Dressier,
Wolfgang
U.
(ed.)
1977
Current
Trends in
Textlinguistics.
Berlin,
NY:
de
Gruyter.
Grice,
H.
Paul
1975
Logic
and
conversation.
Pp.
41-58 in
P.
Cole and
J.
Morgan
(eds.),
Syntax
and
Semantics.
New
York:
Academic.
Grimes,
Joseph
E.
1980
Context structure
patterns.
Paper
pre
sented at the Nobel Symposium on Text
Processing,
Stockholm,
August.
Halliday,
Michael
A. K.
1967
Notes
on
transitivity
and theme
in
En
glish.
Journal
of
Linguistics
3:199-244.
Harrah,
David
1973
The
logic
of
questions
and
its relevance
to
instructional
science. Instructional Sci
ence
1:447-467.
Kintsch,
Walter and
Teun A.
van
Dijk
forth-
Toward
a
model of
text
comprehension
coming
and
production. Psychological
Review.
Lang, Serge
1978 The
professors:
A
survey
of
a
survey.
The New York Review of Books 25:38-42.
Medin,
Douglas
and Edward
Smith
n.d.
Representation
and
processing
of
lexical
concepts.
Unpublished
manuscript.
Norman,
Donald A.
1973
Memory,
knowledge,
and
the
answering
of
questions. Pp.
135-165 in
R.
S.
Solso
(ed.),
Contemporary
Issues
in
Cognitive
Psychology:
The
Loyola
Symposium.
Washington,
DC:
Winston.
Schuman,
Howard
1966
The
random
probe:
A
technique
for
evaluating
the
validity
of closed
questions.
American
Sociological
Review
21:218-222.
Schuman, Howard and Stanley Presser
1977
Question
wording
as an
independent
vari
able
in
survey
analysis. Sociological
Methods
and
Research
6:151-170.
1977-
Attitude
measurement
and
the
gun
control
1978
paradox.
The Public
Opinion
Quarterly
41:427-438.
1979
The
open
and
closed
question.
American
Sociological
Review 4:692-712.
1981
Questions
and Answers
in Attitude Sur
veys:
Experiments
on
the Effects
of
Ques
tion
Form,
Wording
and Context.
New
York:
Academic Press,
forth-
Public
opinion
and
public
ignorance:
The
com- fine line between attitudes
and
non
ing
attitudes.
American Journal
of
Sociology.
Searle,
John
R.
1969
Speech
Acts.
Cambridge:
Cambridge
Uni
versity
Press.
Tversky,
Amos and
Daniel Kahneman
1973
Availability:
A
Hueristic
for
judging
fre
quency
and
probability.
Cognitive
Psy
chology
5:207-232.
van
Dijk,
Teun
A.
1972
Some
Aspects
of Text
Grammers.
The
Hague:
Mouton.