IS 4800 Empirical Research Methods for Information Science

Class Notes April 4, 2012

•quantitative•Descriptive•demonstration•correlational•experiments

•qualitative•ethnography•case studies

•mixed

Overview of Research Methods

Steps in the research process

• Identify a phenomenon of interest

• Iterate:– Investigate current state of knowledge (lit. review ?)– Narrow down your interest to a research question or

hypothesis

• Identify research method to employ (survey, experiment, ethnography, case study)

Steps in the research process (cont.)

• Operationalize research question or hypothesis– Define the source of your data

• Sample population and recruitment method (if relevant)

• And/or the location/activities to be observed

– Define variables and/or data collection methods and instruments

– Identify how the analysis will be carried out

----YOU NOW HAVE A RESEARCH PROPOSAL ------

Steps in the research process (cont.)• Carry out your observations

• Analyze your data

• Draw conclusions, write up the results

Example: Freshman retention – a major concern of every University !!

A quantitative study: midterm exam question

A qualitative study (we will discuss that)

Quantitative vs. Qualitative

Goals Kinds of data Data collection methods Kinds of analyses Kinds of explanations When used in system development

lifecycle?

Goals of qualitative research

• Identify/compare patterns of behavior

• Analyze beliefs/subjective experience– How people view a situation or experience– Distinguish multiple perspectives

• Compare beliefs/self-reports to actual behavior

• DESCRIBE a complex phenomenon

• EXPLAIN why things happen

8

Case studies of failed system development or deployment

Ethnographic studies of workgroup practices prior to introduction of new technology

Examples of qualitative studies

9

• Case study research•Explore in depth one activity or project•Limited in time and place•Data collection by observation, interviews, artifacts . . . •Goal: tell a coherent “story” with lessons learned•Most common methodology for IT empirical studies

• Ethnography•Observation in natural setting•Observer may become part of the group to experience directly how its members interact•Goal: identify patterns of interaction (power structures, problem-solving/goal achievement)

Some qualitative methods in brief

10

• Grounded theory research• Data collection from artifacts and/or interviews• Develop a set of categories and a model telling how they relate to each other• Goal: explain the meaning of what is observed• Involves an bottom-up iterative process of data collection/theory formation

Some qualitative methods (cont.)

+ Systematic rules and procedures already worked out, and can be followed+ Traditional, accepted as “proof”- Closed-ended questions may lead to ignoring important factors and relationships- Quantitative methods cannot handle phenomena that are difficult to turn into variables

Advantages/disadvantages of quantitative studies

Lecture 1 - Introduction 12

+ more innovative and creative

+ capable of addressing issues that do not lend themselves to being described by variables

- conclusions may be less credible

Advantages/disadvantages of qualitative methods

13

Pragmatic philosophy – find out whatever you can using whatever methods are possible

Involves both qualitative and quantitative elements (at least 2 stages of research)

Advantages/Disadvantages

+ combines structure and flexibility- requires more time and resources

Mixed Methods

14

Pattern 1: “instrument” data followed by in-depth interview to get insight on the reasons for the observed relationships and capture any insights you

overlooked in study design

Examples of Mixed Method Designs

Pattern 2: exploratory study followed by survey or experiment to generalize the results – representative of a long-term research program

•Studies of computer/supported learning

•Studies of IT impacts in medicine

•Computer-supported collaboration (in general)•Case study or ethnography for groupware•Grounded theory study of chat groups

Applications of the Methods Which Method would you choose?

Goals of qualitative research

• Identify/compare patterns of behavior

• Analyze beliefs/subjective experience– How people view a situation or experience– Distinguish multiple perspectives

• Compare beliefs/self-reports to actual behavior

• Ideally, propose a model to describe or explain

What is a model ??What counts as evidence ??

Elements/types of qualitative models

• Define a taxonomy – useful aggregation of data– Rogers’ adopter categories

• Innovators

• Early adopters

• Early majority

• Late majority

• Laggards

Elements/types of qualitative models

• Define properties that affect outcomes– Perceived attributes of innovation that help explain

their different rates of adoption• Relative advantage

• Compatibility

• Complexity

• Trialability

• Observability

• Explains individual’s likelihood of adopting

Elements/types of qualitative models• Identify a relationship of interest

– Rogers: homophily/heterophily and diffusion• Homophily is the degree to which two or more

individuals who interact are similar in certain attributes such as beliefs, education, socioeconomic status, etc.

• Although homophily enables better communication, innovation requires heterophily, at least regarding knowledge of and experience with an innovation, and it is likely that heterophily in that area co-occurs with other important differences.

• This explains why some groups are slower to adopt new behaviors

Elements/types of qualitative models

• Characterizing a process as a sequence of steps– Rogers: the innovation/diffusion process

• Knowledge

• Persuasion

• Decision

• Implementation

• Confirmation

• Characterizing the “shape” of a process– Rogers: the S-shaped curve

21

Identify/define the variables (and their coding)Design the “instrument” – a measurement process or

technique

Types of instruments:a survey (paper, phone, Web)a form for observer to fill in (experiment or field

study)a prototype system (with interaction capture)

Field testing/validating instruments part of quantitative methods

Collecting data for quantitative studies

Qualitative Data and Collection Methods

• Direct observation– Participant observation

• In-depth interviews– Focus groups

• “Artifacts” – usually text or Databases

Direct Observation

– May be in person or use audio or videotape, observe through a 1-way mirror

– Unlike participant observation, often focused on specific events (how many, how often, by whom, observe patterns – for example, interruptions at a meeting)

What to observe

• Spatial relations

• Activities

• Communication– Verbal– Other

• Tasks– How work is allocated

How to be an effective observer

• Preparation

• Stay in the background

• Be factual and objective in your notes– (interpretation comes later)

• Taking notes:– Hand written usually– Type in to computer later

• EXPANDING NOTES (ASAP)

“(Participant) observation”: in natural setting

• “Participant” observation occurs when you interact casually and/or form relationships with informants

• How much you actually “participate” depends on the goals of the study.

Participant observation

• Advantages:– Offers insights into complex behavior– Identify the “right questions” for further study– Verify/correct self-reports

• Disadvantage: – Time consuming– Data collection is difficult– Problem of subjectivity

How to operationalize

• Field notes– Text– Diagrams, maps– Can result in numerical data

• Interviews (interviewer more clueful)

• Focus groups (facilitator more clueful)

What to observe

• Spatial relations

• Activities

• Communication– Verbal– Other

• Tasks– How work is allocated– See Table 3 in reading

Ethics

• Do not disrupt the activity your are observing

versus

• Do not mislead

• No formal rules about disclosing your role as a researcher when engaging in casual conversation – article suggests a point where you want to ask specific question

• Disclosure includes: right of refusal, confidentiality

Protecting confidentiality when data is unique

• Separate identify info from field notes entered into the computer

• People, organizations/companies, should be given fictitious names

How to be an effective observer

• Preparation

• Stay in the background

• Be factual and objective in your notes– (interpretation comes later)

• Taking notes:– Hand written usually– Type in to computer later

• EXPANDING NOTES

Tips

• Leave space

• Take notes strategically

• Use abbreviations

• Cover a range of observations:Body language, etc.

Tips

• Leave space

• Take notes strategically

• Use abbreviations

• Cover a range of observations:Body language, etc.

Participant observation: in natural setting

• “Participant” observation occurs when you interact and/or form relationships with informants

• Demanding and time-consuming

• How much you actually “participate” depends on the goals of the study.

• Subjects may “forget” you are a researcher

How to operationalize direct/participant observation

• Field notes– Text– Diagrams, maps– Can result in numerical data

• Interviews (interviewer more clueful in P.O.)

• Focus groups (facilitator more clueful in P.O.)

“Water cooler” effect

Participant observation

• Advantages:– Offers insights into complex behavior– Identify the “right questions” for further study– Verify/correct self-reports

• Disadvantage: – Time consuming– Data collection is difficult– Problem of subjectivity

Ethics of direct observation/ participant observation

• Do not disrupt the activity your are observing

versus

• Do not mislead

• No formal rules about disclosing your role as a researcher when engaging in casual conversation – some authors suggest a point where you want to ask specific question

• Disclosure includes: right of refusal, confidentiality

Protecting confidentiality when data is unique

• Separate identify info from field notes entered into the computer

• People, organizations/companies, should be given fictitious names

In-depth interview/focus group

• Probes the interviewee(s) views of the phenomenon of interest

• Interviewer/facilitator should be neutral

• Data collected: transcript, audio/video recording, notes

In-depth interview/focus group

Interviewer should:

• Start with some open-ended questions

• Follow up by asking “how” and “why”

• Keep the discussion on track

Documents• Memos and meeting notes

• Transcripts of conversations or speeches

• Manuals and policy handbooks

• Newspapers and magazines

• Internet-based research– Email– Web sites– Blogs

• Especially important in case studies

Qualitative Data Analysis by John V. Seidel

• Description of how to go about analyzing transcripts of interviews, documents, and/or field notes.

• Focus on “coding”– First identify “events”

– Assign terms that represent concepts of interest

– Organizing codes into a scheme

– Building qualitative models using the coding scheme as the model vocabulary

• Focus on iterative nature of QDA

Two perspectives on coding

• Objectivist perspective– Condensed representation of facts– Can be subjected to hypothesis testing– Strong burden of consistency/completeness

• Heuristic perspective– Signposts pointing to things you care about– Foundation for further analysis

Elements/types of qualitative models

• Examples from Rogers’ theory of innovation diffusion– VCR’s– Cell phones– Metric system– Seat belts in cars– Dvorak keyboard

Three analogies to explain this

• Jigsaw puzzle analogy

• A little data and a lot of right brain

• Multi-threaded DNA (patterns among the patterns)