Conducting Experiments, Survey Construction and Data Collection

RCS 67407/18/05

Practical Considerations of Research This lecture will focus on some of the

practical aspects of conducting research studies such as: How do you select research participants What should you consider when deciding how to

manipulate an independent variable How do you measure a variable What should you consider when constructing a

survey What should you consider when collecting data How do ensure control in your study

Selecting Research Participants Define your participants

People with Aphasia Women with Breast Cancer Rats with 5 Legs

Define your population Everyone in the United States People in RCS 6740 Consumers at “The Center” (CIL)

Selecting Research Participants Cont. Define your sampling technique

Probability sampling Random sampling Stratified random sampling Systematic sampling Cluster sampling

Non-probability sampling Quota sampling

Determine your sample size Remember, larger sample sizes may result in

more significant data

Manipulating the Independent Variable Construct the Operational Definition of your variables

Exercise as defined by jogging for 20 minutes, 3 times a week, for 6 weeks

Depression as defined by the Geriatric Depression Inventory Set the Stage

Provide the participants with Informed Consent Usually a document that entails the name of the study, purpose(s)

of the study, potential benefits and risks, funding sources, contact information, and a statement that they can withdraw at any time

Explain to the participants in more detail why the study is being conducted Rarely are hypotheses given to participants

Tell them the study looks at employment when the hypothesis looks specifically at bias towards the employment of people with disabilities

Can be completely truthful and precise Deception may be used

When deception is used, it is imperative that participants are told the truth during a debriefing at the conclusion of the study

Types of Manipulations Straightforward Manipulations

Presenting written, verbal, or visual material to participants

Examples: Having participants read material on substance

abuse prevention Read a passage to a participant and ask them

questions later to test recall (memory) Have a computer program that presents different

words that someone needs to pronounce to measure the level of aphasia

Most research is done in a straightforward manner Researchers can vary the difficulty of material,

motivational levels, the way questions are asked, or the characteristics of the participants

Types of Manipulations Cont. Staged Manipulations

Also called Event Manipulations Used for 2 reasons

The researcher may need to create some sort of psychological state (anxiety)

The researcher may need to stage a manipulation to recreate a real-world scenario Having a participant do one task and then having them do more

tasks at the same time Staged manipulations usually employ a confederate

Also called an accomplice A confederate is someone who appears to be another

particpant in an experiment but is really a part of the experiment

Example: Someone who purposely insults a participant in a study in order to provoke anger or frustration

Types of Manipulations Cont. Another example of the use of confederates: Asch (1956) study on conformity

Which line is bigger? 1)--------------------- 2)----------------------------- 3)-------------------

Right before a participant had to choose which line was the longest, a confederate announced an incorrect answer

Repeatedly, Asch found that people conformed to the confederate’s incorrect response

Types of Manipulations Cont. Straightforward Manipulations Vs.

Staged Manipulations: It is easier to recreate real-world situations

using Staged manipulations Staged manipulations are more extensive

in terms of cost and set up procedures It is difficult for other researchers to

replicate staged manipulations Data and results of Straightforward

manipulations are easier to interpret due to a greater sense of control

Strength of Manipulations The strength of a manipulation can effect

how results are interpreted Example: In a study on the use of a new

medication to treat diabetes, it is important that the researcher uses a sufficient dosage of the drug in order to see an effect 5 mgs of Diabetex does not reduce blood sugar

levels 10 mgs of Diabetex reduces blood sugar levels

by 3% 20 mgs of Diabetex reduces blood sugar levels

by 10%

Strength of Manipulations Cont. A strong manipulation maximizes the

differences between groups and increases the chance that the independent variable will have a significant effect on the dependent variable

Strong manipulations are especially important in the early stages of research (Why?)

Although you want a manipulation to be as strong as possible, ethical considerations should be examined You do not want to make someone so

anxious that they decide to jump off a building

Cost of Manipulations

Most researchers are limited in terms of the funding they have for a research project

Confederates and elaborate manipulations may cost a great deal

The straightforward approach is usually more budget friendly

Measuring the Dependent Variable

Types of Measures Self-report measures Behavioral Measures Physiological Measures

Measuring the Dependent Variable Cont.

Self-report Measures Used to measure attitudes, judgments

about oneself or another, emotional states, confidence in oneself or others, and other aspects of human thought and behavior

Example: Please use the following 7 point scale to answer this question. I feel that I am the best student in this class:Strongly Disagree _ _ _ _ _ _ _ Strongly Agree

Measuring the Dependent Variable Cont.

Behavioral Measures Direct observations of behaviors such as:

How many errors someone makes on a test How many times a person can lift a 10lb object How may free throws a person makes in 30 minutes

Things to consider with behavioral measures Rate: How many times a behavior occurs Reaction Time: How quickly a response occurs after a

stimulus Duration: How long a behavior lasts

Measuring the Dependent Variable Cont.

Physiological Measures Recordings of responses of the body

Galvanic Skin Response (GSR): Measures the electrical conductance of the skin to look for arousal or anxiety

Electromyogram (EMG): Measures muscle tension to look for stress

Electroencephalogram (EEG): Measures activity of brain cells such as looking at what area of the brain is active while dreaming

Magnetic Resonance Imaging (MRI): A technique used to gather detailed images of the inner body

Functional Magnetic Resonance Imaging (fMRI): Looking at the inner body while someone performs a physical or a cognitive task

Measuring the Dependent Variable Cont.

What Type of Measure(s) to use? Basically, what type(s) of measure a researcher will use

depends on the nature of the study, cost, and possible ethical considerations

If at all possible a researcher should use a combination of all three types: To study anxiety, a researcher can administer a self-

report measure asking how nervous a person thinks they are, a behavioral measure that looks at how many times a person’s voice cracks, and a physiological measure looking at fluctuations in someone’s heart rate

When administering multiple measures, it is important to remember to account for a possible order effect Example: A self-report measure on how nervous a

person is might cause them to become more nervous thus increasing their heart rate

Measuring the Dependent Variable Cont.

Sensitivity of the Dependent Variable The dependent variable should be sensitive enough to

detect differences Do you like your boss (not sensitive) On a scale of 1 to 10, how much do you like your boss

(very sensitive) Beware of the Ceiling Effect and the Floor Effect

Ceiling Effect: Task is too easy so everyone performs well Lifting a 1lb object

Floor Effect: Task is too difficult so everyone performs poorly Lifting a 200lb object

Information on Survey Construction

DESIGNING SURVEYS A good question is one that produces

answers that are reliable and valid measures of something that we want to describe

2 types of question evaluations: Those aimed at evaluating reliability Those aimed at assessing the validity of

answers

Survey Construction Cont. 5 process standards of questions and

answers Questions need to be consistently understood. Questions need to be consistently administered

or communicated to respondents. What constitutes an adequate answer should be

consistently communicated. Unless measuring knowledge is the goal of the

question, all respondents should have access to the information needed to answer the question accurately.

Respondents must be willing to provide the answers called for in the question.

Survey Construction Cont.

Possible steps to assess the extent to which questions meet process standards Focus group discussions Intensive or cognitive interviews Field pre-tests under realistic conditions

Survey Construction Cont. Some general rules for designing good survey

instruments The strength of survey research is asking people about their

firsthand experiences: what they have done, their current situations, their feelings and perceptions.

Questions should be asked one at a time. A survey question should be worded so that all respondents

are answering the same question. If a survey is to be interviewer administered, wording of the

questions must constitute a complete and adequate script such that when the interviewer reads the question as worded, the respondent will be fairly prepared to answer the question.

All respondents should understand the kind of answer that constitutes an adequate answer to a question.

Survey instruments should be designed so that the tasks of reading questions, following instructions, and recording answers are as easy as possible for interviewers and respondents.

Survey Construction Cont.

Ways of addressing validity in surveys Deriving questions from relevant

literature Expert panel

Data Collection Tips Use more than one survey or measure Come prepared

Extra measures, pens, equipment Be prepared to answer questions about the survey

Important not to sway a response Follow up to ensure all items were filled out

(999=missing, 888=don’t know) Know your survey

How to score it, the factors, the subscales If possible, have a limited amount of researchers collect the

data Enter data as quickly as possible and use a trusting

computer program (SPSS, ECXEL) Have two people enter the data (one reads, one enters)

Control ConsiderationsParticipant Expectations

When a participant knows too much about a study, they may expect to perform in a certain manner

Demand Characteristics: any feature of an experiment that might inform participants of the true purpose of a study Once someone is aware of the hypothesis of a study,

the usually do whatever it takes to confirm the hypothesis Hypothesis: People who take a drug will lessen their

anxiety levels. Once a participant knows this, they may choose to do whatever they can do become less anxious (e.g., exercise, see a counselor, meditate).

Control Considerations Cont.Ways to Control Participant Expectations

Use deception to cover up what is really being studied Administer irrelevant measures Add filler questions to measures

Filler questions are unrelated questions on a survey that should disguise the true purpose of a study

Include a Placebo group in a study Placebo effect: When it is not know if results of a

treatment can be attributed to the treatment itself or the expectancies of the treatment

Placebos may consist of giving someone a pill that is made of sugar or injecting someone with saline solution It is important to remember ethical consideration of

using a placebo group. For example, if someone in the placebo group becomes very ill, you may want to administer treatment to them

Control Considerations Cont.Experimenter Expectations

Sometimes experimenters who know the purpose of a study may expect a participant to act or perform a certain way

Also known as expectancy bias There are 2 potential sources of experimenter bias

When a researcher treats participants in the control group differently than participants in the treatment group Spending more time with the treatment group

Researchers may record or analyze observations in a biased manner Well, even though he spoke 10 words clearly, I thought

he was close on another so I will give him 11/20

Control Considerations Cont.Ways to Control Experimenter Expectations

Researchers should be well trained and highly ethical Run all conditions simultaneously so the

experimenter’s behavior is the same for all participants

Use experimenters who are unaware of the true hypothesis of the study Double-Blind Study: Neither the participant nor the

experimenter knows whether a placebo or the actual treatment is given

Once again, it is highly unethical for experimenters to influence participants or manipulate data!!!

Additional Tips for Conducting Experiments Once a study is designed, there are certain ways to

improve it Write up a thorough Research Proposal

Includes a literature review, purpose of the study, procedures of the study, copies of measures, a budget, and the manner with which the data will be analyzed

Used often for grants, Institutional Review Boards, and Ethics Committees

Conduct a Pilot Study A trial run of a study with small sample sizes Used to “get the kinks” out Used to gather participant feedback

Additional Considerations of Experiments Cont. Manipulation Checks

A manipulation check is a way to measure if the independent variable has an effect on the dependent variable Checks if the independent variable manipulation is a

successful operationalization of the conceptual variable being studied

Provides evidence for Construct Validity Allows researchers to change something in a study before it

is too late Allows researchers to see if they are getting significant

results Example: In a study looking at whether people are nicer to

attractive people, a researcher will determine whether the person who they chose to be attractive really is being viewed as so by the participants. If they are not viewed as attractive, the researchers can hire another model.

Manipulation Checks may consist of any type of measure (i.e., self-report, behavioral, physiological)

Additional Considerations of Experiments Cont. Debriefing

Debriefing is the act of meeting with the participants after the conclusion of the study to discuss the true purpose of the study, ethical considerations of the study, and educational implications of the study

Can also be used to ensure that the participants do not share any information with those still involved in the study

Can also be beneficial as researchers may gain valuable information about how the study looks and feels from the “inside”

Analyzing Data

After a quantitative study has concluded, all data should be analyzed statistically

Communicating Research to Others Once a researcher has finished a study,

they should communicate their results and findings to others Professional Meetings and Conferences (NRA)

that present the opportunity to share results in a verbal, written, or poster format

Journal Articles also are a good way to share results Peer Review is the process of a paper being

accepted or declined entry into a journal based on reviews from peers

About 90% of papers submitted to the more prestigious journals are rejected but do not let this get you down