Designing Consumer Market Research by Fida Karim

1

Mr. Fida KarimMarket Research Analyst

By Fida Karim Hunzai- Research Analyst

High quality evaluation research uses the scientific method to investigate the effectiveness of programs and practices.

Some evaluation studies are higher quality than others, and the research consumer must learn to distinguish among them.

One important index of quality is the rigor of the study’s research design.

2By Fida Karim Hunzai- Research

Analyst

An evaluation’s research design is its structure.

At its most stingy, the structure consists of the new program study participants timing of outcome measures

The research consumer must be able to distinguish between experimental and observational designs and what makes them internally and externally valid.


Analyst

An RCT is an experimental study in which eligible individuals or groups of individuals (e.g., schools, communities) are assigned at random to receive one of several programs or interventions.

The group in an experiment that receives the specified program is called the experimental group.

The term control group refers to another group assigned to the experiment, but not for the purpose of being exposed to the program. The performance of the control group usually serves as a

standard against which to measure the effect of the program on the experimental group.


Analyst

The control program may be typical practice (“usual care”), an alternative practice, or a placebo (a treatment or program believed to be inert or innocuous).

Random assignment means that people end up in the experimental or in the control group by chance rather than by choice.

In some randomized studies, the participants and investigators do not know which participants are in the experimental or the control groups: This is the double-blind experiment. When participants do not know, but investigators do, this is called the blinded trial.


Analyst

Randomized controlled trials are sometimes called true experiments because, at their best, they can demonstrate causality.

That means that, in theory, the researcher can assume that if participants in an RCT achieve desirable outcomes, the program caused them.


Analyst

Two commonly used randomized control designs are:

1. Concurrent controls in which two (or more) groups are randomly constituted and they are studied at the same time (concurrently).

2. Wait-list controls in which one group receives the program first; if the program appears to be effective, participants on the wait list receive it.

Participants are randomly assigned to the experimental and wait-list groups.


Analyst

True experiments (randomized controlled trials) are contrasted with non randomized controlled trials and observational studies.

In non randomized controlled trials, the control group is predetermined (without random assignment) to be comparable to the program group

Non randomized controlled trials are also called quasi experiments.

In observational designs, the evaluator does not intervene but studies the effects of already existing programs

Observational designs are sometimes referred to as descriptive.


Analyst

Non randomized controlled trials rely on participants who --

1) volunteer to join the study OR2) are geographically close to the study site OR3) conveniently turn up (at a clinic, school) while the

study is being conducted

Because the study groups are opportunistically rather than randomly composed, study group characteristics (age, sex) may not be balanced before (at baseline) the study begins.

Baseline differences between groups may confound the study’s results.

9


10

Typical confounding variables include age, educational level, motivation, severity of illness, social structure, and income.

Evaluation researchers worry that study groups in non randomized trials will differ from one another at baseline, and the study’s findings will be compromised.

They aim to create study groups that are as similar to one another as possible (equivalent) at baseline or before “treatment.”

Among the strategies commonly used to ensure equivalence is one called matching.


Matching requires selecting pairs of participants or clusters of individuals who are comparable to one another on important variables.

A researcher who is interested in comparing the acuity of vision among smokers and non smokers can try to balance the two groups by selecting pairs of smokers and non smokers who are same age, sex, and have the same medical history

Statistical methods such as analysis of covariance and propensity score analysis are sometimes used to deal with the problem of confounding after the data are collected for the study.


Analyst

Time-series Designs Time-series designs are longitudinal studies that enable

the researcher to monitor change from one time to the next.

They are sometimes called repeated measures analyses.

Interrupted or Single Time-series The interrupted or single time-series design without a

control group involves repeated measurement of a variable (e.g., reported crime) before and after implementation of a program.

The goal is to evaluate whether the program has "interrupted" or changed a pattern established before the program's implementation.


Analyst

Self-Controlled or Pretest-Post Test Designs Each participant is measured on some important

program variable and serves as his or her own control.

Participants are usually measured twice (at baseline and after program participation), but they many be measured multiple times afterward as well.

Historical Controls Investigators compare outcomes among participants

who receive a new program with outcomes among a previous group of participants who received the standard program.


Analyst

Cohort Designs A cohort is a group of people who have something in

common and who remain part of a study group over an extended period of time.

In public health research, cohort studies are used to describe and predict the risk factors for a disease and the disease's cause, incidence, natural history and prognosis. They tend to be extremely large studies.

Cohort studies may be prospective or retrospective. With a prospective design, the direction of inquiry is forward in time, while with a retrospective design, the direction is backward in time.


Analyst

Case Control Designs Case-control designs are generally

retrospective. They are used to explain why a phenomenon currently exists by comparing the histories of two different groups, one of which is involved in the phenomenon.

For example, a case control design might be used to help understand the social, demographic, and attitudinal variables that distinguish people who at the present time have been identified with frequent headaches from those who do not at the present time have frequent headaches.


Analyst

Cross-Sectional Designs

Cross-sectional designs result in a portrait of one or many groups at one period of time.

They are sometimes called descriptive, pre experimental, or survey designs.


Analyst

Internal Validity is the ability to make accurate inferences about a program’s outcomes and effectiveness. (Program A caused Outcome A.)

External validity refers to the extent to which the results are applicable to other programs, populations, and settings. Another term for external validity is generalizability.


Analyst

The research consumer wants to be sure that Program A which was conducted in Setting A

1. caused Outcome A (internal validity) and 2. will be effective in Settings B and C, etc.

(external validity)

Study design flaws can “threaten” internal and external validity.


Analyst

Threat to Validity Explanation

Selection Group characteristics (e.g., age, gender) are not evenly balanced

History Unanticipated outside events occur while study is in progress

Maturation Participants change or mature

Testing Taking one test effects performance on the second


Analyst

Threat to Validity Explanation

Instrumentation Changes occur in measures or observers

Statistical regression Participants who are selected because of extremely high or low scores return to the average over time

Attrition Participants drop out of the study in a non random manner

Expectancy Participants or researchers have expectations of the program and study activities


Analyst

Threat to External Validity Explanation

Interaction between selection and experiment

A unique mixture is created that is unlikely to occur again

Reactive effects of testing Completing the baseline measures influences program participation uniquely

Reactive effects of experimentation (Hawthorne Effect)

Study participants behave uncharacteristically because they are being observed

Multiple program interference Complementary activities may influence participants’ behavior and study outcomes

21Dated: 09202013By Fida Karim Hunzai - Research

Analyst

Designing Consumer Market Research by Fida Karim

Education

Transcript of Designing Consumer Market Research by Fida Karim