Research Lecture July 31

7/31/2019 Research Lecture July 31

http://slidepdf.com/reader/full/research-lecture-july-31 1/529



Planning is

pointless! –we arenot in control -everything ischanging too

quickly.



Illustrated Story on funding research

Europides has got the instruments!



FP6FUNDING INSTRUMENTS FOR

EU INDUSTRIAL RESEARCH

Illustrated leaflet



RESEARCH ?



INSTRUMENTS ?



INSTRUMENTS for RESEARCH !



NETWORK of EXCELLENCE

…the art of GETTING TOGETHER to cross RIVERS !



INTEGRATED PROJECT

…the art of BUILDING BRIDGES !


http://slidepdf.com/reader/full/research-lecture-july-31 10/529…the art of MAKING PILLARS !

SPECIFIC TARGETEDRESEARCH PROJECT



COORDINATION ACTION

…the art of BRINGING TOGETHER !



…the art of SUGGESTING !

SPECIFIC SUPPORT ACTION



CONCLUSIONS (1):

…founded on S&T excellence, which integrates the

existing knowledge with the creation of new one



CONCLUSIONS (2):

…and generating enthusiasm for science !!!



ACHIEVING BREAKTHROUGH…

…THROUGH INTEGRATION !



INSTRUMENTS ?Purpose Target

audience EU-fundedactivities

IndicativeEU funding

Averageduration

‘Optimum’consortium

Specific characteristics

NoE

Art of getting together

Durableintegration

ofparticipants’research

R&Dinstitutes

Universitiesmainly indirectly: Industry/SMEs

Integrating

Joint Research

SpreadingexcellenceConsort. mgmt

€ 7 million

€ 5-15 million

48-60months

6-12 partners InstitutionalCOMMITMENT at

strategic level from thevery start & for thewhole duration

Limited number of partners

IP Art of breakthrough

Ambitiousmultipleobjectiveresearch

Industry/SMEs

R&Dinstitutes

Universities

Research

Training

InnovationConsort. mgmt

€ 10 million

€ 4-25million

36-60months

10-20partners

RADICAL INNOVATIONthrough INTEGRATIONof disciplines, techs,activities

PROGRAM approach onmultiple issues

STREP Art of new knowledge

Singleobjectiveresearch

Industry/SMEs

R&Dinstitutes

Universities

Research

Training

InnovationConsort. mgmt

€ 1.9 million

€ 0.8-3million

18-36months

6-15 partners PROJECT approach on asingle issue

FRONTIERS ofKNOWLEDGE

CA Art of bringing together

Coordination Networking R&Dinstitutes

Universities Industry/SMEs

Meetingsseminars

workshops

info systemsConsort. mgmt

€ 1 million

€ 0.5-1.8million

18-36months 13-26partners NO funding researchactivities

Activities for coordination(“programme” approach)

SSA Art of

forward-looking supporting

Orient RTDactivities

R&Dinstitutes

Universities Industry/SMEs

Foresight

scenario-building

Technologyroadmaps

€ 0.5 million

€ 0.03-1million

9-30months

1-15 partners NO funding research

“Project” approach

Possibility of one singleparticipant



New Instruments:

Networks of ExcellenceIntegrated projects

Traditional Instruments: Specific targeted research projects

Coordination actions Specific support actions

Europides has got the instruments!



Research Methodology

A science of studying how research is done

scientifically

A way to systematically solve the research

problem by logically adopting various steps

Methodology helps to understand not only

the productsof scientific inquiry but the

processitself

Aims to describe and analyze methods,

throw light on their limitations and

resources, clarify their presuppositions and

consequences, relating their potentialities

to the twilight zoneat the ‗frontiers of

knowledge‘



Outline of Research Process

Phase 1: essential first steps

Phase 2: data collection

Phase 3: analysis and

interpretation



Phase 1

Clarify the issue to be

researched and select research

method(s).

Essential because a question

that is unclear or too broad

cannot be answered.

The research method allows the

research to be conducted

according to a plan or design.



Phase 1 cont.

Clarifying the question and

method enables the researcher

to be clearer about the data that

is needed

Therefore to make a decision

about what sample size, or the

amount of data, is needed.



Phase 2

Collecting the data surveys, interviews, literature review, participant

observation, etc…..

I undertook the interviews I had arranged,

recording them, then transcribing the recordings I attended the EHPA conference and made

copious notes throughout

Summarising and organising the data

Excerpts from and summaries of transcripts

Thoughts arising from notes on conference



Phase 3

Analysis, interpretation

Relating the data to the

research question

Drawing conclusions

Assessing the limitations of the

study



The Scientific Method

Theory

Hypothesis

Operational Definition

Principle of Falsifiability

Subjects

Selection Factor

Replication



Principle of Falsifiability

The principle that a scientific

theory must make predictions

that are specific enough to

expose the theory to thepossibility of disconfirmation;

that is, the theory must predict

not only what will happen, butalso what will not happen.



Methods of Observation

The Case-StudyMethod

The Survey Method

The Testing Method

The Naturalistic-Observation Method

The Laboratory-Observation Method

The CorrelationalMethod

The ExperimentalMethod



The Case-Study Method

A carefully drawn

biography that may

be obtained through

interviews,questionnaires, and

psychological tests



The Survey Method A method of scientificinvestigation in

which a largesample of peopleis questioned

about their attitudes or behavior



Direct Surveys

Interviewer maintains or can

maintain a direct communication

with the respondent and is able

to provide feedback, repeat aquestion, or ask for additional

information



Indirect Surveys

Researcher‘s personal impact is

very small because there is no

direct communication between

the respondent and theinterviewer. The questions are

typically written and handed in,

mailed, or sent electronically tothe respondents‘ homes,

classrooms, or work places.



Focus Group Methodology

A survey method used in

academic & marketing research.

The most common usage is one

where a group responds tospecific social, political, or

marketing messages. The

typical focus group contains 7 – 10 members who are experts,

potential buyers, viewers or

other customers.



The Testing Method Psychologists use

psychological tests

like intelligence,

aptitude, andpersonality, to

measure various traits

and characteristics

among a population



Testing

Standardize: To develop

uniform procedures for giving

and scoring a test.

Norms: Established standards

of performance.

Reliability: Consistency of

scores derived from a test.

Validity: The ability of a test to

measure what it was designed

to measure.

Ob ti



Observations

Naturalistic-Observation

A scientific method inwhich organisms are

observed in their

natural environments

Laboratory-Observation

A method where aplace is found in

which theories,

techniques, and

methods are testedand demonstrated



The Correlational Method

A scientific method that studies

the relationships between

variables

Correlation coefficient is a

number between +1.00 to -1.00

that expresses the strength and

direction of the relationshipbetween two variables



Types of Correlations

Positive correlation: Increases in one

variable are associated with increases in theother; decreases are likewise associated

Negative correlation: Increases in one

variable are associated with decreases in

the other



The Experiment:

Hunting for Causes

Experimental Variables

Experimental and ControlConditions

Experimenter Effects

Advantages and Limitations of Experiments



Experimental Variables

IndependentVariable: A variablethat an experimenter manipulates.

Dependent Variable: A variable than anexperimenter predictswill be affected by

manipulations of theindependent variable.



Experimental Method

Treatment refers to a

condition received by

participants so that its

effects may be observed

Experimental subjects

receive the treatment

Control subjects do not

receive the experimental

treatment but for whomall other conditions are

comparable to those of

experimental subjects



Experimental Method Cont.

Placebo refers to a

bogus treatment that

has the appearance

of being genuine Blind refers to

unawareness as to

whether or not one

has received atreatment

Double-blind refers

to a study where

neither the subjects

nor the personsmeasuring results

know who has

received the

treatment



Different Research Methods

Cross-SectionalStudy: Subjects of different ages arecompared at a given

time. Longitudinal

Study: Subjects arefollowed

and periodicallyreassessed over a

period of time.



Representative

SamplesSubject availability

Willingness to participateGeographic Isolation

Unavailability



Measurement

The behavioral definition and

measurement techniques of

the researcher‘s home

culture may not transfer

easily to another culture

thereby leading to animposed etic (Berry, 1969)



Measure of Central Tendency

The measure that indicates the

location of a score distribution

on a variable, that is, describes

where most of the distribution islocated.

M h i l l i f



Mean= mathematical central point of a

distribution of scores

Median= score in a distribution on the 50th

percentile

Mode= the most frequently occurring score

in the distribution



48

KNES 510

Research Methods in Kinesiology

Survey Research

48



Descriptive research is also known as non-experimental research

Asks the basic question: What is?

No manipulation of variables

Measure and record events that would

happen anyway

49

49

Descriptive Research



Descriptive Research, cont‘d

Manipulation of variables may be impossible

or unethical

Effects of megadoses of anabolic steroids on

strength

Deliberately causing injury to study different

types of therapy

No randomization, therefore less control andmany threats to internal validity

Cause-and-effect is more difficult to establish50

50



Survey – technique of

descriptive research that

seeks to determine

present practices or

opinions of a specified

population

Types of survey research

include the

questionnaire, interview,and normative survey

51

51

Survey



Examples of Survey Research

Durell, D. L., Pujol, T. J., & Barnes,J. T. (2003). A survey of the scientificdata and training methods utilized bycollegiate strength and conditioning

coaches. J Strength Cond Res,17 (2), 368-373.

Schick MG, Brown LE, Coburn JW,Beam WC, Schick EE, Dabbs NC.Physiological profile of mixed martialartists. Medicina Sportiva. 14(4):182-187, 2010.

Rossi MD, Brown LE, Whitehurst M.52



Questionnaire – type of paper-

and-pencil survey used in

descriptive research in which

information is obtained byasking participants to respond to

questions rather than by

observing their behavior Limitation is that results are

simply what people say they do,

believe, like, dislike, etc.53

53

Questionnaire

http://www.youtube.com/watch?v=QOqXlbWf9Io



Determine the objectives

What information is wanted?

How will the results be analyzed? Will comparisons be made between groups of

respondents?

54

54

Questionnaire, cont’d



Delimit the sample

What is the specific population you wish to

examine? Adults vs. children

Exercisers vs. nonexercisers (how do you define

exercisers?)

Elite coaches

55

55




Your sample must be representativeof the population Sampling error

▪ Many samples may be drawn from apopulation

▪ Each sample will yield different results▪ The difference between samples is the

sampling error (amount of error to expect in asingle sample)

Sample size

▪ Must be adequate to represent population of interest

▪ Must be practical from the standpoint of timeand cost

56

56




Construct the questionnaire Open-ended questions

▪ Category of question in questionnairesand interviews that allows the respondentconsiderable latitude to express feelingsand to expand on ideas

▪ Example: ―How do you think things wenttoday?‖

▪ Drawbacks:▪ Respondents don‘t like them

▪ They are time-consuming to answer ▪ Limited control over the types of answers

given

▪ May be more difficult to analyze

57

57




Closed questions

Category of question found in

questionnaires or interviews that

requires a specific response and that

often takes the form of rankings,

scaled items, or categorical

responses

Ranking – type of closed question

that forces the respondent to placeresponses in a rank order according

to some criterion

58

58




Example of Ranking: From what

sources has most of your nutrition

information come? Rank top 3

59

59




60

Scaled items – type of closed question that

requires participants to indicate the strength of

their agreement or disagreement with some

statement or the relative frequency of some

behavior

Example of scaled item: In a required physical

education program, students should be required to

take at least one dance class.

1. Strongly disagree

2. Disagree

3. Undecided

4. Agree

5. Strongly agree 60

Survey Research Process, cont’d



61

Likert-type scale – consists of 3 to 9 items

Equal intervals between responses, i.e., difference

between ―strongly agree, and ―agree is considered

equivalent

61




62

Categorical response –

type of closed question that

offers the participant only

two responses, such as

yes or no

Possible responses include

yes/no, true/false,

female/male, etc.

62




Appearance and design Appearance has been shown to affect response

rate

Provide written instructions for completion

First few questions should be easy to answer

Short questionnaires have higher response rates

63

63

Questionnaire, cont’



Conduct a pilot study

Send questionnaire to colleagues

or acquaintances

Revise and send to sample of population of interest

Analyze results as part of pilot

study Revise again and use

64

64




Write the cover letter

Be professional and concise

Explain purpose and importance

Assure respondent of their privacy

and anonymity

Use institutional letterhead if

applicable Rewards and incentives may be

used, including money

65

65




Send the questionnaire

Include self-addressed, stamped envelope

E-mail, fax, and the internet may also be used

66

66


http://www.surveymonkey.com/?cmpid=us:ps:google&gclid=CPb-

u_7WmqACFQVaagodQSSdTw

http://www.surveymonkey.com/?cmpid=us:ps:google&gclid=CPb-u_7WmqACFQVaagodQSSdTw







Follow-up

Wait at least 10 days for follow-up

Wait another 10 days then send

another questionnaire Keep in mind that respondents are

―self -selected‖ and this biases

your results

67

67




Analyze the results These topics will be discussed later in class

68

68


l hi h d



Delphi survey method – survey

technique that uses a series of

questionnaires in such a way

that the respondents (usuallyexperts) reach a consensus

about the subject

Survey is sent to respondents(experts)

Results are sent to respondents

and they are asked to reconsider

their answers69

69

Delphi Survey Method

l i



Essentially the same as the

questionnaire except

questioning is done orally

instead of in writing Higher response rate but

smaller samples than

questionnaire

70

70

Personal Interview

l i d



Advantages of the

personal interview:

More adaptable

Interviewer canobserve the

respondent

Greater rate of return than

questionnaire

Easier to explain

questions71

71

Personal Interview, cont’d



Normative Survey

Normative survey – survey

method that involves

establishing norms for abilities,

performances, beliefs, andattitudes

Similar to questionnaire except

that tests are administered AAHPERD Youth Fitness Test

(1958)

National Children and Youth72



Next Class

Tonight-intro

Chapter 16

Abstract #2

(descriptive)

73



RESEARCH CONCEPTS

& TERMINOLOGY

Roger Clarke, Xamax Consultancy, Canberra

Visiting Professor, CSIS, Uni of Hong Kong

Visiting Fellow, Australian National University

http://www.anu.edu.au/people/Roger.Clarke/...

...Res /40-CTerm.ppt

ebs, 16-20 January 2003

Alternative Motivations for



Alternative Motivations for

Research

„Pure Research‟

―because it‘s there‖

contribute to abstract, theoreticalunderstanding

„Applied Research‟

―I have a hammer, so go and find me anail‖

„Instrumentalist Research‟

―

e a ure o esearcOutcomes



OutcomesExploratory

The first depiction of something new

Descriptive

The depiction of a

behaviour or a domainExplanatory

Systemic explanation

of how past

behaviours arose Ascription of causes to

prior occurrences

„Predictive‟

Statement of whatoccurrences will arise

Systemic explanation of how

behaviours will arise

Statement and explanation of the effect particular

interventions will have

Normative

Statement of interventionsnecessary to achieve desired

outcomes

Statement of desired

outcomes

Th N t f D t



The Nature of Data

(Measurement Scales)

Quantitative

Ratio a natural zero

Cardinal / Interval no natural zero

Ranked Ordinal sequence (numbers)

Qualitative

Category Ordinal sequence (text)

Nominal differentiation

Dichotomous it is or it isn‘t



Likert Scales

A contrived Ranked Ordinal Scalemuch-used in attitudinal research

Usually 5, 7 or 9 choices

Usually anchored by end-pointssuch as Strongly Disagree

... Strongly Agree

and with a neutral-sounding mid-point

Usually very long lists of questions

The data is generally processed

Characteristics of Data



Characteristics of Data

Collection

Passive (Observation)

or Active (Response Elicitation)

Purpose: Disguised or Openly

Declared

Structured, Semi-Structured or

Unstructured

In Person, By Telephone,

By Written Form (e.g. Mail),

By Email/Web-Form,

B Mechanical Means



Unit of Analysis

A Person

An Event

An Object

A Body of Individuals

Group, Organisational Unit,

Organisation

A Relationship, e.g. a Dyad

An Aggregate

Census District, Industry Segment or



Time Horizon

Study Setting

Snapshot

Cross-Sectional (snapshots of

multiples)

Longitudinal

Contrived or Naturalistic

Researcher Interference – degree

and nature



Th C t f C lit



The Concept of Causality

One or more Variables (the Cause)are asserted to determine another

Variable (the Effect)

Determinant Cause That factor which is the necessary and

sufficient condition for some

subsequent effect Probabilistic Causes

Those factors that are necessary but

individually not sufficient conditions for



Evidence of Causality

Co-variation / Correlation

Time Order, and Chaining

Absence of other variables

that

might be the real cause

Plausibility / Systemic

Relationship



Test Case

I am a manufacturer of raincoats.

I want to increase sales.

I increase my advertising budget by100%.

Sales go up 20%.

What is the relationship between theincrease in the advertising budget and

the increase in sales?

Introduction



Introduction Research involves a range of skills and activities.

To be a good researcher:1. You have to be able to work well with a wide variety of

people

2. Understand the specific methods used to conduct research

3. Understand the subject that you are studying

4. Be able to convince someone to give you the funds to study

it

5. Stay on track and on schedule, speak and write persuasively

86

1 Descriptive Research



1. Descriptive Research

When a study is designed primarily to describe what

is going on or what exists:

- Describe characteristics of relevant groups

(consumers, salespeople, etc)

- Public opinion polls that seek only to describe theproportion of people who hold various opinions are

primarily descriptive in nature.

- Determine the perception of product characteristics

- Make specific predictions (for example, retail sales)

87



3. Causal Research



3. Causal Research

When a study is designed to determine whether one

or more variables (e.g., a program or treatmentvariable) causes or affects one or more outcome

variables (cause-effect relationships).

Examples:

- study the effect of an advertising campaign on

product sales.

- Study the effect of presence and helpfulness of

salespeople on sales of housewares. Causal studies are probably the most demanding of

all (the main method for causal research is

experimentation). 89

Variables



Variables

A variable is any entity that can take on different values.

An attribute is a specific value on a variable Gender has two attributes: male and female.

the variable agreement might be defined as having fiveattributes:

1 = strongly disagree

2 = disagree3 = neutral

4 = agree

5 = strongly agree

The attributes of a variable should be exhaustive andmutually exclusive.

90

Types of variables



Types of variables

Qualitative: described in words (color, gender, etc) Quantitative: described in numbers (age, salary, etc)

Quantitative variables are divided into two types:

Discrete: whole numbers, countable (number of employees)

Continuous: can take fractional values; can take

any value within a certain interval)

91



Objectives of this lesson:Primary and Secondary



Primary and Secondary

Data

Define secondary and primary data

Describe primary data collection

methods Describe sampling techniques

Identify advantages disadvantages of

different data gathering techniques

Construct a survey



Primary Data

Data never gathered before Advantage: find data you need

to suit your purpose

Disadvantage: usually morecostly and time consuming thancollecting secondary data

Collected after secondary datais collected

S li T h i



Sampling Techniques

Population - total group of respondentsthat the researcher wants to study.

Populations are too costly and time

consuming to study in entirety.

Sample - selecting and

surveying respondents (researchparticipants) from the population.

Sampling Techniques



Sampling Techniques

A probability sample is one thatgives every member of the

population a known chance of

being selected. simple random sample - anyone

stratified sample - different groups

(ages) cluster sample - different areas

(cities)

All are selected randomly.

Sampling Techniques



Sampling Techniques

A non-probability sample is anarbitrary grouping that limits the

use of some statistical tests. It

is not selected randomly. convenience sample - readily

available

quota sample - maintainrepresentation



Primary Research Methods Focus Groups – bring together respondents

with common characteristics

Observation - actually view respondents

Experiment - controlled variables andrespondent groups.

Non-personal survey – on site, telephone,mail, fax, computer, panel

Personal interview - one-on-one survey withrespondents

Company records – internal document surveyresearch

Constructing the Questionnaire



Constructing the Questionnaire

Select the correct types of questions:

– harder to score but get“richer” information

– offer two

either/or responses (true/false; yes/no;for/against

– select one or more thanone

– gather range of “values”(strongly disagree, somewhat disagree,neutral, somewhat agree, strongly agree

1 H h d f th f ll i



1. Have you had any of the following

medical preventive tests/exams?

_____ mammogram (if a women)

_____ prostate exam (if a man)

_____ lung x-ray

_____ electrocardiogram _____ stress test



2. Do you currently smoke?

_____ YES

_____ NO

3. Please evaluate the following statement:



I understand the University‘s code of

conduct as it relates to plagiarism.

____ absolutely agree

____ somewhat agree

____ neutral

____ somewhat disagree ____ absolutely disagree

Important characteristics of



good questionnaires

Plan a user-friendly format Gather demographic data – age, gender, etc.,

when necessary.

Guarantee anonymity Ensure ease of tabulation – Scantron forms

Ask well-phrased and unambiguous

questions that can be answered Develop for completeness – get all the data

Pilot test the instrument

Assignment



Prepare a questionnaire (word processed on paper) that

you utilized to gather the primary data for the formalreport case.

The questionnaire will be inserted as an appendix at theend of your formal report.

Remember, you should include on the survey:

an introductory statement about the purpose of thesurvey,

a motivational reason why customers should take it, and

a reminder to participants that they are taking the survey

anonymously.You should NOT include on the survey:

answer percentages



Introductory Statement

The purpose of this survey is to help management identify family

issues that our employees experience that are related to jobperformance. Please respond to the following survey items bychecking the appropriate response next to each question/item. Yourresponses are completely anonymous and will only be used to assessoverall employee characteristics.

ensure anonymity

purpose of the survey

Structuring the main text




Chapter 3 – Research design andmethodology

Introduction: What does Chapter 3

consist of? Research methodology

Data collection (steps you took,

methodology)

Data analysis

Conclusions based on Chapter 3





Make Sure… that the methodology addresses

both the procedure for thecollection of your data and theone for your analysis.

that you section the analysis sothat the argument unfolds in a

clearly stated, detailed, logicalprogression.

that you view the dataobjectively. Don‘t ignore data

that dis roves the h othesis or



Presented by

JUDITH O. GUYOD Adapted from the presentation of CATHERINE JOY A. ADDAUAN

During the Research Class of 2008-2009

WRITING THEMETHODOLOGY





Includes the following:

1. Research Design

2. Research Environment

3. Research Materials

4. Research Method/Data

Gathering Procedure

5. Statistical Tools

Research Design



Research Design

Includes the following:

1. identification of the variables

and their categories

2. the sampling scheme

3. the treatment grouping

Research Environment



Research Environment

States the specific area,date/time the investigation was

conducted

Research Materials



Research Materials

Includes a list of the differentmaterials used in the

investigation

Materials used in the study likeglasswares, equipment and

chemicals are mentioned as the

details of the procedure aregiven but they are NOT

enumerated or listed individually

as in a laboratory manual

Research Method/Data

G th i P d



Gathering Procedure

Presents in chronological order the general procedure of theconduct of the study

Provides all the needed detailsespecially if new to allow othersto use your methodology

Describes in detail what will be

done, as well as when, where,and how



Discusses the different methodsin sampling, experimenting, data

gathering and statistical testing

and analysis Presents the conditions that

would enable the researchers to

observe the differences on thedependent variables which are

actually the results of the

manipulation of the independent

Statistical Tools



Statistical Tools

Discusses the appropriatenessof the statistical tools employed

in treating the data gathered for

analysis Describe how each statistical

test, whether descriptive or

inferential, is used in the study,the level of significance is also

established

Guide Questions for the Checkingthe Different Parts of the



t e e e t a ts o t e

Research Proposal

1. Have I identified the specificresearch problem I wish to

investigate?

2. Have I indicated what Iintended to do about this

problem?

3. Have I put forth an argumentas to why this problem is

worthy of investigation?

4. Have I asked the specific



6. Do I intend to investigate arelationship? If so, have I

indicated the variables I think

may be related?7. Have I identified all key terms

clearly, and operationally?

8. Have I surveyed and describedrelevant studies related to the

problem?

9 Have surveyed the existing



11. Have I describe my sampling plan?12. Have I describe the relevant characteristics of

my sample in details?

13. Have I identified the population to which theresults of the study may legitimately begeneralized?

14. Have I described the instrument/s to be used?

15. Have I indicated their relevance to the presentstudy?

16. Have I stated how will I check the reliability of data obtained from all instrument?

17. Have I stated how will I check the validity of data obtained from all instruments?



18. Have I fully described the procedures tobe followed in the study – what will bedone, where, when, and how?

19. Have I discussed any feasiblealternative explanation that might exist for

results of the study?20. Have I discussed how will I control for

these alternative explanations?

21. Have I described how will I organize thedata I will collect?

22. Have I described how will I analyze thedata, including statistical procedures, andwhy these procedures are appropriate?

Outline



1. Introduction to researchstrategies

2. Validity and its threats

- internal validity

- external validity

3. Research strategies, research

designs and researchprocedures

1. Introduction to research

t t i



strategies

research strategy reflects the general approach and

goals of a research study

types of research strategies descriptive strategy

Nonexperimental strategy

correlational strategy

experimental strategy

quasi-experimental strategy


t t i



strategies

Descriptive strategy the goal is to describe the state of affairs at the time of the study

measures variables as they exist naturally

e.g. 19% of eligible voters participated in the election

Correlational strategy measures two variables, usually as they exist naturally

the goal of this strategy is to describe a relationship between thetwo variables without attempting to explain the cause of therelationship

e.g. Are students GPA‘s related to their parent‘s income?

Nonexperimental strategy Answers questions about the relationship between two variables

by demonstrating a difference between two groups or twothreatment conditions

E.g. verbal scores of 6-years old boy and 6-years old girls


t t i



strategies

Experimental strategy the researcher manipulates one variable

(called independent variable) while

observing or measuring a second

variable (dependent variable) this is the ‗true‘ experiment because

independent variable is manipulated by

the researcher (e.g. room temperature)

the goal of experimental strategy is todetermine whether a causal relationship

exists between two variables


t t i



strategies

Quasi-experimental strategy uses a nonmanipulated variable to

define groups or conditions (e.g.time or age) or pre and post

threatment controls other variables as much

as possible

the goal is to obtain evidence insupport of a cause-and-effectrelationship

however, a quasi-experimentalstrategy can not unambiguously

3. Research strategies,research designs, and



g ,

research procedures Research strategy

refers to the general approach and

goals of the study

Research design

general plan for implementing a

research strategy (e.g. group versus

individual, same individuals vs. different

individuals, number of variables

included)

Research procedure

an exact, step-by-step description of a

specific research study (exact



CHAPTER 10

MEASUREMENT IN MARKETING

RESEARCH

Important Topics of This

Chapter



Chapter

Basic types question-responseformat.

Consideration of choosing a questionresponse-format.

Measurement and scalecharacteristics in a question-response format.

Levels of measurement of scales.

Various types scaled-responsequestion formats.

Reliability and validity of measurements.

The Questionnaire‘s ―Position‖ in theResearch Process



Questionnaire

Survey

Objectives

Respondent’s

Information

Data

Analysis

Findings

Recommendations

Managerial Action

Criteria for a Good

Q ti i



Questionnaire

To design a good questionnaire, the following issues shouldbe considered:

Does it Provide

the NecessaryDecision-Making

Information?

Does it Consider theRespondent?

Basic Question-Response



Format

Editing Refers to going through the questionnaire to

make certain the ―skip patterns‖ are followedand required questions are filled out.

A skip pattern is the sequence in whichquestions are asked.

Open-Ended Response Format Questions: An open-ended question is one that does not

contain prerecorded possible responses:

Un-probed format:

Seeks no additional information from respondents.

Probed format:

Researcher may ask comments or statement fromthe respondents.

Response format:

Researcher may ask additional information.

Basic Question-Response

Format (cont )



Format (cont.)

Closed-Ended Response FormatQuestions: Dichotomous closed-ended questions:

Yes/No options.

Multiple category closed-endedquestions: They are very popular question style.

Scaled-response Questions:

Un-labeled scaled-response format: Purely numerical or only endpoints areidentified.

Labeled scaled-response format: All of the scaled position are identified.

Considerations in Choosing A

Questions Response Format



Questions Response Format

Nature of property being measured: Different type of question format must

be used.

Previous research studies:

Questionnaires may be used withpermission.

Data collection mode: Mail, telephone, personal/computer

interviews. Ability of the respondents:

Previous research experiences mayhelp.

Scale level desired:

Basic Concepts in

Measurement



Measurement

Objects: Consumers, brands, stores,

advertisements.

Properties:

Demographic characteristics. Objective properties:

Physically verifiable.

Subjective properties: Cannot be directly observed, such as

person‘s attitude and intentions.

Scale Characteristics



Description: Agree/Disagree, Approve/Disapprove

Order:

Size of the descriptor .

Distance:

Two cars Vs. one car family.

Origin:

0 or 1.

Scale

Primary Scales of Measurement



Nominal NumbersAssigned

to Runners

Ordinal Rank Orderof Winners

Interval PerformanceRating on a0 to 10 Scale

Ratio Time toFinish, inSeconds

7 38

Thirdplace

Secondplace

Firstplace

Finish

Finish

8.2 9.1 9.6

15.2 14.1 13.4



A Classification of ScalingTechniques



Scaling Techniques

Non-comparativeScales

ComparativeScales

PairedComparison

Rank Order

ConstantSum

Q-Sort andOtherProcedure

s

ContinuousRating Scales

ItemizedRatingScales

LikertSemanticDifferential

Stapel

Techniques

Attitude Scales



•Scaling Defined:

•The term scaling refers to procedures

for attempting to determine quantitativemeasures of subjective and sometimesabstract concepts. It is defined as aprocedure for the assignment of numbers to a property of objects inorder to impart some of thecharacteristics of numbers to the

Unidimensional and Multidimensional

Scaling



Scaling

UnidimensionalScaling

MultidimensionalScaling

Procedures designedto measure only oneattribute of a

respondent or object

Procedures designedto measure several

dimensions of arespondent or object

Different Type of Scales



e e t ype o Sca es

Graphic Rating Scale: Present respondents with a graphic

continuum typically anchored by twoextremes.

Itemized Rating Scale: Itemized rating scales are very similar to

graphic rating scales, except thatrespondents must select from a limitednumber of ordered categories rather

than placing a check mark on acontinuous scale.

Rank-Order Scale: Itemized and graphic scales are non-

comparative because the respondent

Different Type of Scales (cont.)



yp ( )

Q-Sorting: Q-Sorting is basically a sophisticated

form of rank ordering. A set of objects -

verbal statements, slogans, product

features, potential customer services,and so forth - is given to an individual to

sort into piles according to specific

rating categories.

Paired Comparison: Paired comparison scales ask a

respondent to pick one of two objects

from a set based upon some stated

criteria

Different Type of Scales

(cont )



(cont.)

Constant Sum Scales: Constant sum scales are used

more often by market researchers

than paired comparisons because

the long list of paired items is

avoided.

This technique requires the

respondent to divide a givennumber of points, typically 100,

among two or more attributes

based on their importance to the




Semantic Differential Scale: The construction of the semantic differential

scale begins with the determination of a conceptto be rated. The researcher selects dichotomouspairs of words or phrases that could be used todescribe the concept. Respondents then rate

the concept on a scale. The mean of theseresponses for each pair of adjectives iscomputed and plotted as a ―profile‖ or image.

Stapel Scale: The Stapel scale is a modification of the

semantic differential. A single adjective is placedat the center of the scale. Typically it is designedas a 10-point scale ranging from +5 to -5. Thetechnique is designed to measure both thedirection and intensity of attitudessimultaneously.




Likert Scale: The Likert scale consists of a

series of statements that expresseither a favorable or an

unfavorable attitude toward theconcept under study.

Purchase Intent Scale:

Scale designed to measure thelikelihood that a potentialcustomer will purchase a productor service.

Some Basic Considerations When

Selecting a Scale



Selecting a Scale

Selecting a Rating,Ranking, Sorting, or

Purchase Intent Scale

Balanced Versus Non-balanced Alternatives

Number of Categories

Odd or Even Number of Scale Categories

Forced Versus Non-forced Choice

Approaches to Identifying

Determinant Attitudes



Determinant Attitudes

Direct

Questioning

Indirect

Questioning Observation

Obtaining Shampoo PreferencesUsing Paired Comparisons

Instructions: We are going to present you with ten pairs of shampoo brands



Instructions: We are going to present you with ten pairs of shampoo brands.

For each pair, please indicate which one of the two brands of shampoo you would

prefer for personal use. Recording Form: Jhirmack Finesse Vidal

Sassoon

Heads &

Shoulders

Pert

Jhirmack 0 0 1 0

Finesse 1a

0 1 0

Vidal Sassoon 1 1 1 1

Head & Shoulders 0 0 0 0

Pert 1 1 0 1

Number of Times

Preferred b

3 2 0 4 1

a A 1 in a particular box means that the brand in that column was preferred over thebrand in the corresponding row. A 0 means that the row brand was preferred over

the column brand. bThe number of times a brand was preferred is obtained bysumming the 1s in each column.

Paired Comparison ScalingThe most common method of taste testing is paired comparison. Theconsumer is asked to sample two different products and select the



p pone with the most appealing taste. The test is done in private and a

minimum of 1,000 responses is considered an adequate sample. A blind taste test for a soft drink, where imagery, self-perception andbrand reputation are very important factors in the consumer‟spurchasing decision, may not be a good indicator of performance inthe marketplace. The introduction of New Coke illustrates this point.New Coke was heavily favored in blind paired comparison taste tests,

but its introduction was less than successful, because image plays amajor role in the purchase of Coke.

A paired comparison taste test

Preference for ToothpasteBrands Using Rank Order Scaling



Instructions: Rank the various brands of toothpaste in order of preference. Begin by picking out the one brand that you like most and assign it anumber 1. Then find the second most preferred brand and assign it a number 2.Continue this procedure until you have ranked all the brands of toothpaste inorder of preference. The least preferred brand should be assigned a rank of 10.

No two brands should receive the same rank number.

The criterion of preference is entirely up to you. There is no right or wronganswer. Just try to be consistent.

Brand Rank Order



1. Crest _________

2. Colgate _________

3. Aim _________

4. Gleem _________

5. Macleans _________

6. Ultra Brite _________

7. Close Up _________

8. Pepsodent _________

9. Plus White _________

10. Stripe _________

Importance of Toilet Soap Attributes Using a Constant Sum



ScaleInstructionsOn the next slide are eight attributes of bathingsoaps. Please allocate 100 points among theattributes so that your allocation reflects the

relative importance you attach to each attribute.The more points an attribute receives, the moreimportant the attribute is. If an attribute is not atall important, assign it zero points. If an attribute

is twice as important as some other attribute, itshould receive twice as many points.



Scale BasicCharacteristics

Examples Advantages Disadvantages

C ti Pl k R ti t TV E t t t S i b

Basic Non-comparative Scales



ContinuousRating

Scale

Place a mark on acontinuous line

Reaction to TVcommercials

Easy to construct Scoring can becumbersome

unlesscomputerizedItemized RatingScales

Likert Scale Degrees of

agreement on a 1(strongly disagree)to 5 (stronglyagree) scale

Measurement

of attitudes

Easy to construct,

administer, andunderstand

More

time-consuming

SemanticDifferential

Seven-point scalewith bipolar labels

Brand, product,and companyimages

Versatile Controversy asto whether thedata are interval

Stapel Scale Unipolar ten-pointscale, -5 to +5,without a neutral

point (zero)

Measurementof attitudes andimages

Easy to construct,administer over telephone

Confusing anddifficult to apply

A Semantic Differential Scale forMeasuring Self- Concepts, Person Concepts,and Product Concepts

1) Rugged :---:---:---:---:---:---:---: Delicate



and Product Concepts) gg

2) Excitable :---:---:---:---:---:---:---: Calm

3) Uncomfortable :---:---:---:---:---:---:---: Comfortable

4) Dominating :---:---:---:---:---:---:---: Submissive

5) Thrifty :---:---:---:---:---:---:---: Indulgent

6) Pleasant :---:---:---:---:---:---:---: Unpleasant

7) Contemporary :---:---:---:---:---:---:---: Obsolete

8) Organized :---:---:---:---:---:---:---: Unorganized 9) Rational :---:---:---:---:---:---:---: Emotional

10) Youthful :---:---:---:---:---:---:---: Mature

11) Formal :---:---:---:---:---:---:---: Informal

Balanced and Unbalanced Scales



Jovan Musk for Men is Jovan Musk for Menis Extremely good Extremely good

Very good Verygood GoodGood BadSomewhat good Very bad Bad

Extremely bad Very bad

A variety of scale configurations may be employed to measure thegentleness of Cheer detergent. Some examples include:

Cheer detergent is:

Rating Scale Configurations



Cheer detergent is:1) Very harsh --- --- --- ---

--- --- --- Very gentle

2) Very harsh 1 2 3 4 5 6 7 Verygentle

3) . Very harsh.. Neither harsh nor gentle.. Very gentle

4) ____ ____ ____ ____ ____ ____ ____

Very Somewhat Neither harsh Somewhat GentleVery

harsh Harsh harsh nor gentle gentle

-3 -1 0 +1 +2-2 +3

Cheer

Thermometer Scale Instructions:

Some Unique Rating ScaleConfigurations



Thermometer Scale Instructions:

Please indicate how much you like McDonald’s hamburgers by

coloring in the thermometer. Start at the bottom and color up to thetemperature level that best indicates how strong your preference is.

Form:

Smiling Face Scale Instructions:

Please point to the face that shows how much you like the BarbieDoll. If you do not like the Barbie Doll at all, you would point to Face

1. If you liked it very much, you would point to Face 5.

Form:

1 2 3 4 5

Likevery

muchDislikeverymuch

1007550

250

1) Number of Categories Although there is no single, optimal

Summary of Itemized ScaleDecisions

Table 9.2



number, traditional guidelines

suggest that there should be between fiveand nine categories 2) Balanced vs. unbalanced Ingeneral, the scale should be balanced to obtainobjective data 3) Odd/ even no. of categories If a neutral or indifferent scale response is possible

from at least some of the respondents, an odd

number of categories should be used 4) Forced vs. non-forced In situations where

the respondents are expected to have noopinion, the accuracy of the data may beimproved by a non-forced scale 5) Verbal description

An argument can be made for labeling all or manyscale categories. The category descriptions

should be located as close to the responsecategories as

possible

6) Physical form A number of options should be tried and

Reliability of Measurements



Reliability

Scale

Evaluation

Validity Generalizabilit y

Test/Retest

Alternative Forms

InternalConsistenc

y

Content

CriterionConstruct

Discriminant

Nomological

Convergent

Potential Sources of Error onMeasurement

1) Other relatively stable characteristics of the individual that



1) Other relatively stable characteristics of the individual thatinfluence the test score, such as intelligence, social

desirability, and education.

2) Short-term or transient personal factors, such as health,emotions, fatigue.

3) Situational factors, such as the presence of other people,

noise, and distractions.4) Sampling of items included in the scale: addition, deletion,

or changes in the scale items.

5) Lack of clarity of the scale, including the instructions or the

items themselves.6) Mechanical factors, such as poor printing, overcrowding

items in the questionnaire, and poor design.

7) Administration of the scale, such as differences amonginterviewers.



Sampling Techniques



Good sampling techniques arerequired of all researchers.



Different types of Samples



Simple random sample Systematic sample

Stratified sample

Cluster sample

Proportional sample

Simple Random Sample



Table of Random Numbers Choosing numbers out of a hat

Systematic Sample



Systematically selects every nth

person

Stratified Sample



A stratified sample assures arandom sample, however the

sample has equal numbers

within a particular characteristic.

Cluster Sample



A sample is chosen because itis difficult to sample the entire

population, e.g., choosing all

members of a particular classrather than individuals.

A cluster sample is often easier

and less costly, but

generalizability is limited

because of an N of 1.



Things to Keep in Mind When Creating

Surveys



Clearly identify the survey purpose Outline the field of study

Avoid overlapping questions

Order questions in a logical format Simple to complex

Make sure questions are clear Eliminate ambiguities

Eliminate all grammatical errors

Pre-code data for computation



Validity in Research



174

Construct validity is present when there isa high correspondence between the scores

obtained on a measure and the mental

definition of a construct it is designed to

represent.

Internal validity is present when variation in

scores on a measure of an independent

variable is responsible for variations in

scores on a measure of a dependent

variable.

External validity is present when

generalizations of findings obtained in a

research study, other than statistical

eneralization are made a ro riatel

Construct Validation



175

Involves proceduresresearchers use to developmeasures and to makeinferences about a

measure‘s construct validity It is a continual process

No one method alone will

give confidence in theconstruct validity of your measure

Construct Validation Steps

From Schwab (1999)



176

Define the construct and develop

conceptual meaning for it

Develop/choose a measure consistentwith the definition

Perform logical analyses and empiricaltests to determine if observations

obtained on the measure conform to

the conceptual definition

From Schwab (1999)

Content validity

Factor analysis

Reliability

Criterion-related/

Convergent/

Discriminant/

Nomological

validity

Survey Instrument



177

Why is it important?

How to do it?

What are some of the best

practices?

Development

Instrumentation in

Perspective



178

Selection and application of atechnique that operationalizes

the construct of interest

e.g., physics = colliders e.g., MDs = MRI

e.g., OB = Job descriptive index

Instruments are devices withtheir own advantages and

disadvantages, some more

precise than others, and

Survey Instruments



179

3 most common types of instrumentation in social

sciences

Observation Interview

Survey instrumentation

Survey instrumentation Most widely used across

disciplines

Most abused technique---people

Why do we do surveys?



180

To describe the populations: What is

going on?

Theoretical reasons: Why is it going on? Develop and test theory

Theory should always guide survey

development and data collection

What construct does this scale measure?

(1)



181

1. Have a job which leaves you sufficient time for your personal or family life. (.86)

2. Have training opportunities (to improve your skillsor learn new skills). (-.82)

3. Have good physical working conditions (goodventilation and lighting, adequate work space, etc.).

(-.69)4. Fully use your skills and abilities on the job. (-.63)

5. Have considerable freedom to adapt your ownapproach to the job. (.49)

6. Have challenging work to do---work from which you

can get a personal sense of accomplishment. (.46)7. Work with people who cooperate well with one

another.(.20 )

8. Have a good working relationship with your manager.(.20 ) Adapted from Heine et al.

(2002)

What construct does this scale measure?

(2)



182

(2)

I would rather say ―no‖ directly, than risk being misunderstood.(12)

Speaking up during a class is not a problem for me. (14)

Having a lively imagination is important to me. (12)

I am comfortable with being singled out for praise or rewards.

(13) I am the same person at home that I am at school. (13)

Being able to take care of myself is a primary concern for me.(12)

I act the same way no mater who I am with. (13)

I prefer to be direct and forthright when dealing with people Ihave just met. (14)

I enjoy being unique and different from others in many respects.(13)

My personal identity, independent of others, is very important tome. (14)

I value being in good health above everything. (8)

Adapted from Heine et al.(2002)



183

Example:Computer

satisfaction

Construct Definition



184

Personal computer satisfaction is an emotionalresponse resulting from an evaluation of the speed,durability, and initial price, but not the appearanceof a personal computer. This evaluation is expected

to depend on variation in the actual characteristics of the computer (e.g., speed) and on the expectations aparticipant has about those characteristics. Whencharacteristics meet or exceed expectations, theevaluation is expected to be positive (satisfaction).

When characteristics do not come up to expectations,the evaluation is expected to be negative(dissatisfaction).

From Schwab (1999)



Construct Validity ChallengesFrom Schwab (1999)



186

Construct Variance

Observed Score Variance

Systematic Variance

Unreliability

Reliable

Contamination

Construct Valid

VarianceDeficiency

Scale Development Process

From Hinkin (1998)



187

Step1: Item Generation

Step 2: Questionnaire Administration

Step 3: Initial Item Reduction

Step 4: Confirmatory Factor Analysis

Step 5: Convergent/Discriminant

Validity

From Hinkin (1998)

Step 1:

Item Generation -Deductive Approach



188

Item Generation Deductive Approach

It requires:

(a) an understanding of the

phenomenon to be investigated;(b) thorough review of the literature

to develop the theoretical definition

of the construct under examinationFrom Hinkin (1998)

Step 1:Item Generation-Deductive

Approach



189

pp

Advantages: through adequate

construct definitions, items should

capture the domain of interest, thus to

assure content validity in the final scale

Disadvantages: requires the

researchers to possess working

knowledge of the phenomena; may notbe appropriate for exploratory studiesFrom Hinkin (1998)

Step 1:Item Generation - Inductive

Approach



190

pp

Appropriate when the conceptual basis may not

result in easily identifiable dimensions for which items

can then be generated

Frequently researchers develop scales inductively byasking a sample of respondents to provide

descriptions of their feelings about their organizations

or to describe some aspects of behavior

Responses classified into a number of categories bycontent analysis based on key words or themes or

using a sorting process

Step 1:Item Generation - Inductive

Approach



191

pp

Advantages: effective in exploratory

research

Disadvantages: Without a definition of construct under

examination, it is difficult to develop items that

will be conceptually consistent.

Requires expertise on content analyses Rely on factor analysis which does not

guarantee items which load on the same

factors share the same theoretical construct

Characteristics of Good Items



192

As simple and short as possible

Language should be familiar to targetaudience

Keep items consistent in terms of perspectives (e.g., assess behaviors vs.affective response)

Item should address one single issue(no double-barreled items)

Leading questions should be avoided

What about these items?



193

I would never drink and drive for fear of that I might be stopped

by the police (yes or no )

I am always furious (yes or no ) I often lose my temper (never to

always )

滿招損，謙受益



Content Validation Ratio



195

CVR =2 n e

- 1 N

n e

N

is the number of Subject Matter Experts (SMEs) rating the

selection tool or skills being assessed is essential to the job, i.e.,

having good coverage of the KSAs required for the job.

is the total number of experts

CVR = 1 when all judges believe the tool/item is essential;

CVR = -1 when none of the judge believes the tool/skill is essential;

CVR = 0 means only half of the judges believe that the tool/item is

essential.

How many items per construct?



196

4 - 6 items for most constructs.For initial item generation, twice

as many items should be

generated

Item Scaling



197

g

Scale used should generate sufficient variance

among respondents for subsequent statistical

analyses

Likert-type scales are the most frequently used insurvey questionnaire. Likert developed the scale to

be composed of five equal appearing intervals with a

neutral midpoint

Coefficient alpha reliability with Likert scales hasbeen shown to increase up to the use of five points,

but then it levels off

Step 2:

Questionnaire Administration



198

Sample size: Recommendationsfor item-to-response ratios

range from 1:4 to 1:10 for each

set of scales to be factor

analyzed

e.g., if 30 items were retained to

develop three measures, a sample

size of 150 observations shouldbe sufficient in exploratory factor

analyses. For confirmatory factor

analysis, a minimum sample size

Step 3: Initial Item Reduction



199

Interitem correlations of the variables tobe conducted first. Corrected item-total

correlations smaller than 0.4 can be

eliminated

Exploratory factor analysis. An

appropriate loadings greater than 0.40 and

/or a loading twice as strong on an

appropriate factor than on any other factor.

Eigenvalues of greater than 1 and a scree

test of the percentage of varianceexplained should also be examined

Be aware of construct deficiencyproblems in deleting items

Step 3:Internal Consistency

Assessment



200

Reliability is the accuracy or precision of a measuring

instrument and is a necessary

condition for validity

Use Cronbach‘s alpha to

measure internal consistency.

0.70 should be served as

minimum for newly developed

measures.



An example of coefficient



202

Subject

Item A B C Variance

1 6 5 4 1.00

2 6 4 5 1.00

3 5 3 3 1.33

4 4 4 4 .00

5 4 5 4 .33

3.67

Total 25 21 20 7.00

Variance of total = 7.0 ; Total of variance = 3.67

2

t

2

2

5

1

2

i

i

60.)0.7

67.30.7(

4

5)(

12

1

22

t

n

i

it

n

n

How High Cronbach Alpha

Needs to be?



203

In exploratory research wherehypothesized measures are

developed for new constructs,

the Alphas need to exceed .70

In basic research where you use

well-established instruments for

constructs, the Alphas need to

exceed .80.

In applied research where you

need to make decisions based

Step 4:Confirmatory Factor Analysis

(CFA)



204

Items that load clearly in an exploratory factor

analysis may demonstrate a lack of fit in a multiple-

indicator measurement model due to lack of external

consistency It is recommended that a Confirmatory Factor

Analysis be conducted using the item variance-

covariance matrix computed from data collected from

an independent sample.

Then assess the goodness of fit index, t-value, and

chi square

Step 5:

Convergent/Discriminant Validity

C t lidit h th i hi h



205

Convergent validity—when there is a high

correspondence between scores from two

or more different measures of the same

construct.

Discriminat validity---when scores from

measures of different constructs do not

converge.

Multitrait-Multimethod Matrix (MTMM)

Nomological networks---relationships

between a construct under measurementconsideration and other constructs.

Criterion-related validity

Convergent Validity

From Schwab (1999)



206

Construct

Measure B

Measure A



Theoretical Model: Org.Justice

(Colquitt, 2001, JAP)



217

Distributive Justice

Procedural Justice

Interactive Justice

Informational Justice

OutcomeSatisfaction

Rule Compliance

Leader Evaluation

Collective Self-esteem



Four Types of Scale Development Approaches in ChineseManagement Research

Farh, Cannella, & Lee (2006, MOR)



219

Expectations about Cultural Specificity

Etic Orientation Emic Orientation

Source of the scale

Use or Modify anExisting Scale

Translation Adaptation

Develop a NewScale

De-contextualization

Contextualization

Four Types of Scale Development Approaches in Chinese

Management Research



220

ScaleDevelopment

Approaches

Key Assumptions

MajorStrengths

MajorLimitations

Translationapproach

Target construct is equivalentacross cultures in terms of overall definition, content

domain, and empiricalrepresentations of the contentdomain

Availability of high qualityculturally unbiased Westernscales for target construct

Low developmentaltime and costs

Preserve the possibility

of a high level of equivalence

Allow for direct cross-cultural comparison of research findings

Difficulty in achievingsemantic equivalencebetween the Chinese and

Western scalesCulturally unbiasedWestern scales are hardto come by

Adaptationapproach

Target construct is equivalentbetween cultures in terms of overall definition and contentdomain

Availability of high qualityWestern scales for targetconstruct

Low to moderatedevelopmental time andcosts

Ease of scholarlyexchanges of researchfindings with theWestern literature

Difficulty in conductingcross-cultural research

Drastic adaptation maycreate new scale thatrequires extensivevalidation in the Chinesecontext



Should you use well-established scalesfrom the (western) literature or develop

local scales?



222

Align your measure with your theoretical orientation When you take an etic (universal or cultural invariant)

perspective to a research topic, you assume that theChinese context is largely irrelevant. Here your study isbased on general theories, and you should use well-

established measures in the literature. When you take an emic (cultural specific) perspective to a

research topic, you assume that the phenomenon is Chinesecontext specific. Here your study is based on contextembedded theories, and you should consider usingmeasures appropriate for the Chinese context.

When you do cross-cultural research, you try to studyphenomena common across societies. You model cultureexplicitly in your theories (either as a main or a moderatingeffect) and should apply measures that work in multiplecultural contexts.



223

A Close Look atItem Generation

Using Inductive

Approach

Item Generation Process

Content domain clarity?

low

Collect behavioral incidents

Key issueshigh

sampling/method



224

Classify into categories

Form dimensions from categories

Domain definition

Item development &refinement

Empirical testing

p g

classification/pan

el test

Empirical/

conceptual

Creativity &

insight

content validation

Research project in focus



225

Investigate theconstruct domain

of moral

leadership in thePRC…



Instruments…



tools researchers use to collect data for

research studies (alternatively called

―tests‖)

The types of instruments…



2. Affective Instruments

3. Projective Instruments

1. Cognitive Instruments



aptitude tests

…measure the intellect and abilities

not normally taught and often are

used to predict future performance…typically provide an overall score, a

verbal score, and a quantitativescore

2. Affective instruments...



Measure characteristics of individualsalong a number of dimensions and toassess feelings, values, and attitudes

toward self, others, and a variety of other activities, institutions, andsituations

Types of affectiveinstruments...



attitude scales

…self -reports of an individual‟s beliefs,

perceptions, or feelings about self,

others, and a variety of activities,institutions, and situations

…frequently use Likert, semantic

differential, Thurstone , or Guttmanscales



values tests

…measure the relative strength of an

individual‟s valuing of theoretical,

economic, aesthetic, social, political,and religious values



personality inventories

…an individual‟s self -report measuringhow behaviors characteristic of defined

personality traits describe thatindividual

3. Projective instruments...



Measure a respondent‘s feelings or thoughts to an ambiguous stimulus

Primary type of projectivetest...



associational tests

…participants react to a stimulus

such as a picture, inkblot or wordonto which they project adescription

Selecting an instrument...



2. identify and locate appropriateinstruments

1. determine precisely the type ofinstrument needed

3. compare and analyze instruments

4. select best instrument

Instrument sources…



Tests in Print

Burros‟ Mental Measurements Yearbook

PRO-ED Publications Test Critiques Compendium

ETS Test Collection Database

ERIC/AE Test Review Locator

ERIC/Burros Test Publisher Directory

Rules governing the selectioninstruments...



2. the highest reliability

1. the highest validity

3. the greatest ease of administration,scoring, and interpretation

4. test takers‟ lack of familiarity with

instrument5. avoids potentially controversialmatters

Administering the instrument...



2. ensure ideal testing environment

1. make arrangements in advance

3. be prepared for all probablecontingencies

Two issues in using instruments...



2. Reliability: the degree to which theinstrument consistently measures

what it purports to measure

1. Validity: the degree to which theinstrument measures what it purports

to measure

Types of validity...



2. Criterion-related validity

3. Construct validity

1. Content validity



1. Content validity: the degree to whichan instrument measures an intended

content area



forms of content validity …

…sampling validity : does the instrument

reflect the total content area? …item validity : are the items included on

the instrument relevant to the

measurement of the intended contentarea?



2. Criterion-related validity: anindividual takes two forms of an

instrument which are thencorrelated to discriminate betweenthose individuals who possess acertain characteristic from thosewho do not



forms of criterion-related validity …

…concurrent validity : the degree to whichscores on one test correlate to scoreson another test when both tests areadministered in the same time frame

…predictive validity : the degree to which atest can predict how well individual willdo in a future situation



3. Construct validity: a series of studiesvalidate that the instrument really

measures what it purports to measure

Types of reliability...



2. Equivalence

1. Stability

3. Internal consistency



1. Stability (“test-retest”): the degree to

which two scores on the same

instrument are consistent over time



2. Equivalence (“equivalent forms”): the

degree to which identical instruments

(except for the actual items included)yield identical scores



3. Internal consistency (“split-half”

reliability with Spearman-Brown

correction formula , Kuder-Richardson and Cronback‟s Alpha

reliabilities, scorer/rater reliability):the degree to which one instrumentyields consistent results

Terms associated with

instruments...



Data…

…the pieces of information researchers

collect through instruments toexamine a topic or hypothesis



Constructs…

…abstractions of behavioral factors

that cannot be observed directly andwhich researchers invent to explainbehavior



Variable…

…a construct that can take on two or

more values or scores



Raw scores…

…the number of items an individual

scored on an instrument



Measurement scales…

…the representation of variables so

that they can be quantified

Measurement scales...



1. nominal variables

Qualitative (categorical)

Quantitative (continuous)2. ordinal variables

3. interval variables

4. ratio variables



1. nominal (“categorical”): classifies

persons or objects into two or more

categories



2. ordinal (“order”): classifies persons

or objects and ranks them in terms of

the degree to which those persons orobjects possess a characteristic ofinterest



3. interval: ranks, orders, and classifiespersons or objects according to equal

differences with no true zero point



4. ratio: ranks, orders, classifies personsor objects according to equal

differences with a true zero point



Norm reference…

…provides an indication about how one

individual performed on aninstrument compared to the otherstudents performing on the same

instrument



Criterion reference…

…involves a comparison against

predetermined levels of performance



Self reference…

…involves measuring how an

individual‟s performance changesover time



Operationalize…

…the process of defining behavioral

processes that can be observed



Standard error of measurement…

…an estimate of how often a researcher

can expect errors of a given size onan instrument

Types of Research

Descriptive - attempts to describe

and explain conditions of the



and explain conditions of the

present. It relies on qualitative and

quantitative data gathered from

written documents, personalinterviews, test results, surveys, etc.

Often people will call this type of

research ―Survey Research‖


Because of its flexibility and the fact

that it deals with current topics



that it deals with current topics,

descriptive research is probably the

most popular form of research in

education today. It is also popular because data can

be collected from a wide variety of

sources.


Basic characteristics of descriptive



Basic characteristics of descriptiveresearch are: It provides a descriptive analysis of a given

population or sample. Any inferences are left

to the readers. Qualitative, quantitative or a combination of

both types of data can be presented.

Hypotheses or broad research questions areused .


Data Sources

Persons such as teachers



Persons such as teachers,students, parents, administrators,etc.

Documents such as policystatements, curricular guidelines.

Records such as studenttranscripts.


Research Tools

St t d i t i



Structured interviews.

Structured questionnaires and

surveys

Standardized tests.

Descriptive ResearchExamples

What are the characteristics of

agricultural education students?



agricultural education students?

What is the level of job

satisfaction of extension

agents?

Why do teachers leave

teaching?

Research methodology(459500)

Lecture 4



278

Lecture 4Tuesday 23/9/2008

Dr Jihad ABDALLAH

Source: Research MethodsKnowledge Base

http://www.socialresearchmethods.net/

Sampling Terminology • In most applied social research, we are interested in generalizing to specific

groups.

• The group you wish to generalize to is often called the population in yourstudy.

• This is the group you would like to sample from because this is the group



279

s s t e g oup you ou d e to sa p e o because t s s t e g oupyou are interested in generalizing to.

• Let's imagine that you wish to generalize to urban homeless males betweenthe ages of 30 and 50 in the United States. If that is the population of interest, you are likely to have a very hard time developing a reasonablesampling plan.

You are probably not going to find an accurate listingof this population, and even if you did, you would

almost certainly not be able to mount a national

sample across hundreds of urban areas.

So we probably should make a distinction between



280

So we probably should make a distinction between

the population you would like to generalize to (the

theoretical population ), and the population that will

be accessible to you (the accessible population). In this example, the accessible population might be

homeless males between the ages of 30 and 50 in six

selected urban areas across the U.S.

Once you've identified the theoretical and accessiblepopulations, you have to do one more thing beforeyou can actually draw a sample.

You have to get a list of the members of the



281

ou ave to get a st o t e e be s o t eaccessible population.

The listing of the accessible population from which

you'll draw your sample is called the samplingframe.

If you were doing a phone survey and selecting

names from the telephone book, the book would beyour sampling frame.



282

Finally, you actually draw your sample (using one of the many sampling procedures).

The sample is the group of people who you select tobe in your study.

The sample is not necessarily the group of people



283

p y g p p pwho are actually in your study.

You may not be able to contact or recruit all of thepeople you actually sample, or some could drop outover the course of the study.

The group that actually completes your study is asubsample of the sample -- it doesn't include non-respondents or dropouts.

Sampling is a difficult multi-step process and thatthere are lots of places you can go wrong.

In fact, as we move from each step to the next inidentifying a sample, there is the possibility of introducing systematic error or bias.



284

introducing systematic error or bias.

For instance, even if you are able to identify perfectlythe population of interest, you may not have access toall of them.

And even if you do, you may not have a complete andaccurate enumeration or sampling frame from whichto select. And, even if you do, you may not draw thesample correctly or accurately. And, even if you do,they may not all come and they may not all stay.

Statistical Terms in Sampling



285

The Sampling Distribution



286

Sampling Error

In sampling contexts, the standard error is calledsampling error.



287

Sampling error gives us some idea of theprecision of our statistical estimate.

A low sampling error means that we hadrelatively less variability or range in the samplingdistribution.

We base our calculation on the standard deviation of our sample. The greater the sample standarddeviation, the greater the standard error (and thesampling error).

The standard error is also related to the sample size.



288

pThe greater your sample size, the smaller the standarderror. Why? Because the greater the sample size, thecloser your sample is to the actual population itself.

If you take a sample that consists of the entirepopulation you actually have no sampling errorbecause you don't have a sample, you have the entirepopulation. In that case, the mean you estimate is the

parameter.

Probability Sampling methods

probability sampling method is any

method of sampling that utilizes



289

p gsome form of random selection.

Each unit in the study population hasa known probability of being selected

in the population).

Some Definitions

N = the number of cases in the sampling

frame



290

n = the number of cases in the sample

NCn = the number of combinations(subsets) of n from N

f = n/N = the sampling fraction

1- Simple Random Sampling

The simplest form of random sampling

is called simple random sampling



291

p p g It is used when the population is

homogenious.

Objective: To select n units out of N such that each unit and each NCn has anequal chance of being selected.

Procedure: Use a table of randomnumbers, a computer random numbergenerator, or a mechanical device toselect the sample.



292

Advantages and Disadvantages

Simple random sampling is simple to

accomplish and is easy to explain to



293

others.

But it requires an accurate list of thestudy population.

The members of the population may be

scattered over a large area (makes it

more difficult and more expensive to

achieve)

2- Stratified Random Sampling Sometimes called proportional or quota random

sampling.

Involves dividing the population into homogeneousb ( ll d t t ) d th t ki i l



294

subgroups (called strata) and then taking a simplerandom sample in each subgroup.

Procedure:

1. divide the population into non-overlapping groups(i.e., strata) N1, N2, N3, ... Ni, such that N1 + N2 +N3 + ... + Ni = N.

2. Then take a simple random sample f = n/N fromeach stratum.

proportionate stratified random sampling: When we

use the same sampling fraction within strata.

disproportionate stratified random sampling: When



295

p p a a a p g

we use different sampling fractions in the strata

Example Let's say that a population (N = 1000) is composed of

three groups:

- 850 Caucasians: 85%100 Af i A i 10%



296

- 100 African-Americans: 10%

- 50 Hispanic-American: 5%

If we just did a simple random sample of n =100 witha sampling fraction of 10%, we would expect by

chance alone that we would only get 10 and 5 persons

from each of our two smaller groups.

And, by chance, we could get fewer than that.

Using stratified sampling

Proportionate sampling:

- Caucasians: (100/1000) x 850 = 85African Americans: (100/1000) x 100 = 10



297

- African-Americans: (100/1000) x 100 = 10

- Hispanic-American: (100/1000) x 50 = 5

Disproportionate: for example:

- 50 Caucasians (50/850 = 5.88%)

- 25 African-Americans (25/100 = 25%)

- 25 Hispanic-American (25/50 = 50%).



298

Advantages It is preferred over simple random sampling when the

population is composed of different groups of

different sizes.h h l l h



299

It assures that the sample represents not only the

overall population, but also key subgroups of the

population, especially small minority groups.

Stratified random sampling will generally have more

statistical precision than simple random sampling.

This will only be true if the strata or groups are

homogeneous.

3- Systematic Random Sampling

Steps to follow in order to achieve a systematic

random sample:

1. Number the units in the population from 1 to N



300

2. Decide on the n (sample size) that you want or need

3. k = N/n = the interval size

4. Randomly select an integer between 1 to k

5. Then take every k th unit

It is essential that the units in the population are

randomly ordered, at least with respect to thecharacteristics you are measuring.

Example Let's assume that we have a population that only has

N=100 people in it and that you want to take a sampleof n=20.

To use systematic sampling, the population must be



301

y p g, p plisted in a random order.

The sampling fraction would be f = 20/100 = 20%.

in this case, the interval size, k, is equal to N/n =100/20 = 5.

Now, select a random integer from 1 to 5. In ourexample, imagine that you chose 4. Now, to select the

sample, start with the 4th unit in the list and takeevery k-th unit (every 5th, because k=5).



302

1. It is fairly easy to do. You only have to select

a single random number to start things off.

Advantages



303

2. It may also be more precise than simple

random sampling.

3. Finally, in some situations there is simply no

easier way to do random sampling.

4- Cluster (Area) Random Sampling

The problem with random sampling methods whenwe have to sample a population that's disbursed

across a wide geographic region is that you will haveto cover a lot of ground geographically in order to get



304

to cover a lot of ground geographically in order to getto each of the units you sampled.

Your interviewers are going to have a lot of traveling

to do. For this reason cluster or area random sampling

was invented.

Steps

Divide population into clusters (usually

along geographic boundaries)



305

Randomly sample clusters

Measure all units within sampled

clusters

Cluster sampling of five counties (marked in red in the figure).



306

5. Multi-Stage Sampling

When we combine sampling methods, we call this

multi-stage sampling.



307

For example, we may do cluster sampling as a first

stage, then do stratified sampling or simple random

sampling within each cluster.

Nonprobability Sampling Methods

The difference between nonprobability and

probability sampling is that nonprobability sampling

does not involve random selection and probabilitysampling does

http://www.socialresearchmethods.net/kb/sampprob.php

http://www.socialresearchmethods.net/kb/sampprob.php



308

sampling does.

Does that mean that nonprobability samples aren't

representative of the population? Not necessarily.

But it does mean that nonprobability samples cannot

depend upon the rationale of probability theory.

At least with a probabilistic sample, we know theodds or probability that we have represented thepopulation well. We are able to estimate confidenceintervals for the statistic.

With nonprobability samples, we may or may nott th l ti ll d it ill ft b



309

represent the population well, and it will often behard for us to know how well we've done so.

In general, researchers prefer probabilistic or randomsampling methods over nonprobabilistic ones, andconsider them to be more accurate and rigorous.

However, in applied social research there may becircumstances where it is not feasible, practical ortheoretically sensible to do random sampling.

We can divide nonprobability sampling methods intotwo broad types: accidental or purposive.

Most sampling methods are purposive in naturebecause we usually approach the sampling problem



310

y pp p g pwith a specific plan in mind.

The most important distinctions among these types of sampling methods are the ones between the differenttypes of purposive sampling approaches.

Accidental, Haphazard or Convenience

Sampling

Used in exploratory research where the researcheris interested in getting an inexpensive

approximation of the truth without incurring thecost of time required to select a random sample.



311

Examples:

1. "man on the street" interviews conducted frequentlyby television news programs to get a quick (although non-representative) reading of publicopinion.

2. Ask patients who come into a clinic or a hospital tovolunteer in a certain study

3. Take the first 10 people entering a supermarket.

Purposive Sampling In purposive sampling, we sample with a purpose in

mind.

We usually would have one or more specificpredefined groups we are seeking.



312

For instance, have you ever run into people in a mallor on the street who are carrying a clipboard and who

are stopping various people and asking if they couldinterview them?

Most likely they are conducting a purposive sample(and most likely they are engaged in market

research). For example, they might be looking forCaucasian females between 30-40 years old.

They size up the people passing by and anyone wholooks to be in that category they stop to ask if theywill participate.

One of the first things they're likely to do is verify

that the respondent does in fact meet the criteria forbeing in the sample.



313

g p

Purposive sampling can be very useful for situationswhere you need to reach a targeted sample quickly

and where sampling for proportionality is not theprimary concern.

With a purposive sample, you are likely to get theopinions of your target population, but you are also

likely to overweight subgroups in your populationthat are more readily accessible.

Types of purposive sampling

1. Modal Instance Sampling:

In statistics, the mode is the most frequently

occurring value in a distribution.



314

In sampling, when we do a modal instance sample,

we are sampling the most frequent case, or the

"typical" case. In a lot of informal public opinion polls, for

instance, they interview a "typical" voter.

There are a number of problems with this sampling

approach.

First, how do we know what the "typical" or "modal"case is?

We could say that the modal voter is a person who isof average age, educational level, and income in thepopulation.

But, it's not clear that using the averages of these is



315

the fairest (consider the skewed distribution of income, for instance).

And, how do you know that those three variables --age, education, income -- are the only or even themost relevant for classifying the typical voter?

What if religion or ethnicity is an important

discriminator? Clearly, modal instance sampling is only sensible for

informal sampling contexts.

2. Expert Sampling:

Expert sampling involves the assembling of a

sample of persons with known or demonstrable

experience and expertise in some area. Often, we

convene such a sample under the auspices of a



316

"panel of experts."

There are actually two reasons you might do expert

sampling. First, because it would be the best way to elicit the

views of persons who have specific expertise. In

this case, expert sampling is essentially just a

specific subcase of purposive sampling.

The other reason you might use expert sampling is toprovide evidence for the validity of another samplingapproach you've chosen.

For instance, let's say you do modal instancesampling and are concerned that the criteria you used

for defining the modal instance are subject tocriticism.



317

You might convene an expert panel consisting of persons with acknowledged experience and insightinto that field or topic and ask them to examine yourmodal definitions and comment on theirappropriateness and validity.

The advantage of doing this is that you have someacknowledged experts to back you.

The disadvantage is that even the experts can bewrong.

3. Quota Sampling: In quota sampling, you selectpeople nonrandomly according to some fixed quota.

There are two types of quota sampling: proportional and non proportional.

In proportional quota sampling you want to

represent the major characteristics of the populationby sampling a proportional amount of each.



318

y p g p p

For instance, if you know the population has 40%women and 60% men, and that you want a total

sample size of 100, you will continue sampling untilyou get those percentages and then you will stop.

So, if you've already got the 40 women for yoursample, but not the sixty men, you will continue tosample men but even if legitimate women

respondents come along, you will not sample thembecause you have already "met your quota."

Non-proportional quota sampling is a bit lessrestrictive.

In this method, you specify the minimum number of sampled units you want in each category. here, you're

not concerned with having numbers that match theproportions in the population.



319

p p p p

Instead, you simply want to have enough to assurethat you will be able to talk about even small groups

in the population. This method is the non-probabilistic analogue of

stratified random sampling in that it is typically usedto assure that smaller groups are adequately

represented in your sample.

4. Heterogeneity Sampling: We sample forheterogeneity when we want to include all opinionsor views, and we aren't concerned about representingthese views proportionately (sampling for diversity).

our primary interest is in getting broad spectrum of ideas, not identifying the "average" or "modali li id l



320

instance" ones (we sampling ideas not people).

We imagine that there is a universe of all possible

ideas relevant to some topic and that we want tosample this population, not the population of peoplewho have the ideas.

Clearly, in order to get all of the ideas, and especially

the "outlier" or unusual ones, we have to include abroad and diverse range of participants.Heterogeneity sampling is, in this sense, almost theopposite of modal instance sampling.

5. Snowball Sampling: In snowball sampling, youbegin by identifying someone who meets the criteriafor inclusion in your study. You then ask them torecommend others who they may know who alsomeet the criteria.

Snowball sampling is especially useful when you aretrying to reach populations that are inaccessible or



321

trying to reach populations that are inaccessible orhard to find.

For instance, if you are studying the homeless, you

are not likely to be able to find good lists of homelesspeople within a specific geographical area.

However, if you go to that area and identify one ortwo, you may find that they know very well who theother homeless people in their vicinity are and howyou can find them.



Reliability

Do the methods

andtools/instruments



produce consistent

results across

multiple

observations?

https://reader009.{domain}/reader009/html5/0428/5ae3de9957cc

Reliability and Validity (I)

• Reliability consists of demonstrating that the operations of a study -such

as the data collection procedures- can be repeated, with the sameresults

• It is a question of documenting the research procedure



PhD Seminar Series. Qualitative Research Methodology

K.E. Soderquist

• It is a question of documenting the research procedure

• Reliability is ensured by keeping data in different forms:

- Directly taken field notes - from interviews and observations,

- Expanded typed notes made as soon as possible after the fieldwork (this includes comments on problems and ideas that ariseduring each stage of the fieldwork and that will guide furtherresearch),

- A running record of analysis and interpretation (open coding

and axial coding).

Reliability and Validity (II)

• Construct validity means establishing correct operational measures forthe concepts being studied. It is ensured through:

- The use of multiple sources of evidence,- The establishment of a chain of evidence,



PhD Seminar Series. Qualitative Research Methodology

K.E. Soderquist

- Letting key informants review draft result reports

• External validity means establishing the domain to which a study'sfindings can be generalized. It is ensured through the use of a

replication logic “Analytical Generalisation”

- Relate case findings to existing or emerging bodies of literature,part of which will have been analysed in the literature section of the thesis

Introduction to

Questionnaire Design



Q g

Spring 2005 Seminar Series

Survey Research Laboratory

University of Illinois at Chicago

www.srl.uic.edu

Questionnaire designencompasses:

1. How to write questions



q

2. How to draft and organize a

questionnaire

The Art of AskingQuestions

You must ask the right question



Respondents must understand your

question

Respondents must know the answer

Respondents must be willing and

able to tell you the answer

Three Simple Rules for Writinga Good Questionnaire

1) Think through your research

questions and objectives before you write questions



q

2) Prepare an analysis plan before you

write questions

3) Ask yourself, in relation to points #1

and #2 above, if each question on

your list is necessary? Even if the

data would be „interesting‟ it has toultimately be used in analysis to

make the cut!

Types of Survey Questions

1) Those that ask about behaviors or facts Non-threatening behavior questions



g q

Threatening behavior questions

Demographics2) Those that ask about psychological states

or attitudes

3) Those that ask about knowledge

What Is A Good

Question?

One that yields a truthful, accurate answer

One that asks for one answer on one dimension One that accommodates all possible



contingencies of response

One that uses specific, simple language

One that has mutually exclusive response options One that produces variability in response

One that minimizes social desirability

One that is pretested

Ask questions one at atime

Bad question :

In the past 6 months, what major appliances

h h h ld

Better question :

Now I‟m going to reada list of household

li A I d



has your household

purchased new from the

store?

appliances. As I read

each one, please tell me

whether or not your

household has purchased this type of

appliance new from the

store during the past 6

months. How about…

a refrigerator?a kitchen range or

oven?

a microwave?

Ask questions one at a time

Bad Question:

Compared to one yearago, are you paying more,

l b t th

Better Question :

Compared to one yearago, are you now paying

l b t th



less, or about the same

for your auto and life

insurance?

more, less, or about the

same for …

a. auto insurance?

b. life insurance?

Specify

Specify who, what, when, where and how .

For example whose income? What‟s included?



For example, whose income? What‟s included?

Over what period of time? Example:

“In 2002, what was your total household income,

before taxes? Please count income from all

members of your household, including wages

from employment, disability, social security, and public aid.”

Specify through cues

Example

People drink beer in many places – for



example, at home, at restaurants, at bars,

sporting events, at friends‟ homes, etc.

During the past 30 days, did you drink

any beer?

Use words with singularmeanings

Ambiguous:

How would youcompare how close you are to

More Clear:

Compared to yourlast neighborhood,



close you are tofamily in yourcurrent

neighborhood tohow close you

were in your oldneighborhood?

Would you say

your family iscloser here,further here, orthe same?

do you now live

closer to your

family, are youfurther from your

family, or are you

about the same

distance?

Social Desirability

Respondents will try to represent

themselves to the interviewer in a way that

reflects positively on them



As questions become more threatening,

respondents are more likely to overstate orunderstate behavior, even when the best

question wording is used

Minimizing SocialDesirability

• For socially desirable behavior, it is better to ask whether

respondents have ever engaged in the behavior before asking

whether they currently engage in the behavior

• For socially undesirable behavior, it is better to ask about

current behavior first, rather than ask about their usual ortypical behavior



• Train interviewers to maintain a professional attitude

• Self-administered computer-assisted procedures can reduce

question threat and improve reporting on sensitive questions

• Longer questions reduce sensitivity when obtaining

information on frequencies of socially undesirable behavior

Open vs. Closed Questions

General rule: closed questions are usually

b tt



better

Easier for the respondent

Less coding later

Better to have respondent do categorizing

Categories help define the question

Disadvantages of

Closed Questions

Categories may be leading to respondents



g y g p

May make it too easy to answer without

thinking May limit spontaneity

Not best when

asking for frequency of sensitive behaviors there are numerous possible responses

General Principles for

Response Options

Response categories should be consistent with



gthe question

Categories must be exhaustive , including every

possible answer Categories must be mutually exclusive (no

overlap)

If appropriate, include a “don‟t know” category

Constructing Response Scales

Respondents can generally only remember a maximumof 5 responses unless visual cues are used



p

Using graphic images such as thermometers andladders and using card sorting for compex ratings is

effective Number of points in scale should be determined by how

you intend to use the data

With scales with few points, every scale can be labeled;

in longer scales, only the endpoints are labeled

Ordering ResponseCategories

• Usually better to list responses from the lower

level to the higher level• Associate greater response levels w/ greater



g p / g

numbers

•

Start with end of a scale that is least sociallydesirable

Should you use a middle category?

Common practice is to omit it to push respondents(Rs ) toward one end or the other, on the theory thatfew individuals are truly in the middle on a particular



y pissue

Evidence from empirical studies shows that use of

an explicit middle alternative will often be taken byRs in a forced choice situation if offered; at the sametime, it does not affect the ratio of „pro‟ to „con‟responses or the size of the don‟t know category

Our usual recommendation is to include it unlessthere are persuasive reasons to exclude

Part 2:

How to draft and organize yourquestionnaire



How to Start the Questionnaire

Start with easy questions that all respondentscan answer with little effort



First questions should also be non-threatening

Don‟t start with knowledge or awarenessquestions

First questions should be directly related to thetopic as described in the introduction or

advance/cover letter

Survey Intro/Cover Letter

Introduction should indicate:

h i d ti th



who is conducting the survey

the topics to be covered in the survey

an assurance of confidentiality

any IRB stipulations

whether you offer how long it will take

depends on mode, topic, population

Physical Format of the Self-

Administered Questionnaire

Careful formatting is necessary to



Careful formatting is necessary to

decrease errors and increase motivation

Respondent‟s needs must always take priority, followed by interviewer and data

processors

Physical Format Checklist

Number all questions sequentially Use large, clear type; don‟t crowd

„Whi ‟ Pl bl k b



„White space:‟ Place more blank space between

questions than between subcomponents of questions

List answer categories vertically instead of horizontally Avoid double/triple „banking‟ of response choices

Be consistent with direction of response categories

Be consistent with placement of response categories

Physical Format, Continued

Don‟t split questions across pages. If necessary(e.g., question requires 1.5 pages), restate

question and response categories on next page



question and response categories on next page

Put special instructions on questionnaire as

needed, next to question Distinguish directions from questions

Precode the questionnaire (vs. check boxes)

Mail questionnaires

Include a cover letter and contact information if the respondent needs help

U b kl f



Use a booklet format Easier to turn pages

Prevents lost pages Permits double-page formats

Looks more professional

Include a title, graphic, name/address of sponsor on

cover

Testing the Questionnaire

Preferable to test the questionnaires with



people like those in your main study

study population Test in same mode to be used for main

study

Consider cognitive pretesting

Questionnaire Design Steps

1. Decide what information is needed2. Search for existing questions



3. Focus groups

4. Draft new questions/revise existing ones5. Sequence the questions

6. Get peer evaluation

7. Revise and test on self/co-workers


8. Think-aloud interviews9. Revise/eliminate questions

10 P i i i i f



10. Prepare interviewer instructions for pilot test

11. Pilot test (10-20 cases)12. Revise eliminate questions based on

respondent & interviewer comments


13. Pilot test again, if necessary14. Prepare final interviewer instructions



15. Be prepared to modify questionnaires if

interviewer training raises problems

16. After interviewing is complete, debrief

interviewers for potential problems

17. Use experience from one study for

future planning

Reliability and Validity of Research Instruments



An overview

Measurement error

Error variance--the extent of

variability in test scores that isattributable to error rather than a



true measure of behavior.

Observed Score=true score +

error variance

(actual score obtained) (stable score) (chance/random error)

(systematic error)

Validity

The accuracy of the measure in

reflecting the concept it issupposed to measure.



pp

Reliability

Stability and consistency of the

measuring instrument.

A measure can be reliable



A measure can be reliable

without being valid, but it cannot

be valid without being reliable.

Validity

The extent to which, and how

well, a measure measures aconcept.



face

content construct

concurrent

predictive criterion-related

Face validity

Just on its face the instrument

appears to be a good measureof the concept. ―intuitive, arrived



at through inspection‖

e.g. Concept=pain level Measure=verbal rating scale ―rate

your pain from 1 to 10‖.

Face validity is sometimesconsidered a subtype of content

validity.Question--is there any time when face validity is

Content validity

Content of the measure is justified byother evidence, e.g. the literature.



Entire range or universe of the construct

is measured. Usually evaluated and scored by experts

in the content area.

A CVI (content validity index) of .80 or more is desirable.

Construct validity

Sensitivity of the instrument to pick up

minor variations in the concept being

measured.Can an instrument to measure anxiety pick up different

levels of anxiety or just its presence or absence?



levels of anxiety or just its presence or absence?

Measure two groups known to differ on the construct.

Ways of arriving at construct validity Hypothesis testing method

Convergent and divergent

Multitrait-multimatrix method

Contrasted groups approach

factor analysis approach

Concurrent validity

Correspondence of one

measure of a phenomenon withanother of the same



construct.(administered at the same time)

Two tools are used to measure thesame concept and then a

correlational analysis is performed.

The tool which is already

demonstrated to be valid is the ―goldstandard‖ with which the other

measure must correlate.

Predictive validity

The ability of one measure to

predict another future measureof the same concept.



If IQ predicts SAT, and SAT predicts QPA, then shouldn‘t IQ

predict QPA (we could skip SATs for admission

decisions)If scores on a parenthood readiness scale indicate levels of

integrity, trust, intimacy and identity couldn‘t this test be

used to predict successful achievement of the

devleopmental tasks of adulthood?

The researcher is usually looking for a more efficient

way to measure a concept.

Criterion related validity

The ability of a measure to

measure a criterion (usually set bythe researcher).

If th it i t f f i li i



If the criterion set for professionalism is

nursing is belonging to nursing

organizations and reading nursing journals,then couldn‘t we just count memberships

and subscriptions to come up with a

professionalism score.

Can you think of a simple criterion to measureleadership?

Concurrent and predictive validity are often

listed as forms of criterion related validity.

Reliability

Homogeneity, equivalence and stability

of a measure over time and subjects.

The instrument yields the same resultsover repeated measures and subjects.

E d l ti ffi i t (d f



Expressed as a correlation coefficient (degree of

agreement between times and subjects) 0 to +1.

Reliability coefficient expresses the relationshipbetween error variance, true variance and the

observed score.

The higher the reliability coefficient, the lower the error

variance. Hence, the higher the coefficient themore reliable the tool! .70 or higher acceptable.

Stability

The same results are obtained

over repeated administration of the instrument.



Test-restest reliability

parallel, equivalent or alternateforms

Test-Retest reliability

The administration of the same

instrument to the same subjectstwo or more times (under similar

di i b f d f )



conditions--not before and after treatment)

Scores are correlated andexpressed as a Pearson r.(usually .70 acceptable)

Parallel or alternate formsreliability

Parallel or alternate forms of a

test are administered to thesame individuals and scores are



correlated.

This is desirable when theresearcher believes that

repeated administration will

result in ―test-wiseness‖ Sample: ‖I am able to tell my partner how I

feel‖

―M artner tries to understand m

Homogeneity

Internal consistency

(unidimensional) Item-total correlations



split-half reliability

Kuder-Richardson coefficient Cronbach‘s alpha

Item to total correlations

Each item on an instrument is

correlated to total score--an itemwith low correlation may be

d l d Hi h d l



deleted. Highest and lowest

correlations are usuallyreported.

Only important if you desire

homogeneity of items.

Spit Half reliability

Items are divided into two

halves and then compared.Odd, even items, or 1-50 and

51 100 t t lit



51-100 are two ways to split

items. Only important when homogenity

and internal consistency is

desirable.

Kuder-Richardson coefficient(KR-20)

Estimate of homogeneity when

items have a dichotomousresponse, e.g. ―yes/no‖ items.



Should be computed for a test

on an initial reliability testing,and computed for the actual

sample.

Based on the consistency of responses to all of the items of a

single form of a test.

Cronbach‘s alpha

Likert scale or linear graphic

response format. Compares the consistency of



response of all items on the

scale. May need to be computed for

each sample.

Equivalence

Consistency of agreement of

observers using the samemeasure or among alternate

f f t l



forms of a tool.

Parallel or alternate forms(described under stability)

Interrater reliability

Intertater reliability

Used with observational data.

Concordance between two or more observers scores of the



same event or phenomenon.

Critiquing

Was reliability and validity data

presented and is it adequate? Was the appropriate method



used?

Was the reliability recalculatedfor the sample?

Are the limitations of the tool

discussed?

Educational Research &

StatisticsResearch Application to



8/2/2012 Marian College 380

PracticeSummer 2003Dr. Chiang

Major Components of the Course

Understanding basic research principles

and methods



Marian College 381

and methods

Becoming familiar with educational

statistics

Basic descriptive & inferential statistics Using SPSS to analyze data

Descriptive Statistics*

Measures of central tendencies

Mode



Marian College 382

Mode

Median

Mean

Measures of variance

Range

Standard deviation

Examples of Research Questions

Do women and men do equal amount of housework?(gender equity?) Definition of housework



Marian College 383

Data gathering procedure

Are black drivers more likely to be ticketed for speeding than white drivers? (racial profiling?) Research method

Validity threats

Does hormone-replacement therapy (HRT) do more

harm than good for women with PMS? (estrogenstudy, July 2002) Sample

Statistics

Variables*

Independent (grouping) variable (IV)

Dependent (test) variable (DV) IV precedes DV

IV is to be experimentally manipulated

DV is to be measured



Marian College 384

DV is to be measured

A hypothesis should contain only one IV

and one DV

Using t-tests to compare

the means* Why is t-test considered an inferential

statistics?

When do we use independent t?



Marian College 385

To compare two groups that are mutually

exclusive (e.g. experimental vs. controlgroups, males vs. females)

When do we use dependent (correlated)t? To compare pretest vs. posttest (paired

samples t)

To compare two groups that have been

Sampling*

purpose random sampling:

Each individual in the population has an



Marian College 386

Each individual in the population has anequal & independent chance to be

selected.

stratified sampling

Either proportional or not proportional to

population systematic sampling

cluster sampling

When should we have larger N?*

For studies of significant consequence

If the sample is very diversified

Minute differences are expected



Marian College 387

Minute differences are expected

For longitudinal studies If you are to have subgroup analyses

Attrition of subjects are anticipated

Test measures are unreliable Variables are complex and difficult to

control

Use chi-square to analyzesurvey results

Chi-square is a non-parametric test (n > 5per cell)

The data are in frequency counts

Compare observed frequencies vs.expected frequencies



Marian College 388

expected frequencies

Arrange data in 2 X 2, 2 X 3 contingency

tables Use Crosstab function in the SPSS

program to obtain crosstabulations and chisquares

RESEARCH DESIGN



Planning the Research Design

I. RESEARCH DESIGN (Methodology)

• The complete sequential steps the

researcher undertakes in order toachieve the goal of the study(Cadornigara 2002)



(Cadornigara, 2002)

• An intelligent plan of the researcher inpursuing the goals of the study.

• It describes in sufficient detail theprocedures employed in the research so

it can be evaluated and repeated if necessary.

CONTENT

• It includes the following:

1. All processes done duringactual experimentation

2 All materials and amounts



2. All materials and amounts

used in the study3. Description of experimental

and control set-ups

4. Kind of data gathered

5. Number of trials and replicatesdone

6. Description of the samples and reference

population

7. Management of sample plants and/or animals

8. Sampling techniques

9. Identification and classification of variables

10. Chemical, physical and microbiological analysesof samples

11 Manner of data collection organization and



11. Manner of data collection, organization andprocessing

12. Statistical analysis (test of significance)

13. Limitation in the methods that have beendiscovered during the study

KINDS OF RESEARCHDESIGN (Research Method)

1. DESCRIPTIVE METHOD

- Involves a careful study,observation and detaileddescription of living or non-



description of living or nonliving things and phenomenaas they occur in nature.

- Includes studies that makecomparisons and evaluations

of science concepts,techniques and procedures.

II. RESEARCH MATERIALS

- List of all the materials, reagents,

chemicals, plants, animals, andother experimental units as well

as equipment that will be used



as equipment that will be used

in performing the experiment

III. RESEARCHPROCEDURE

- the experimentation process



Introduction to




396

January 2007

Today and Tomorrow‘s Agenda

What is Methodology?

Types of Research Studies

Formulating Research Questions and Objectives Research Designs

Research Designs cont‘d

Sampling Strategies



397

Sampling and Non-Sampling Errors

Methodology Limitations Measurement Scales

Designing Data Collection Tools

Basic Qualitative Analysis

Writing Research Proposals

Methodology

A set of procedures for the

purpose of answering aresearch question(s) that

describes:



398

describes:

How study participants will be

selected

How you will analyze the data How and when you will gather

data from artici ants

Types of Research Studies



399

Types of Research

Research Types

Exploratory Conclusive



400

Descriptive Causal

Experimental

Observational

Exploratory Research

Conducted as the first step in

determining appropriate action; Helps clearly outline the

information needed as part of



401

information needed as part of

any future research; Tends to rely on qualitative

research techniques such as

in-depth case studies, one-on-one interviews, and focus

groups.

What are some of the exploratoryresearch studies that you have

conducted?

http://www.ryerson.ca/~mjoppe/ResearchProcess/QualitativeResearch.htm



402

conducted?

Types of ExploratoryResearch

Baseline study (CSR)

Needs assessment (DOC)



403

Conclusive Research

Conclusive research tends to bequantitative research

It f th b b di id d i t



404

It can further be sub-divided into

two major categories:descriptive and causal.

Conclusive Research


Provides data (usuallyquantitative) about the

population being studied



405

population being studied.

It can only describe thesituation, not what caused it .

Conclusive Research

Causal Research

To determine whether there is acause and effect relationship

between variables



406

between variables

To determine whether a specificindependent variable isproducing an effect on another

dependent variable .

Causal Research

There are two types of causal

research: Experimental

Observational (quasi



407

Observational (quasi-

experimental)

Causal Research

There are two types of causal

research: Experimental

Observational (quasi



408

Observational (quasi-

experimental)

Experimental and observational

studies try to demonstrate acausal relationship between two

variables.

Causal Research

Experimental Research: In

experimental studies, units(people, etc.) are put into control

or exposure groups by the



409

p g p y

researcher .

Causal Research

Observational Research:

In an observational study,members of the control group

are pre-determined They can



410

are pre determined. They can

be matched according todemographic information to a

member of the exposure group.

Causal Research

Think of some examples of

causal research Are they



411

causal research. Are they

experimental or observational?

Causal Research

Examples of causal research:

A drug trial for a new medicationthat has not yet been approved

by the FDA



412

by the FDA.

A study testing the long-termhealth effects of exposure to

high levels of radiation.

A study comparing asthma ratesamong children who live on

farms with those living in urban

Types of Research

Research Types

Exploratory Conclusive



413

Descriptive Causal

Experimental

Observational

What you gather data on:

Research Questions and

Objectives



414

j

Steps in Formulating a

Research Question

Steps:

Formulate initial researchquestion

Literature review based on initial



415

Literature review based on initial

question Refine research question

How and when you gather data:

Research Designs



416

Research Designs

Research Design

Primary vs. Secondary Data Quantitative vs. Qualitative

Longitudinal vs Non



417

Longitudinal vs. Non-

Longitudinal

Using Control / Comparison

Groups

Research Design

What‘s the difference betweenprimary and secondary data?



418

Data Sources

Primary Data

Observations Direct communication with

subjects (surveys, interviews,

http://www.ryerson.ca/~mjoppe/ResearchProcess/PrimaryResearch.htm



419

etc.)

Secondary Data

Existing data sources collectedfor some other purpose than the

proposed study (reports,databases, results of paststudies or surve s .

Qualitative vs. QuantitativeResearch



420

What are the advantages and

disadvantages of qualitative and

quantitative research?

When should you use quantitative



421

When should you use quantitative

or qualitative research?

Quantitative vs. Qualitative

Quantitative

research: Structured

research

instruments

Qualitative

research: Less structured

instruments

S ll l



422

instruments

Larger samplesize

Results easily

replicated Information

about how often

Smaller sample

size

Results difficult

to replicate

Informationabout why and how

Quantitative vs. Qualitative

Quantitative

research: Researcher

should know

clearly what he /

Qualitative

research: Researcher

may only know

roughly what he



423

clearly what he /

she is lookingfor

Statistical

analysis

roughly what he

/ she is lookingfor

Analysis is

subjective

Can be used to

determine the

-

Qualitative or Quantitative?

How did you develop your

strategic plan? Who participated in the strategy

development process?



424

p p

What are the major achievements since this plan

was written?

What suggestions do you havefor improving either the strategy

itself or the strategic planning


How satisfied were you with the

service you received today?

Completely dissatisfied



425

Somewhat dissatisfied Neither satisfied nor dissatisfied

Somewhat satisfied

Completely satisfied


How regularly do you review

progress against the plan andmake adjustments as needed?

Once a month



426

2-3 times a year Annually

Every other year

Never


Do you like your job?



427


Why do you like your job?



428


Have you or a family member ever

attended Disneyland before?

Yes



429

es

No

Don‘t Know


Is there anything else that youwould like the management to

know about the service that you



430

received today?

Qualitative vs. Quantitative

What would be most appropriate?

Community needs assessment Study of the frequency of

various reported health issues



431

p

among homeless women Impact evaluation

Population census

What Kind of Research

Design?


Longitudinal vs. Non-Longitudinal

With or Without a Control Group (whoassigns?)

Community needs assessment



432

Community needs assessment

Study of the frequency of various reportedhealth issues among homeless women

Study on whether exposure duringchildhood to excessive sun leads toincreased risk of skin cancer


Design?




Study of whether cross-border communityd l t j t lt i h f



433

development projects result in changes of

perception among community members of the ―other‖ ethnic group

Clinical drug trial

Study of changes in attitudes amongYerevan residents towards racially mixedmarriages


Design?




Study of the long term health effects of



434

Study of the long-term health effects of

exposure to nuclear fallout

Study to try to determine whether there is alink between childhood immunizations and

mental disability

DOC Program Design

From whom you gather your data:Samples and Sampling Strategies



435



436

What is a Sample?

A sample is a finite part of apopulation whose properties are

studied to gain information



437

about the whole.

Samples and Sampling

Strategy

The degree to which your sample mirrors the population

from which it comes will depend



438

to a large extent on your sampling strategy.

Why is this important?

Sampling Strategy

The sampling strategy is theway in which you select units

from the population for inclusion



439

into your study.

Sampling Frame

A list of all the individuals (units)

in the population from which the



440

sample is taken.

What are some examples of

sampling frames?

Sampling Frame

Examples of Sampling Frames:

List of businesses registeredwith the Chamber of Commerce

The phone book



441

List of clients served by aresource center

List of labor migrants registered

with authorities in a particular city

Sampling Frame

What could you use as a sampling

frame? A study on CSR practices in

Armenia



442

A study on the health of homeless women

A study of the reading habits of

children between the ages of 6and 8 in Yerevan

A stud of the attitudes of

Probability Samples

Sampling Strategies

Non-Probability Probability

Simple Stratified Systemic Cluster



443

Random Random

y

Random Sample

Probability vs. Non-

Probability Sampling

What‘s the primary differencebetween probability and non-

probability sampling?



444


Types:

Simple Random: Units are randomly

chosen from the sampling frame Stratified Random: Random

sampling of units within categories(strata) that are assumed to exist



445

( )

within a population Systemic Random: Number units

within the sampling frame and selectevery 5th, 10th, etc.

Cluster Sample: Clusters (each withmultiple units) within a samplingframe are randomly selected.




Stratified Random Sample:

If you want to conduct interviewswith businesses in NYC about

their SI practices, you could



447

categorize your list of businesses into small, medium

and large. Within each strata

you could then randomly select

a small number.

Non-Probability Samples

Sampling Strategies

Probability Non-Probability

Convenience Purposive Quota



448

Convenience

Sampling

Purposive

Sampling

Quota

Sampling

Non-Probability Sampling

Types:

Convenience sampling: selectionbased on availability or ease of inclusion

Purposive sampling: selection of individuals from whom you may be



449

individuals from whom you may beinclined to get more data

Quota sampling: selection on thebasis of categories that are assumed

to exist within a population

What are some examples of these?

Non-Probability Sampling

Types:

Convenience sampling:selecting individuals whohappen to be walking down thestreet



450

Purposive sampling: selectingresource center clients that usemany services

Quota sampling: selectingbusinesses for a survey that fallinto the categories of small,

Sample Size

Quantitative

Research:

A function of the

variability or

variance one

expects to find in

Qualitative

Research:

As big as

possible

No definite rules



451

p

the population(standard

deviation), and the

statistical level of

confidence(usually 95%) one

wishes to use.

to be followed

Sampling and Non-Sampling

Errors: Threats to Reliability andInternal Validity



452

Reliability

The extent to which results areconsistent

A study has high reliability if its



453

results hold true across differentsettings and participants.

If a study is reliable its results

will generalize to the larger population.

Internal Validity

Your research has high internal validity when it has successfully

measured what it set out to



454

measure.

How you gather data:

Measurement Scales



455

What is a Measurement

Scale?

A scale used to measureresponses to closed-ended

questions (quantitative data)



456

Measurement Scales

Types:

Nominal Scale Interval Scale

Ordinal Scale



457

Ratio Scale

Nominal Scales

Categorizes events, attributes or characteristics.

Does not express any values or



458

relationships between variables.

Nominal Scales

What is your sex?

Male Female

Unsure / Neither



459

Labeling men as "1" and women as

"2" does not mean that women are

―twice something" when compared to

men. Nor does it suggest that 1 issomehow better than 2.

Ordinal Scales

Categories have a logical or ordered relationship to each

other.



460

The specific amount of difference between points on the

scale can not be measured.

Very common in marketing,satisfaction and attitudinal

research.

Ordinal Scales

Example:How would you rate the service of

our wait-staff?



461

(1) Very Good (2) Good (3) Fair (4) Poor (5)

Very Poor

Interval Scales

Rank items in order The distance between points on

the scale are equal.



462

No starting or ending points

Interval Scales

Example:

The Fahrenheit scale is an intervalscale since the distance

between each degree is equal



463

but there is no absolute zeropoint.

Ratio Scales

Items are ranked in order The scale consists of equidistant

points and has a meaningful



464

zero point Each category should be the

same size

Categories should never overlap

Ratio Scales

Examples:

age, income, years of participation, etc.



465

What is your age?0-15 16-30 31-45 46-

60 61+



How you gather data:

Designing Tools for Gathering



467

Primary Data

Data Gathering Tools

What tools are you familiar withfor gathering primary data?



468

Data Gathering Tools

What do you understand by

these? Survey

Confidential



469

Anonymous

Designing Questionnaires

Question Order?



470


Orient the respondent!

First, briefly describe thepurpose of the research study,

explain how data gathered by



471

the survey will be used, and bywhom it will be used.

Is the survey confidential or

anonymous?


First Questions

The first several questionsshould be relevant to the study

itself so that the respondent



472

quickly understands what thesurvey is about and becomes

engaged.

The first questions should be

straightforward with relatively

few categories of response.


First Questions

Do not place sensitive questionstoo early! They can lead the

respondent to abandon the



473

survey and result in a high non-response rate for the whole

survey.


Middle Questions

Respondents should be easedinto sensitive topics by asking

them what they think is



474

important or what they prefer. Do not first ask respondents to

agree or disagree with a position

or sensitive issue.


Final Questions

Put demographic questions atthe end of the questionnaire.

Since they are easy to answer,



475

they are much better at the endwhen respondents are getting

tired.


Sequence your questions logically!

Questions should be grouped insections

Each section should have a logical

sequence



476

Avoid making the respondent jumparound mentally

Help respondents shift gears by

introducing a new section. For example, ―Now, we would like to get

your opinion on some related areas".

Pre-Testing Questionnaires

Step 1

Administer the tool to a smallgroup of people, who know little

or nothing about the research .



477


Can they clearly understand

what is being asked? Does the flow of the questions

make sense?



478

Will other people have difficulty? Which questions in particular

might pose problems?


Step 2

Test your tool with a smallnumber of people from your

sampling frame



479


Are there too many "neutral", "don‘tknow" or "don‘t remember"

responses? Do you need additional questions

relevant to the research?

Do you need to provide more space



480

for written responses? Did respondents respond

appropriately to open-endedquestions?

Will other people have difficulty?

Which questions in particular mightpose problems?

Qualitative Data Analysis



481

Qualitative Data Analysis

Steps of Qualitative Data Analysis:

Data transcription

Organizing data & numbering

Familiarization



482

Initial coding

Combining and renaming codes

Re-coding

Generating & summarizingthemes

Coding

A code is a name given to theideas or concepts that emerge

from the qualitative data

gathered. ―Open coding‖ is the



483

g p g

process of developing codes as

you review the transcripts or

notes.

Coding

How did the heart attack impactyour life?

―I became afraid to stay at homealone.‖

―I decided to start taking care of



484

my health.‖ ―I made a decision to spend more

time with my family.‖

―I am always afraid that I will haveanother.‖

―

Coding

Preliminary Codes: ―Fear‖ and―Change‖

―I became afraid to stay at homealone.‖

―I decided to start taking care of



485

my health.‖ ―I made a decision to spend more

time with my family.‖

―I am always afraid that I will haveanother.‖

―

Research questions

Descriptive questions

Where do children ages 10-13 choose

to read?

How often do they read?

How long do they read at a stretch?

Where do they go to choose books?



What do they look at when choosing a

book?

What do children‘s parents read? How

often? Where? For how long? Do/did children‘s parents read with

them?

Method

population

Describe who/what will be

studied Describe how the subjects will

be recruited/obtained. Discuss

your selection criteria/methods



Explain/justify the sample size

Method

instruments

Questionnaires

Field guides

Participant screeners

Usability test plan

Pre- or post-test materials



Surveys

Other data collection/data recording

methods

Data processing methods

Making sense of unstructured

data (interviews, think alouds)

Read it all, to get a sense of thewhole. Jot down any initial

categories that seem persistent Pick one document, go through it

thoroughly, write category notes inmargins. Repeat for a few more

d t



documents. Make a list of topics. Do some

mapping, to group similar topics.Sort into major, minor, and leftover

topics. Code your data, to see if you get

interestin atterns. Use

Making sense of your data

(cont.)

Rewrite category labels to bevery descriptive, very brief. Tryto reduce your total number of categories.

Make sure category/data

i i till t



mapping is still accurate.Recode data if necessary.

Group data for each category,analyze the groups. (Again,recode data if necessary.)

Draw final conclusions.

Research methodology

Quantitative Methods

Qualitative procedures



Quantitative Methods

A definition

A survey or experiment that provides

as output a quantitative or numeric

d i ti f f ti f th



description of some fraction of the

population, called the sample.

Components of a survey

method

The survey design The population and

sample

Th i t t ti



The instrumentation Variables in the study

Data analysis

The survey design

Purpose of the survey

The research question Type of survey

Cross sectional

L it di l



Longitudinal Form of data collection

The population and sample

Description of the population

Sampling design Single stage

Multistage

St tifi d



Stratified Sample selection

The instrumentation

The instrument (tool) Existing

New

Rating scale Likert scale: Rating the Items. 1-to-5 rating scale where:

1. = strongly unfavorable to the concept

2. = somewhat unfavorable to the concept

3. = undecided

4. = somewhat favorable to the concept

5 = strongly favorable to the concept



5. = strongly favorable to the concept

Pilot

Administration Postal survey

email

Variables and analysis

The research question

Variable in the research E.g. Number of years of academic study

The questions in the instrument E.g. How many years of study in a

University As an undergraduate?



As an undergraduate?

As a postgraduate?

Data analysis Steps

Bias in the data Non-response

Statistics, e.g. mean, standard deviation etc.

Components of an

experimental method

Subjects

Instruments and materials

The experimental design



Subjects

Selection

Conveniently

Random (RCT)

Group assignment

Random

Matched E g Ability Age



Matched. E.g. Ability, Age

Size

Variables

Dependent

Independent

Randomized Controlled Trial

(RCT) A true experiment, in which the researcher

randomly assigns some patients to at least

one maneuver (treatment) and other patientsto a placebo, or usual treatment. Key features

= the classic way to evaluate effectiveness of

drugs (or exercise, diet, counseling). Patients

are followed over time (Prospective) If



are followed over time (Prospective). If properly done, an RCT can be used to

determine cause and effect

Instrumentation and Materials

Description

Validation Pilot

Content validity

Prediction validity



Prediction validity Materials

The experimental design

Type

Pre-experimental No control group

Quasi-experimental

Control group, but not randomly

assigned



assigned

Single subject design (over time)

Pure experiment

Repeated measures Change groups

• Historical routes in anthropology

• Generates new understanding by naming and framingconcepts and themes

• Removes bias by questioning preconceived assumptions of

the social group under study

•

Promotes neutrality through adoption by the researcher of

Overview of Qualitative Research

Design



Promotes neutrality through adoption by the researcher of naïve stance or critical discussion, challenges pre-conceived

assumptions of both the researcher and the social group under

study

• Produces new understanding about the world, changes the

way power, culture and social interaction are understood

•Observation (Videoed, non-participant, semi-

participant and participant observation, field notes)•Interviews (individual and group - known as focus

groups, tape recorded and transcribed, field notes)

•Secondary data analysis (using written material

collected for purposes other than research)

Q e ti i e ( t t ed t l i te ie )

Data Collection in Qualitative Research



•Questionnaires (unstructured, postal, interviews)

•A mixture of all four

In qualitative research questions are open-ended.

Sometimes a check list or topic guide will be used

by the researcher to ensure all the relevant areas

are covered. This is known as semi-structured data

collection. It is used in all four methods of data

collection

Sometimes the only guide is the topic itself andh h ll b i ll

Questions in Qualitative Research



Sometimes the only guide is the topic itself andthe researcher collects verbatim or naturally

occurring data. This is known as unstructured data

collection. It is used in all four methods of data

collection

The sampling method of choice is theoretical

sampling (queuing behaviour)

However, often this is not possible and people

resort to convenience sampling (students) and

snowball sampling (mental health in black and

ethnic minority communities)

Sampling in Qualitative Research



Neither of the latter two methods are considered

strong but maybe all that can be achieved.

Research must be viable.

•Read and re-read data, become engrossed in it.

•Identify themes: common, conflicting, minority

•Test themes across the data set, where are they

common, under what circumstances are they found, notfound. This sets the parameters on the interpretation and

generalisation of data

•Get more than one person to analyse the data

independently then together

Data Analysis in Qualitative Research



independently then together

•Demonstrate trustworthiness in data analysis

•Examples

•Biographical continuity

•Nursing routines as a method of managing a transient

workforce

Qualitative research

Interpretative research

Process orientated Researcher(s) are the primary

data collection instrument

Descriptive research



Descriptive research Outputs are an inductive

process

References

MSc project web pageshttp://www.comp.glam.ac.uk/gis/start.asp?whatfile=gis/gi

src/msc-proj.htm

Creswell, J. W. (1994) Research

design : qualitative and



design : qualitative andquantitative approaches. -

Thousand Oaks, Calif.; London :

Sage Publications, ISBN0803952546

“Things won are done; joy's soul lies in the

doing.”

William Shakespeare



Workshop objectives

By the end of this session you will be ableto:

Understand why questionnaires are used and

when to use them



when to use them Understand the process of constructing a

questionnaire

Acknowledge the key features of goodquestion design

Questionnaire design in the context of

the survey process

Research aim and research questions

Identify the population and sample

Decide how to collect replies




Design your questionnaire

Run a pilot survey

Carry out main survey

Anal se the data



Questionnaire design in the context of

the survey process

Research aim and research questions

Identify the population and sample




Decide how to collect replies Design your questionnaire

Run a pilot survey

Carry out main survey Analyse the data

What is a questionnaire

A research tool for data collection

―It‘s function is measurement‖ (Oppenheim,

1992)

The term ‗questionnaire‘ used in different



The term questionnaire used in different

ways:

often refers to self-administered and postal

questionnaires (mail surveys) some authors also use the term to describe

Why would you use aquestionnaire?



Why use a questionnaire?

Target large amount of people

Use to describe, compare or explain

Can cover activities and behaviour,

knowledge attitudes preferences



knowledge, attitudes, preferences

Specific objectives, standardised and

highly structured questions

Used to collect quantitative data –

information that can be counted or

Strengths

and

limitations



Strengths

Can target large number of people Reach respondents in widely dispersed

locations

Can be relatively low cost in time and money



Can be relatively low cost in time and money Relatively easy to get information from people

quickly

Standardised questions Analysis can be straight-forward and

Limitations

Low response rate and consequent biasand confidence in results

Unsuitable for some people

e.g. poor literacy, visually impaired, young



e.g. poor literacy, visually impaired, young

children

Question wording can have major effect

on answers Misunderstandings cannot be corrected

Limitations

No opportunities to probe and developanswers

No control over the context and order

questions are answered



questions are answered

No check on incomplete responses

Seeks information only by asking, can we

trust what people say? e.g. issues withover-reporting

Maximising the response rate

If you were sending out a

questionnaire, what would you do to

maximise the response rate?



maximise the response rate?

In groups of 3 or 4, 5 minutes

Techniques for minimising non-

response

Good design

Thoughtful layout, easy to follow, simple

questions, appearance, length, degree of

interest and importance thank people for



interest and importance, thank people for

taking part

Pre-notification

Explanation of selection

Sponsorship, e.g. letter of introduction /

Techniques for minimising non-

response

Incentives

Small future incentives, e.g. prize draw

Understanding why their input is important



Understanding why their input is important

Reminders

Confidentiality

Anonymity

-

Clear specification



Political questionnaire

exercise

In groups of 3 or 4, spend 15 minutes

What research question(s) do you think the

questionnaire is trying to answer?

What are you reactions to:



y The question wording and structure?

The answer options?

Which are open questions and which are closed

questions? How could the questions be improved?



Question wording – things to avoid

Leading questions

Memory issues

Social desirability

Question complexity



Question complexity

Question wording – other things to

think about

Missing categories – include ‗other‘, ‗don‘t

know‘ and ‗not applicable‘

Sensitive questions



Sensitive questions

Simple language – not technical or slang

Question ordering

Open or closed questions?

Closed uestion – choice of alternative re lies

Open and closed questions(from Oppenheim, 1992)

Strength Limitation

Research Lecture July 31

Documents

Transcript of Research Lecture July 31