Survey Methods

Survey Methods

Nawanan Theera-Ampornpunt, M.D., Ph.D.

Faculty of Medicine Ramathibodi Hospital

July 1, 2015

www.SlideShare.net/Nawanan

2

2003 M.D. (First-Class Honors) (Ramathibodi)

2009 M.S. in Health Informatics (U of MN)

2011 Ph.D. in Health Informatics (U of MN)

2012 Certified HL7 CDA Specialist

• Deputy Executive Director for Informatics (CIO/CMIO)

Chakri Naruebodindra Medical Institute

• Lecturer, Department of Community Medicine

Faculty of Medicine Ramathibodi Hospital

Mahidol University

[email protected]

www.SlideShare.net/Nawanan

Introduction

3

Lecture

• Overview of Surveys

• Survey Methodology

• A Sample Survey

Group Exercise

• Sample Surveys & Critique

Outline

4

OVERVIEW OF SURVEYS

5

• An activity in which many people are

asked a question or a series of

questions in order to gather

information about what most people

do or think about something

Survey (การส ารวจ)

Merriam Webster Dictionary

6

• A written set of questions that are

given to people in order to collect

facts or opinions about something

Questionnaire (แบบสอบถาม)

Merriam Webster Dictionary

7

• To know something– Personal information

– Knowledge, Opinions & Attitudes

– Behaviors & Practice

– etc.

• About someone– Individuals

– Organizations

• In order to understand, create knowledge,

or make decisions

Why Do a Survey?

8

Examples of Surveys

Say2KFC.com

Service Satisfaction Survey (Online)

9

Examples of Surveys

http://www.kmutt.ac.th/building/pdf/application_form_2552_1.pdf

Service Satisfaction Survey (Paper-based)

10

Examples of Surveys

Image Source: http://pixabay.com/en/agent-business-call-center-18741/

Service Satisfaction Survey (Telephone)

11

SURVEY METHODOLOGY

12

• Survey Design (การออกแบบการส ารวจ)– Study Design (การออกแบบการศึกษา)

– Modes of Data Collection (วิธีการเก็บข้อมูล)

• Instrument Design (การออกแบบแบบส ารวจ)

• Sampling (การสุ่มตัวอย่าง)

• Survey Conduct (การด าเนินการส ารวจ)

• Data Analysis (การวิเคราะห์ข้อมูล)

• Reports (การรายงานผล)

Survey Methodology

13

WHAT WOULD YOU DO TO

INCREASE SURVEY

RESPONSES?

14

The Tailored Design Method

1978 1999 2007 2008

Dillman et al.

15

The Tailored Design Method

2014

Dillman et al.

16

• “The development of survey

procedures that create respondent

trust and perceptions of increased

rewards and reduced costs for being

a respondent, which take into account

features of the survey situation and

have as their goal the overall

reduction of survey error”

The Tailored Design

Dillman et al. (2007)

17

• Rewards

• Costs

• Trust

Social Exchange Theory


Image Source: http://horsebusinessschool.com/using-stick-and-carrot-to-motivate-employees/

18

• “The likelihood of responding to the

request to complete a self-

administered questionnaire, and

doing so accurately, is greater when

the respondent trusts that the

expected fix rewards of responding

will outweigh the anticipated costs”

Social Exchange


19

Providing Rewards• มีท่าทีเชิงบวก (Show positive

regard)• กล่าวขอบคุณ (Say thank you)• ขอค าแนะน า (Ask for advice)• สนับสนุนค่านิยม/คุณค่าของ

กลุ่มเป้าหมาย (Support group values)

• ให้รางวัลที่จับต้องได้ (Give tangible rewards)

• ท าให้แบบสอบถามน่าสนใจ น่าตอบ (Make questionnaire interesting)

• ส่งสัญญาณว่าสังคมสนับสนุนให้ตอบ (Give social validation)

• แจ้งว่าโอกาสตอบมีน้อย (Inform respondents that opportunities to respond are scarce)

Survey &



20

Reducing Social Costs• หลีกเลี่ยงค าที่ท าให้รู้สึกถูกสั่งการ

(Avoid subordinating language)

• หลีกเลี่ยงการสร้างความอับอาย (Avoid embarrassment)

• หลีกเลี่ยงความรู้สึกไม่สะดวก (Avoid inconvenience)

• ท าให้แบบสอบถามดูสั้นและตอบง่าย (Make questionnaires appear short & easy)

• ลดการสอบถามข้อมูลส่วนตัว (Minimize requests to obtain personal information)

• ท าให้การตอบ คล้ายกับการส ารวจที่เขาเคยตอบแล้ว (Keep requests similar to other requests to which a person has already responded)

Survey &



21

Establishing Trust

• ให้รางวัลล่วงหน้า (เช่น พร้อมกับแบบสอบถาม) (Provide a token of appreciation in advance)

• ได้รับการสนับสนุนจากหน่วยงานที่เกี่ยวข้อง (Sponsorship by legitimate authority)

• ท าให้เรื่องนั้นดูส าคัญ (Make the task appear important)

• เชื่อมโยงกับความสัมพันธ์อื่นที่มีการแลกเปลี่ยน (Invoke other exchange relationship)

Survey &



22

• Study Design– Cross-sectional vs. Longitudinal

– Descriptive vs. Analytical

Survey Design

23

• Mode of Data Collection– Self-administered survey

• Paper-based

• Online

• Telephone (Automated/IVR)

– Interviewer-administered survey

(aka. structured interview)

• In-person

• Telephone

– Mixed-mode survey

Survey Design

24

• เลือกใช้ค าง่ายๆ มากกว่าค าหรูๆ

• ใช้ค าให้น้อยที่สุดเท่าที่ท าได้ในการตั้งค าถาม

• ถามด้วยประโยคเต็มประโยค

• หลีกเลี่ยงค าเชิงปริมาณที่กว้างๆ เมื่อใช้ค าที่ “เป๊ะ” กว่าได้

• หลีกเลี่ยงความจ าเพาะ (specificity) ที่เกินความสามารถของผู้ตอบในการตอบได้ตรงความจริงอย่างรวดเร็ว (Avoid specificity that exceeds respondent’s potential for having an accurate, ready-made answer

• มีตัวเลือกด้านบวกและด้านลบจ านวนเท่ากัน

Instrument Design


25

Problem

• Number of years lived in Idaho

Years

• Your city or town

City or Town

• Your county

County

Use complete sentences


26

Revision

• How many years have you lived in Idaho?

Years

• In what city or town do you live?

City or Town

• In what Idaho county do you live?

Idaho County

Use complete sentences


27

Problem

• How often did you attend religious services

during the past year?

Never

Rarely

Occasionally

Regularly

Avoid vague quantifiers


28

Revision

• How often did you attend religious services

during the past year?

Not at all

A few times

About once a month

Two to three times a month

About once a week

More than once a week

Avoid vague quantifiers


29

Problem

• About how many books have you read for

leisure during the past year?

Number of books

Avoid too much

specificity


30

Problem

• About how many books have you read for

leisure during the past year?

less that 10

11-25

26-50

51-75

76 or more

Avoid too much

specificity


31

• แยก “ไม่แน่ใจ” (undecided) กับ “เฉยๆ” (neutral) ออกจากกัน โดยเอา undecided อยู่ท้ายสุดของ scale

• หลีกเลี่ยงอคติจากการเปรียบเทียบที่ไม่เท่ากัน (Avoid bias from unequal comparisons)

• ใช้ค าที่กล่าวถึงทัศนคติทั้งด้านบวกและลบในค าถาม (State both sides of attitude scales in the question stems)

• หลีกเลี่ยงค าถาม “check-all-that-apply” เพื่อลดปัญหาจาก primacy effects (เลือกตอบเฉพาะ choice ต้นๆ แล้วหยุดตอบ)

• ออกแบบค าตอบที่ไม่ทับซ้อนกัน (mutually exclusive)

Instrument Design (2)


32

Problem

• Which one of the following do you feel is most

responsible for recent outbreaks of violence in

American’s schools?

Irresponsible parents

School policies

Television programs

Avoid bias from

unequal comparisons


33

Revision

• Which one of the following do you feel is most

responsible for recent outbreaks of violence in

American’s schools?

The way children are raised by parents

School policies

Television programs

Avoid bias from

unequal comparisons


34

Problem

• From which one of these sources did you first

learn about the tornado in Derby?

Radio

Television

Someone at work

While at home

While traveling to work

Mutually Exclusive


35

• ใช้เทคนิคที่ช่วยการฟื้นความจ า (Use cognitive design techniques to improve recall)

• เลือกใช้จุดอ้างอิงเวลาที่เหมาะสม (Provide appropriate time referents)

• ตรวจสอบความถูกต้องทางทฤษฎีของค าถาม (Be sure each question is technically accurate)

• เลือก wording ของค าถามที่ช่วยให้เปรียบเทียบกับข้อมูลจากการส ารวจครั้งก่อนๆ ได้

• หลีกเลี่ยงการถามให้ตอบ “ใช”่ เพื่อให้หมายถึง “ไม่ใช”่

• หลีกเลี่ยงค าถาม double-barreled questions



36

Problem

• Should the city build a new swimming pool that

includes lanes for swimming laps that is not

enclosed for winter use?

Yes

No

Double Barreled


37

• พยายามลดผลกระทบของค าถามที่อาจมีประเด็นขัดแย้งหรือเห็นต่าง (Soften the impact of potentially objectionable questions)

• หลีกเลี่ยงการขอให้ผู้ตอบค านวณตัวเลขที่ไม่จ าเป็น (Avoid asking respondents to make unnecessary calculations)



38








Survey Methodology

39

• Target Population

• Sample

• Inclusion & Exclusion Criteria

• Sampling Frame

• Sampling Method/Technique

Sampling Design

40

• Census

• Random (probability) sampling

– Simple random sampling

– Systematic random sampling

– Stratified random sampling

– Cluster random sampling

– Multi-stage random sampling

• Non-probability sampling

– Purposive sampling

– Convenience sampling

– Snowball sampling

– etc.

Sampling Design

41

Example sequence

• Pre-notice Letter

• 1st Questionnaire Mailout

• Reminder + 2nd Questionnaire Mailout

Survey Conduct

Simplified from Dillman et al. (2007)

42

• Sampling error

• Coverage error

• Measurement error

• Nonresponse error

– Questionnaire effects

– Data collection mode effects

– Interviewer effects

– Respondent effects

• Processing error

Errors in Surveys

Dillman et al. (2007) & Office of Management and Budget (2001)

43

• Sampling error

• Coverage error

• Measurement error

• Nonresponse error

• Processing error

– Data entry error

– Pre-edit coding errors

– Editing errors

– Imputation errors

Errors in Surveys

Dillman et al. (2007) & Office of Management and Budget (2001)

44

A SAMPLE SURVEY

45

NAWANAN’S DOCTORAL

STUDY

46

Unknown State of IT Adoption in Thai Hospitals

Need to improve theoretical knowledge

This

Study

Dual Opportunities

47

• To describe current state of IT adoption in Thai

hospitals nationwide

• To test proposed conceptual framework & explore

relationships between organizational

characteristics, IT management, and IT adoption

Study Objectives

48

Hypothesized Model

49

• Study Design: Nationwide cross-sectional mail survey

• Sample: All hospitals in Thailand except pilot (N = 1,302)

• Pilot: 5 hospitals (10 respondents each)

• Sampling Frame: List of hospitals from Ministry of Public Health’s Web site

• Subjects: Hospital’s staff responsible for managing information systems (CIO/IT manager or equivalent; hospital director if N/A)

• Data Collection Period: 16 weeks

Design & Population

50

• Modified from original instrument

• Face & content validity established (Theera-Ampornpunt, 2009)

• Further modified based on pilot findings

• Translated to Thai

Section 1 Hospital Profile

Section 2 IT Adoption & Use Profile

Section 3 Respondent’s Information

English version

Survey Instrument

51

• Managerial: To what extent do you agree or disagree with each statement? e.g.,

• Those who will use the information systems are fully involved in hospital IT development

• Functional: How much is each activity supported by computerized information systems in your hospital?

• Technological: To what extent is each technology made available in your hospital?

• Integration: To what extent is information shared or transmitted among information systems within/outsideyour hospital?

Sample Questions

52

Week 0:

Prenotice Letter

Week 1:

1st Questionnaire

Mailout

(Dec. 18, 2010)

Week 6:

2nd

Questionnaire

Mailout

(Jan. 27, 2011)

Weeks 9-13:

Planned Follow-up Phone

Calls to Nonrespondents

and 3rd

Questionnaire

Mailout If Necessary

(Never Conducted

Because of Satisfactory

Response Rate)

Week 16:

Survey Returns

Closed

(Apr. 8, 2011)• 150-baht (~US$5) incentive if completed

• Endorsed by President of the Thai Medical Informatics Association

• Funded by a leading medical school with known informatics focus

• Anonymous unless contact information provided for incentive & results

mailing

Survey Methodology

(Nationwide)

53

• 64% response rate

• Some items problematic

– Differing within-hospital responses on total & IT

budgets,

No. of IT staff, quality accreditation status

– Poor interrater reliability for some dimensions

• Quality accreditation status dropped

• Item wording revised & survey shortened

• Integration sophistication items restructured

Pilot Study Findings

54

• 4 of 1,302 hospitals ineligible

• Response rate 69.9%

Characteristic Overall Responding

Hospitals

Non-

Responding

Hospitals

N of eligible hospitals 1,298 908 390

Bed size** 106.9 117.5 82.9

Public status**

Private

Public

24.0%

76.0%

17.4%

82.6%

39.2%

60.8%

Geography*

Central

East

North

Northeast

South

West

33.4%

7.5%

11.1%

27.1%

15.3%

5.6%

31.1%

7.8%

13.5%

26.9%

14.9%

5.8%

39.0%

6.7%

5.4%

27.7%

16.2%

5.1%

*p < 0.01, **p < 0.001.

Nationwide Findings

55

Characteristic Number of Responses Statistic†

Public status

Private

Public

908

158

750

17.4%

82.6%

Teaching status

Non-teaching

Teaching

901

716

185

79.5%

20.5%

Total employees 890 368.2 ± 573.5 (10-5269)

IT employees 901 4.3 ± 5.3 (0-60)

Total budget (million baht) 443 146.67 ± 313.60 (0.25-3,067)

IT budget (million baht) 598 2.77 ± 8.79 (0-100)

Ratio of IT budget to total budget‡

< 1%

1-4%

5-8%

> 8%

416

135

218

40

23

2.7% ± 4.6% (0-43.3%)

32.5%

52.4%

9.6%

5.5%

Extent of overall IT utilization

Very low

Low

Moderate

High

Very high

905

5

35

169

454

242

0.6%

3.9%

18.7%

50.2%

26.7%

Total PCs in use 883 126.1 ± 218.6 (0-3,000)

Nationwide Findings

56

Estimate (Partial or Complete Adoption) Nationwide

Basic EHR, outpatient 86.6%

Basic EHR, inpatient 50.4%

Basic EHR, both settings 49.8%

Comprehensive EHR, outpatient 10.6%

Comprehensive EHR, inpatient 5.7%

Comprehensive EHR, both settings 5.3%

order entry of medications, outpatient 96.5%

order entry of medications, inpatient 91.4%

order entry of medications, both settings 90.2%

order entry of all orders, outpatient 88.6%

order entry of all orders, inpatient 81.7%

order entry of all orders, both settings 79.4%

Adoption Estimates

57

• Overview of Surveys

• Survey Methodology

• A Sample Survey

Recap

58








Survey Methodology

Survey Methods

Education

Transcript of Survey Methods