2010 Research Methods Handbook

8/3/2019 2010 Research Methods Handbook

1/57

Copyright 2010 by the President and Fellows of Harvard College. All Rights Reserved.

RESEARCH METHODS

HANDBOOK

HARVARD LAW SCHOOL

FALL 2010

Office of Clinical and Pro-Bono Programs Negotiation & Mediation Clinical ProgramAustin Hall, Rooms 102 & 108 Pound Hall, Suite 513

T: 617.495.5202 T: 617.496.7109

F: 617.496.2636 F: 617.495.7818www.law.harvard.edu/academic/clinical www.law.harvard.edu/negotiation


2/57


TABLE OF CONTENTS

INTRODUCTION ..........................................................................................................1

A. What should you take away from this material?........................................................................................1

B. How is this guide organized?........................................................................................................................2

CRASH COURSE IN STATISTICS .................................................................................3

A. What is Statistics? .........................................................................................................................................3

B. Error...............................................................................................................................................................4

Measurement Error................................................................................................................................................................ 4Sampling Error.......................................................................................................................................................................4

C. Sampling.........................................................................................................................................................5

When to Sample......................................................................................................................................................................5How to Sample ....................................................................................................................................................................... 5Probability Samples ............................................................................................................................................................... 6Random Numbers Table ......................................................................................................................................................... 6Systematic Sampling...............................................................................................................................................................7Cluster Sampling .................................................................................................................................................................... 7Stratified Sampling................................................................................................................................................................. 7

D. Sample Size ....................................................................................................................................................8

E. Descriptive Summary Methods ....................................................................................................................9

Measures of Central Tendency............................................................................................................................................... 9Measures of Dispersion........................................................................................................................................................10

F. Wait a Moment Why am I Reading About Statistics ??? .......... .......... ........... ........... .......... ........... ....10

G. Probability ...................................................................................................................................................11

H. Normal Distribution....................................................................................................................................11

I. Sampling Distributions ...............................................................................................................................12J. Confidence Intervals ...................................................................................................................................12

K. Hypothesis Testing ......................................................................................................................................13

L. Regression ....................................................................................................................................................13

M. References....................................................................................................................................................13

SURVEYS.................................................................................................................. 15

A. Getting Started ............................................................................................................................................15

B. Types of Surveys..........................................................................................................................................15

Mail Surveys......................................................................................................................................................................... 16Telephone Surveys................................................................................................................................................................ 17

Face-to-Face Surveys........................................................................................................................................................... 18 Internet Surveys.................................................................................................................................................................... 19 Hybrid/Mixed Mode Surveys................................................................................................................................................20

C. Survey Errors ..............................................................................................................................................21

Coverage Error .................................................................................................................................................................... 21Sampling Error.....................................................................................................................................................................21Measurement Error.............................................................................................................................................................. 21

Nonresponse Error...............................................................................................................................................................21

D. Sampling.......................................................................................................................................................22

E. Question writing..........................................................................................................................................23


3/57


Turning Ideas into Questions ...............................................................................................................................................23Attitudes & Beliefs vs. Behavior & Attributes ......................................................................................................................23

F. Question Structure ......................................................................................................................................24

Open-ended Questions ......................................................................................................................................................... 24Close-ended Questions......................................................................................................................................................... 25Some Final Tips.................................................................................................................................................................... 26

G. Questionnaire Design ..................................................................................................................................28

The First Question................................................................................................................................................................ 28

Question Order.....................................................................................................................................................................28Question Clusters ................................................................................................................................................................. 29Narrative ..............................................................................................................................................................................29Oral vs. Written Surveys....................................................................................................................................................... 29

H. Analyzing Results ........................................................................................................................................30

Cleaning and Coding ........................................................................................................................................................... 30Analysis ................................................................................................................................................................................30Interpretation .......................................................................................................................................................................31

I. Ethics............................................................................................................................................................31

J. References....................................................................................................................................................31

INTERVIEWS ............................................................................................................ 33

A. General Guidelines......................................................................................................................................33Introduction..........................................................................................................................................................................33Questions.............................................................................................................................................................................. 33Closing .................................................................................................................................................................................34

B. Recommended Questions............................................................................................................................34

General Questions................................................................................................................................................................34Ongoing Conflict .................................................................................................................................................................. 35

No Specified Conflict............................................................................................................................................................35

C. Positions to Interests ...................................................................................................................................36

FOCUS GROUPS ....................................................................................................... 38

A. Uses of Focus Groups..................................................................................................................................38B. Combining Focus Groups with Other Research Methods.......................................................................39

C. Strengths and Limitations of Focus Groups .............................................................................................40

D. Planning for the Focus Group....................................................................................................................41

Choosing Participants.......................................................................................................................................................... 41Number of Groups................................................................................................................................................................41 Heterogeneous vs. Homogeneous Groups ............................................................................................................................ 42Group Size............................................................................................................................................................................ 42

E. Preparing the Focus Group........................................................................................................................43

Recruiting Participants ........................................................................................................................................................ 43 Choosing the Location ......................................................................................................................................................... 43

Interview Guide.................................................................................................................................................................... 44

F. A Word About Group Dynamics ...............................................................................................................44G. Conducting the Focus Group .....................................................................................................................45

The Moderator ..................................................................................................................................................................... 45How involved should the moderator be? .............................................................................................................................. 45Getting Started ..................................................................................................................................................................... 47Guiding the Discussion ........................................................................................................................................................ 47Making Sure Everyone Participates ..................................................................................................................................... 47

Probing ................................................................................................................................................................................47 Managing Time .................................................................................................................................................................... 48

Dealing with Common Problems..........................................................................................................................................48

H. Analysis ........................................................................................................................................................49


4/57


Qualitative Analysis ............................................................................................................................................................. 49Content Analysis................................................................................................................................................................... 49

I. Ethics............................................................................................................................................................50

J. References....................................................................................................................................................50

BIBLIOGRAPHY ....................................................................................................... 51

GLOSSARY............................................................................................................... 52


5/57

HNMCPRESEARCH METHODS HANDBOOK 1


INTRODUCTION

Welcome to the Negotiation and Mediation Clinic! By now you should know what project youll beworking on this semester, and youre probably thinking about how you should get started. But asyou crack open this handbook, you might be wondering why we want you to learn about researchmethods. Well, here are some examples of how research methods can help you execute yourproject and produce solid results:

A military college recently became co-ed, but is having trouble retaining its female students.Youd like to learn about the female experience at the college, but you dont even know whatkinds of questions to ask. Using a focus group to bring together female students for adiscussion could help you figure out why theyre leaving the school.

The Catholic Church has found an increasing number of disputes arising among clergymembers, and wants to develop a dispute resolution system to help manage them.Administering a survey to clergy members might help you figure out what the currentsystem looks like, and how people within that system would like to see it changed.

A large international NGO wants to streamline its decision-making procedures, which arecurrently ad hoc and decentralized. Conducting interviews of key players within the NGOmight help you understand what the current procedures are and which changes are likely tobe feasible.

The purpose of this handbook is to give you a brief overview of major research methods as theyapply to projects for this clinic. Its not designed to function as an in-depth explanation of thesemethods, but as an introduction to get you started on the path to reliable research that you canpresent to your client. Remember, these are real clients with real problems that need real

solutions, so we expect you to be responsible in your research and analysis, which might meanconsulting additional resources. So with that said, happy researching!

A.What should you take away from this material?Our intent in giving you this manual was to give you a very generalized understanding of best-practice research methodology. To be clear: In the ideal world, the clinics research would itself bein line with those best practices, and our findings statistically impeccable. That said, we of courserealize that for most of our projects, PhD-level statistical accuracy is difficult if not impossible toachieve given our clinics resources and given the time you have to complete a project.1 So fear not

we are not expecting your projects to demonstrate gold standard research methodology!

1This is of course leaving aside entirely the question of whether perfect research

design is even necessary given the analytical and think tank nature of most of our

clinical projects.


6/57

2 HNMCPRESEARCH METHODS HANDBOOK


After reading this manual, you hopefully should be able to do three things:

1. Strategize about your own research methodology: You and your teammates should beable to strategize about how to maximize the rigor of your projects research, taking intoconsideration your groups limited capacity, time and resources available to carry out andanalyze research.

2. Identify your projects research shortcomings: After reading this guide, you should beable to identify ways in which your projects research methodology deviates from the goldstandard to which you would be heldif you were a PhD candidate or professionalstatistician. In other words if a statistician or an econometrician were to carefullyscrutinize your projects conclusions and research methods, what would she look for whenjudging whether or not our conclusions were accurate?

3. Speculate about the impact of your research methodologys shortcomings: Finally, youshould be able to speculate about the impact of your justifiable research methodologyshortcomings. If you collected data in a certain way and suspect that there might be anunavoidable error of some sort, how might your conclusions be different had you somehow

been able to correct for that error?

Most final projects will include a methodology section of some sort, in which you shouldspecifically put into writing your thinking about the above three points.

B.How is this guide organized?Given our purposes for subjecting you to this material, we structured the guide with the overviewof statistics preceding the description of survey and focus group methodology. Our thinking wasthat a presentation of the GOAL of an activity should generally precede the description of HOW to

achieve that goal.

So, if you are the kind of person whose eyes glaze over when reading about statistics and researchdesign, keep in mind that the first section covering statistics is the aspiration, which will hopefullymake it clear why the methods we are prescribing will help you achieve good research outcomes.


7/57



CRASH COURSE IN STATISTICS

Afraid of statistics? Breathe a sigh of relief. We dont expect you to be - or become, for that matter -a statistics expert for this course. But since the research you do in this clinic is for real clients whohave real stakes in your work, we want you to be able to identify how dependable your research is,where inaccuracies may lie, and how further analysis (including math!) could strengthen yourwork. This section will introduce the basics of statistics, but it is by no means designed to replace arigorous statistics class. And dont worry - even though well introduce some key statisticalconcepts and methods, our goal is not to have you do complex regression analyses or hypothesistests after reading the next few pages. Instead, we want to familiarize you with considerations youshould take into account when sampling, analyzing, and reporting data. Weve listed a couple ofadditional resources at the end of this section, but also we recommend that you consult otherresources, like Harvards economics department, if you feel that you need more guidance.

A.What is Statistics?So what is statistics? Well, the basic idea is that its a way to learn about a population thats too bigto study in its entirety say, all doctors in the United States or all lawyers in China by looking at asample of that population. But first things first: before we go on, youll need a few key definitionsto get you speaking the language of statistics:

Population - a population includes all of the individuals in a group you want to study.Individuals dont have to be people; they could be organizations, laws, or even rocks (ifyoure a geologist)!

Sample - a sample is any subset of a population.

Parameter- a parameter is a characteristic of a population. For example, if your populationwere lawyers in China, one parameter would be the percentage that is environmentallawyers.

Statistic- a statistic is a guess of a parameter

So, using these definitions, the general idea of statistics is that when you have a population thatstoo big to examine as a whole, you cant properly analyze its parameters (or characteristics). To getinformation about the parameters youre interested in, you can look at the characteristics of a moremanageable and representative portion, or sample, of the population. Then you can use statistics tomake educated guesses about the populations parameters based on that sample. In other words,

statistics are educated guesses, based on samples, about population parameters.


8/57



B.ErrorAll research methods have room for error and bias, and well discuss these more in the followingsections. When doing statistics, though, there are two main errors to be aware of: measurement

error and sampling error.

Measurement Error

Measurement error relates to the quality of the data you collect, and is a concern in any researchmethod. With regard to statistics, the idea is that if you collect data from a sample in a way thatproduces biased or unclear results, it simply cant represent the population accurately. Forexample, lets say we asked a sample of Harvard students Do you agree that the dining hall isfantastic? A question like this primes respondents to think about the dining hall more positivelythan they otherwise would, so it would probably produce more yes answers than wed get if weasked the question more neutrally. This, in turn, would cause us to report, based on our sample,

that most Harvard students find the dining hall fantastic which probably isnt true. This meansthat no matter what research method you use, you need to make sure your data is as reliable andclear as possible; otherwise, you wont be able to use it to accurately judge the larger population.

Sampling Error

When you do statistics, the very fact that youre examining only a sample of a population meansthat youll have some degree of inaccuracy in your research this is sampling error. For example,lets say youre trying to estimate how many of Chinas lawyers specialize in environmental law bylooking at graduates of a few major law schools in China. If these law schools represent thepopulation of Chinese lawyers well, they should give you a good idea of what percentage of thepopulation specializes in environmental law. Even so, there is simply no way that the sample cangive you a precise number. So no matter what you do, you can never avoid sampling error entirely.

What you can do, however, is minimize sampling error by increasing your sample size. The otherthing you can do is report it, which most professional researchers do. Youve probably noticed, forexample, that when polls results appear in places like newspapers, theyre usually reported asaccurate to plus or minus a few percentage points; those percentage points are the sampling errorof the poll. While its fortunate that sampling error can be calculated, we dont expect you tocalculate it for this class. We do, however, want you to be aware that its there and that its affectingthe accuracy of your results.


9/57



C.SamplingAs youve probably noticed by now, samples are the basis of statistics. Since the point of a sample(at least for statistics) is to generalize its results to an entire population, taking a good sample is

fundamental to producing good statistics. In this section, well discuss how to sample and welldescribe a few of the most common sampling methods.

When to Sample

In almost all cases, you will have to sample. Only when you are analyzing a target group smallenough so that you can accurately contact and question every single memberof that population, orwhere your resources are so large that you can again contact and question every single memberof alarger population, can you avoid sampling. In this case, you would be conducting a census.

Think of the well known US Census, which is carried out every ten years. Estimates project that the

2010 census, which is intended by the Constitution only to enumerate the American population,will cost the US taxpayer $14.5 billion. Imagine the simple task of counting the number of peoplein the United States costs over 1600 times as much as the entire cost of constructing HLSNorthwest corner! Unless your project is dealing with a small target population, or unless thesubject matter of your clinical project is REALLY important, it is highly unlikely that HNMCP couldever muster sufficient resources to allow you to conduct a census.

How to Sample

There are three basic steps to sampling:

1. Identify the target population the first thing you need to do is decide who your targetpopulation is. In other words, whom are you studying? Is it the Chinese environmental lawyers?Students at HLS? Its helpful to have as precise a definition as possible.

2. Compile a sample frame a sample frame is a list of population members that youll use to drawyour sample, like a phone book or a staff directory. Ideally, your sample frame should be complete,up to date and composed only of population members of interest. It should also list each populationmember only once. These characteristics will protect your research against errors, like choosingthe same population member twice or accidentally making it impossible to select some populationmembers.

3. Select the sample all samples fall into two broad categories: probability samples and non-

probability samples. Probability samples are ones chosen randomly, which allows them to begeneralized to larger populations. Non-probability samples are ones whose members are chosenfor specific reasons, like convenience or role in an organization. Because theyre chosensubjectively, non-probability sample results cant be generalized to the broader population, so wewont focus on them here. Instead, well review characteristics and methods of probabilitysampling.


10/57



Probability Samples

A probability sample, sometimes called a random sample, is one where each member of the targetpopulation has an equal or known possibility of being selected for it. The randomness of this kindof sample, together with the laws of probability, make it likely that it will represent the population;this is what allows the researcher to generalize its results to the larger population with a calculated

estimate of error.

Its important to stress, however, that just because a particular sample is chosen randomly doesntmean that its necessarily representative of the population. In fact, its quite possible for probabilitysampling to produce strange samples. For example, lets say youre studying Chineseenvironmental lawyers, and youre interested in where theyre from. Although the population as awhole may come disproportionately from Beijing, its entirely possible, although improbable, thatyou could draw a random sample that contains only lawyers from Shanghai, Guangzhou, and HongKong. The value of random sampling lies in probability at a broader level - its highly improbablethat if you took ten or twenty more random samples of the Beijing-dominated population youwould still end up with samples dominated by lawyers from other cities.

There are several ways to take a probability sample, the simplest being drawing sample membersfrom a hat. Well describe a few other common methods below:

Random Numbers Table

Random numbers tables are usually computer generated, although you can also find them instatistics textbooks and even online. They consist, as you might have guessed, of numbersgenerated randomly and ordered in a table format, as below:

39634 35050 38449 61303 15140

14595 95113 46574 46699 52925

61885 00020 86558 87474 03395

To illustrate this method, lets say you want a sample of 50 out of a population of 5000. The firstthing you need to do is number the entire population from 1 to 5000. Then, start at the top of therandom numbers table and work your way down the columns systematically, selecting anypopulation member whose number appears until you have chosen 50 sample members. Using theabove table, you would look at the first 4 digits of each number, since 5000 has 4 digits, choosingpopulation member 3963, 1459, 3505, 0002, and so on.


11/57



Systematic Sampling

This method relies on the proportion of sample members to population members, using theproportion as an interval to select sample members. For example, if youre choosing a sample of 50out of a population of 5000, youll be choosing 1 in 100 population members. To use this method,number all of the population members and then choose a starting point randomly. Then go down

your population list choosing every 100th population member until you have 50 of them. Beforeyou use this method, make sure that the population list doesnt break down into intervals thatmight bias your sample. For example, if youre dealing with days of the week and you choose every7th population member, youll get the same day every time.

Cluster Sampling

This method breaks the population down into clusters most often geographic ones and thensamples each cluster randomly. Its helpful for when a good, comprehensive list frame cant beeasily compiled, since it allows researchers to construct lists only for clusters being sampled. Toillustrate this method, lets say youre researching evictions by the Boston Housing Authority. You

would break the city into clusters, like by neighborhood or voting ward, then randomly choosewhich clusters to sample, and then take random samples of the clusters chosen. Its important topoint out that this method is problematic when each cluster is homogeneous within itself, sinceeach one wont accurately represent the population as a whole.

Stratified Sampling

Researchers using this method divide the population into subgroups (or strata), and then takerandom samples of each one. This technique is helpful for when youre looking at a population aswell as some of its subgroups, like if youre studying Chinese lawyers in general, but also want tolearn about Chinese environmental lawyers in particular. When using this method, youll have to be

aware of what percentage of the overall population each subgroup represents, since sometimesyoull want samples to reflect their true representation in the population, but other times youll bemore concerned with having samples of equal size. As a result, this technique can have complicatedimplications for both generalizing back to the overall population and for calculating sampling error.


12/57



D.Sample SizeWhen youre getting ready to sample, one important thing to think about is how big your sampleshould be. Sample size depends on a few factors:

Accuracy generally speaking, increasing sample size reduces sampling error, meaningthat bigger samples are usually more accurate. You should consider your need for precisionwhen deciding how big your sample should be.

Variation populations that have a lot of variation in the characteristics youre studyingusually require bigger samples. The idea here is that you want to be confident that anysample you choose would produce roughly the same results, which is unlikely to happen ifyoure taking small samples from a heterogeneous population.

Subgroups - If youre looking at subgroups within a population, youll have to divide yoursample into corresponding subgroups. In effect, this creates smaller samples of eachsubgroup, which increases the sampling error for each one. So keeping accuracy intact forsubgroups means using a larger overall sample.

Resources Finally, dont forget that youll probably have limited resources, especiallytime. Bigger samples require more resources, so youll have to find a balance betweenprecision and efficiency.

This is just a quick summary of some important and complicated considerations. Wed recommendlooking at the resources listed at the end of this section for more guidance.


13/57



E.Descriptive Summary MethodsOnce youve taken your sample, youll need to analyze it to figure out what it says about thepopulation. Statistics uses a variety of measurement techniques, but two of the most fundamental

ones are measures of central tendency and measures of dispersion. Well focus on these two typesin this section, but we encourage you to do some outside research to learn about other methods.

Measures of Central Tendency

Measures of central tendency describe the middle or majority of the population. Theyre useful forstatistics because we have a tendency to ascribe to an entire population what its average or middlegroup does. Just think, for instance, about how often you use an average to tell you whats normalfor a population - like batting averages for baseball players or LSAT scores for HLS students. Thereare three main measures of central tendency:

Mean : the mean (or average) is calculated by adding up all of the values in a group andthen dividing that sum by the number of group members. Believe it or not, thisis one of the most powerful tools in statistics! For this clinic, though, youllmostly look at the mean of a sample and use it to describe the correspondingpopulation. The main problem with the mean is that outliers (numbers that areextreme or dont otherwise fit with the majority of the data set) throw off itscalculation.

Median : unlike the mean, the median isnt a calculated value. Its simply the number inthe middle of a data set, so you can find it by putting all data set numbers inascending or descending order, and then picking the middle one. If you have aneven amount of data set members, calculate the median by averaging the middletwo. Although the median is not a particularly strong statistical tool, itsadvantage is that outliers do not affect it, so its a more stable measure of centraltendency.

Mode : the mode is the number that that occurs most commonly in a data set. Like themedian, its not a calculated number and isnt a particularly strong statisticaltool, but its a more stable measure of central tendency.


14/57



Measures of Dispersion

Dispersion looks at the range of data within a data set, so its most useful when you want tounderstand the variability in your data. For example, we would probably want to use some kind ofdispersion to understand what different types of law Chinese lawyers are specializing in and in

what numbers. There are several ways of measuring dispersion; well discuss a few of them below:

Range : the range is simply the difference between the highest and lowest numbers in a dataset. Since its not a calculated number, and since it depends only on the top andbottom extremes of a data set, its not a particularly strong statistical tool, but itdoes give you a sense of how much variation there is in your data.

Distributions : distributions are more helpful measures of dispersion, and are relativelysimple to calculate. First, divide your data set into appropriate categories, and thencount how many members of the data set fit into each one. So, for example, youmight divide the Chinese lawyers into categories like environmental lawyers,business lawyers, immigration lawyers, etc. Then youd count up how many lawyers

fit into each category to get your distribution. Its often most helpful to turn thistype of distribution into apercent distribution, which tells you what percent of thepopulation each category represents and thus provides a more readily comparablemeasure. Calculate the percentages by dividing the number of category members bythe total number of data set members.

Standard Deviation : standard deviation is the average difference between an individualdata set member and the data sets mean. In other words, it tells you how values ina data set spread around the mean. This is one of the most common measures ofdispersions, and is an extremely powerful statistical tool. Calculating standarddeviation is fairly tedious, and since we dont expect you to do it for this course wewont go into an explanation here. The resources listed at the end of the chapter

have both the formula for calculating standard deviation and more informationabout its uses.

F.Wait a Moment Why am I Reading About Statistics ???Lets pause for a minute to sum up where we are so far: measures of central tendency anddispersion are some of the most useful and user-friendly ways of analyzing any data that youll bedealing with in this clinic. Combined with a good sample, these tools will help you present datathats fairly reliable and easy to understand to your clients. Youll also be able to recognize that, byvirtue of using a sample, the accuracy of your research is limited.

What we havent explained is why statistics works, or some of the methods statisticians use toexplain more complicated issues like how variables affect each other how does being Chineseaffect the likelihood that a lawyer will do environmental work? What follows is a brief summary ofsome of these topics, with the goal of familiarizing you with concepts of statistics and helping yousee where more rigorous research and calculations could significantly hone your results.


15/57



G.ProbabilityWhen we look at a random sample and then translate its results to a population, were relying onprobability, defined as the relative frequency with which something is likely to happen. Toillustrate the basis for this idea, think of flipping a coin. Theres an equal probability of getting

heads or tails, or a probability of 0.5. But if you flip the coin twice, you wont necessarily get headsonce and tails once. Instead, to really see heads and tails an equal number of times, youd probablyhave to flip the coin hundreds of times. This is the idea behind random sampling: if you took awhole lot of random samples, on average theyd represent the population accurately. Thisfavorable probability allows you to make judgments about the population based on a single sample,even though you can never really know if thatparticularsample represents the populationaccurately.

H.NormalDistribution

The normal distribution isoften called a bell curve its a distribution of datathat looks like a bell, wherethe mean, median, andmode are all equal. Thearea underneath the curverepresents probability.

Normal distributions arefundamental to statisticsbecause they relatemeasures of centraltendency and dispersion in a key way: the area underneath the curve that lies between onestandard deviation in each direction from the mean encompasses 68% of all values in the data set.In other words, there is a 68% probability that any value in a normal distribution will fall withinone standard deviation of the mean. Similarly, the area between two standard deviations

represents 95% of all values, and the area betweenthree standard deviations represents 99% of all values.

Statisticians calculate the numerical value of thestandard deviation, and use it to indicate their certaintyin claiming that the findings of their representativesample is either 68%, 95%, or 99% certain to be within

x of the findings had we surveyed the entire population (see confidence intervals below).Calculating the numerical value of the standard deviation requires calculations which can be foundin any statistics book, but is most likely beyond the scope of a typical HNMCP clinical project.

Example: Imagine you were given the (incredibly boring) task of flippinga quarter 100 times, and counting up how many times you got heads andtails respectively.

Q: how many times would you expect to get heads?

The reason you (hopefully) answered both of the above questions with50, is that we all know that chances of getting heads in a coin toss shouldbe 50/50.

If your clinical supervisor asked you to verify this assumption, and youhad to conduct the 100 coin-flip exercise 100 times (using a fair coin) youwould ultimately be able to chart your results as follows:

y-axis: number of times a 100-coin-flip exerciseresulted in a result of x number ofheads tosses.

x-axis: number of times you got heads as aresult

50

50

Onestandarddeviation


16/57



I. Sampling DistributionsImagine you took a sample from a population, and then calculated its mean. Then imagine you tooka new sample, and calculated that ones mean. Then imagine that you repeated this over and overagain, until you decided to stop and graph all of the sample means you had calculated. What youd

have would be a sampling distribution. Even though youd never do this in practice, samplingdistributions are fundamental to statistics because they have a key property, which comes from theCentral Limit Theorem: as long as youre working with a population with at least 600 members andsamples with at least 30 members, then the sample means will be normally distributed around amean that, on average, is equal to the population mean. This property of sampling distributionsallows statisticians to do a lot of things, including making guesses about populations that arentnormally distributed based on the characteristics of normal distributions.

Thanks to the tireless work of statisticians who tediously proved the truth of the Central LimitTheorem, we can confidently present the findings we calculate from our sample mean, and estimatehow likely it is that the true population figures are within x of our findings.

J. ConfidenceIntervals

Even though samplingdistributions let statisticiansmake strong assumptionsabout how a sample relates tothe population, they still cannever be sure that thoseassumptions are correct.Confidence intervals are a wayto manage but not eliminate -this uncertainty. For example,we cant be absolutely sure thata sample mean is the same asthe population mean. We do,however, know that in a normaldistribution, 95% of all valuesfall within two standarddeviations from the mean, and that sample means are normally distributed around the truepopulation mean. So we can construct an interval that is two standard deviations in eitherdirection from the sample mean, and know that 95% of the time, or 95 times out of 100, thepopulation mean will fall within that interval. Its important to highlight, however, that we stillhave no way of knowing if this particular samples confidence interval contains the populationmean. That is, any particular sample could be one ofthe 5 times out of 100 that the population mean is notin the confidence interval.

42%(of U.S. electorate support John McCain)

(Aug. 25-27, 2008)

Twostandarddeviations

Example: Political Polling. Gallup is one of the most well-knowninternational polling consulting firms.

During the 2008 election for U.S. president, Gallup conducted dailysurveys of 1000 or more registered U.S. voters. Using the methodsdescribed above, Gallup statisticians have estimated that this samplesize will give them 95% accuracy within a +/- 3% confidence interval. Ifinstead of 1000 voters Gallup interviews 2000 registered voters for aparticular poll, their 95% accuracy confidence interval shrinks to +/- 2%.

Excerpt from August 28, 2008 poll tracking popularity of John McCainand Obama among registered US voters:

Survey MethodsFor the Gallup Poll Daily tracking survey, Gallup is interviewing no fewerthan 1,000 U.S. adults nationwide each day during 2008.

The general-election results are based on combined data from Aug. 25-27, 2008. For results based on this sample of 2,723 registered voters,the maximum margin of sampling error is 2 percentage points

(http://www.gallup.com/poll/109897/Gallup-Daily-Obama-Moves-Ahead-48-42.aspx)


17/57



K.Hypothesis TestingHypothesis testing is a way of using a sample distribution to test whether a hypothesis is likely tobe correct, based on probability. Typically, researchers first create a null hypothesis to be eitheraccepted or rejected, and then an alternative hypothesis, which states the opposite. For example, a

null hypothesis might be that the population mean equals 50. The alternative hypothesis, then,would be that the population mean does not equal 50. Researchers then use their sample data todetermine how probable it is that the null hypothesis is true; if it appears highly improbable that itis correct, researchers reject the null hypothesis, and if it appears probable they fail to reject thenull hypothesis. Of course, since hypothesis testing is based on probability, and we can never besure how well the sample reflects the population, it is entirely possible to reject a true nullhypothesis (a type I error) or to fail to reject a false null hypothesis (a type II error).

L.RegressionRegression analysis is one of the more complex and useful topics in statistics. Essentially, it is away to use statistics to explain reasons for variance in data. For example, if youre studyingprofessors salaries and you notice that they vary a lot, you might use regression to understand howvariables like gender, area of expertise, and number of articles published affect those salaries. Thegeneral idea of regression is that if you plot your data on a graph, usually with the independentvariable on the x-axis and the dependent variable on the y-axis on a graph, you can draw a line(usually using a computer) through the data that is as close to as many points as possible. This isthe regression line, and it gives you information about correlation, or how the two variables movetogether. It cant, however, tell you about causation; thats for the researcher to hypothesize.

M.References

1. Druckman, D. (2005). Doing Research: Methods of Inquiry for Conflict Analysis. ThousandOaks, CA: Sage.

2. Newport, F., Saad, L. and Moore D. (1997) How are Polls Conducted? in Where AmericaStands 2d ed.. John Wiley & Sons, Inc.(available at Gallup.com website - http://media.gallup.com/PDF/FAQ/HowArePolls.pdf)

3. Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. London: New York:Wiley.

4. Wright, S. (1979). Quantitative Methods and Statistics. London: Beverly Hills: Sage.


18/57


19/57



SURVEYS

Surveys are everywhere these days! You cant stay in a hotel, take a class, or even use the internetwithout someone asking you to complete a survey. When internet pop-ups ask you to complete asurvey about the website youre using, or when the U.S. Census Bureau sends you a profile to fillout, its because youre part of a population from whom these organizations want information.Their surveys are a systematic way of getting that information. The ubiquity of surveys, however,doesnt mean that theyre easy to administer. In fact, there is room for error at every step of theway, from choosing whom to survey to putting the survey questions in order. This section serves asan introduction to basic methods of designing, administering, and analyzing surveys. It is notintended as a comprehensive guide, however you should absolutely consult your professorsthroughout the survey process, and you may also need to consult the resources listed at the end ofthis section for further guidance.

A.Getting StartedThe first thing you need to decide before doing a survey is what, specifically, you need to learn.Once you know what information youre looking for, you can begin designing the survey. Forexample, lets say youre helping a legal aid clinic find out why so many of their clients are beingevicted by the local housing authority. One thing youll needto find out are the most commonreasons for evictions are most people evicted for non-payment of rent? Noise complaints?Something else? Keep in mind that although you mightlike to know other information like theeducation level of most of those evicted; you may notneedthis information to assist the legal clinic.If youre not careful about limiting the amount of it would be nice to know information you seek,your survey will quickly become unwieldy. Once you have a sense of what you need to know, youcan figure out who to ask and how to ask them.

B.Types of SurveysIn deciding what type of survey to use, youll need to consider the information you need, thecharacteristics of the population youre targeting, and your own resources (especially your time,given the length of academic semesters). For example, if youre pressed for time and want yoursurvey results quickly, a phone survey might be the best option for you. On the other hand, ifyouve decided to target individuals who have been evicted from their homes, they are unlikely tohave telephones and addresses; a face-to-face survey would be a better choice in that case. Below isa brief overview of survey methods, along with their major advantages and disadvantages:


20/57



Mail Surveys

As you might imagine, researchers using this method send the survey to a target population orsample by mail, and then respondents mail the completed survey back to the researcher.Researchers typically get addresses from a list frame like a list of utility company customers or a list

of employees from a human resources department, taking care to ensure that the list is as completeand accurate as possible. To improve response rates, researchers should send the survey with acover letter that explains the purpose and importance of the survey and a pre-addressed andstamped envelope. It can also help to send an introductory letter before the survey and a reminderletter (or phone call) after the survey has been sent.

Mail surveys are best for when you have limited resources, when you have an accurate address list,when a rapid response time is not required, and when the respondents have the necessary skills torespond accurately in writing. Their strengths and weaknesses are summarized in the chart below:

Mail Survey Strengths Mail Survey Weaknesses

Low cost, infrastructure and staff needs Long data collection time

Good control over howquestions are asked

List frames oftencontain inaccuracies

High accessibility to most populations

(most people have addresses)

High non-response rates

(think: junk mail!)

Gives respondentsa sense of privacy

Little control over who responds and inwhat order they respond to questions

Enables the use of graphicsand visual aids

Difficult to use complex and open-endedquestions, since researcher is not presentto provide explanations


21/57



Telephone Surveys

Like mail surveys, phone surveys are conducted using an existing list frame, like the yellow pages oran organizational staff phone directory. The researcher then calls population or sample membersusing that list, and either administers the survey then or calls back at a more convenient time.

(While various kinds of random dialing techniques also exist, you are not likely to use them for thisclinic. If you find that you need to, however, we suggest that you consult an appropriate resource.)Unlike mail surveys, though, the surveyor and the respondents interact personally, meaning thatyou can introduce bias and inconsistencies if youre not careful to present the questions in the sameway to everyone.

Phone surveys are best for when you need to get responses quickly, when you have an accurate listframe, and when your target population is likely to have phones. Their strengths and weaknessesare summarized in the chart below:

Phone Survey Strengths Phone Survey Weaknesses

Quick data collection time List frames often contain inaccuracies.

Good control over who respondsto the survey, question order, etc.

Interviewer can introducebias if not careful

Low to moderate cost,

depending on long distancecalls and rates

No visual communication

High accessibility to population(most people have phones)

Certain populations areunlikely to have phones

Easy to use complexand open-ended questions


22/57



Face-to-Face Surveys

Face-to-face surveys differ from interviews (discussed in the interview section below) in thatsurveyors administer the same questionnaire in person to a variety of respondents, as opposed toasking a series of questions tailored to the person being interviewed. The surveyor can either use a

list frame or can find respondents randomly, like by going to homeless shelters to interviewrecently evicted people. Its important to note, though, that if youre not using a list frame, youllneed to take care to make sure you arent biasing your results by administering the survey in onlyone place or at only one time a day, thereby missing lots of potential respondents. Its alsoimportant to keep in mind that, as with phone surveys, its easy for interviewers to introduce biasby asking questions in a non-uniform way.

Face-to-face surveys are best for when you dont have a list frame, for when your questionnaire iscomplex or requires a lot of explanation, or for when the survey requires personal, visual, oremotional cues. Their strengths and weaknesses are summarized in the chart below:

Face-to-Face Survey Strengths Face-to-Face Survey Weaknesses

No list frame is required High cost and time requirements

Provides access to populationswithout addresses, phones,and/or internet

Data collection is slow

Good control over who answers thequestions and in what order

Interviewer can introducebias if not careful

Easy to use complexand open-ended questions

Allows for incorporation ofvisual and emotional cues

Can decrease non-responserate by allowing interviewerto explain surveys importance


23/57



Internet Surveys

Internet surveys are relative newcomers to survey methodology, but they work similarly to mailsurveys. The main difference, of course, is that theyre administered online, not on paper and by

mail. They can also be more dynamic than mail surveys, since you can include links and animatedvisual effects. You should consider using an internet survey if you have limited resources, strongcomputer skills, and are convinced that your population is likely to respond electronically. Theirstrengths and weaknesses are summarized below:

Internet Survey Strengths Internet Survey Weaknesses

Low cost, infrastructure,

and resource needs High nonresponse rates

Good accessibility topopulations with internet

Some populations do nothave internet access

Good control over the orderof questions answered,depending on survey design

Little control over whoanswers questions

Utilization of links andanimations make them moredynamic than mail surveys

Difficult to use complexand open-ended questions


24/57



Hybrid/Mixed Mode Surveys

Mixed mode surveys combine survey methods, the most common combination being the drop-offsurvey. In a drop-off survey, the researcher leaves the survey with the respondent personally, butthen the respondent replies by mail. This allows the researcher to convey the importance of the

survey in person, but also utilizes the strengths of mail surveys. Another common type of hybridsurvey combines mail surveys with follow-up phone surveys of those who didnt respond by mail.The biggest strength of hybrid surveys is their ability to increase response rates, either byincreasing the number of contacts between the researcher and respondents or by giving theresearcher a chance to explain the importance of the survey. On the other hand, combining surveymethods can have complicated repercussions for analyzing and comparing responses.


25/57



C.Survey ErrorsAs you decide what survey method to use and get ready to design your survey, its important to becognizant of survey errors that are easy to make and have serious repercussions for the accuracy of

your survey. The four main survey errors are described below:

Coverage Error

Coverage error happens when not all population members have a chance of being surveyed. Forexample, lets say you use a phone book to administer a phone survey to Massachusetts housingadvocates. Any advocate who doesnt have a phone or whose number is unlisted has no chance ofbeing surveyed this is coverage error. To counteract it, make sure that every population memberhas an equal (or known) chance of being surveyed.

Sampling Error

Sampling error, discussed in the statistics crash course earlier, happens any time you survey only asample of your target population. Although it cant be entirely avoided, it can be estimated withreasonable accuracy and can be reduced by increasing the sample size.

Measurement Error

Measurement error, also discussed in the statistics crash course above, relates to the quality of datacollected, and occurs whenever a response is inaccurate, vague, or cant be compared to otheranswers. For example, lets say youve included a survey question asking clergy members if they

usually lump it when a dispute arises, but you havent explained what it means to lump it. Youcan expect to receive answers that are vague, left blank, or impossible to compare uniformly thisis measurement error. Minimize measurement error by being extremely cautious about whatquestions you ask, how you ask them, and whether youve inadvertently introduced bias whenasking them.

Nonresponse Error

Nonresponse error happens when a significant number of people surveyed do not respond andwhen those who dont respond are different from those who do in some significant way. Forexample, you may administer a mail survey to housing advocates across Massachusetts and then

discover that a high proportion of advocates in cities with a lot of subsidized housing didntrespond. This might indicate that they are busier and have less time to respond to surveys, whichwould have an impact on the accuracy of your results. Minimize nonresponse error by making yoursurvey easy to complete for allmembers of your target population.


26/57



D.SamplingAfter youve decided who to survey and how to survey them, youll need to decide whether to do acensus or whether to survey a sample of your target population. The main reason to sample is tosave yourself time and resources if your population is all residents of Massachusetts, then using a

sample can save you a lot of both. But youll need to take into account factors other than populationsize in deciding whether or not to sample. For example, what survey method will you use? If youhave a population of 200, then sending a mail survey to all members may make sense. If youreconducting face-to-face surveys, however, interviewing 200 people may be infeasible. Other majorconsiderations are how precise you need to be and the amount of variation within the population.(See the sampling section in the statistics crash course above for more information on how tosample.)

Its important to note that taking a census is not inherently more accurate than using a sample. Forexample, lets say youre conducting a mail survey of all housing advocates in Massachusetts. If youwere to do a census, you would contact each advocate, but you would have less time and resourcesto send preview and follow-up letters, and to call nonrespondents. This could lead to a bigger

nonresponse error than you would have if contacting a more manageable sample and engaging inefforts to raise the response rate. Furthermore, unlike the sampling error introduced by using asample, you would have no way to estimate the nonresponse error of your census survey. In otherwords, a well-conducted sample survey can be more reliable than a badly conducted census survey.


27/57



E.Question WritingSo, youve decided what information you need, who to get it from, and how youre going to get it.Now its time to write your questions. This process can sound intimidating, especially since there is

no one recipe for great question writing. Youll have to be careful to avoid introducing bias intoyour questions, and to ensure that they are written in a way that yields the information you needand that considers the abilities and characteristics of respondents. In this section well reviewsome considerations youll need to take into account, but we encourage you to consult otherresources as you write your questions.

Turning Ideas into Questions

To begin, youll need to go back to your reasons for conducting the survey in the first place andtranslate those information needs into good questions. This involves setting up questions thatprovide information in a way that can be measured and compared. For example, lets say youre

conducting a survey to find out how to improve the Catholic Churchs system for resolving disputesamong clergy members. In part, youll want to find out how clergy members feel about the currentsystem, an attitude that can be influenced by a number of factors like how often disputes areresolved under the current system and how often clergy members feel they have to lump it. Toget the information youre looking for, then, youll have to write a number of specific questions thattogether inform you about what youre researching.

Attitudes & Beliefs vs. Behavior & Attributes

Its helpful to be aware of the type of information youre seeking so that you can make sure yourquestions ask for it. Most information falls into two categories: the first is behaviors and attributes relatively concrete information like age or employment information and the second is attitudesand beliefs. Keep in mind that attitudes and beliefs are difficult to measure, and that differentpeople usually have different perceptions of the same thing or situation. For example, yourprofessor might rate a local restaurant as good, while you might rate it as excellent.Furthermore, your definition of excellent might be something like your professors definition ofgood. As a result, attitude and belief questions are more susceptible to measurement error thanbehavior and attribute questions.


28/57



F.Question StructureThere are two basic types of question structure: open-ended and close-ended. When youre writingquestions, youll need to consider which structure will better capture the information you need.

Open-ended Questions

Open-ended questions provide no answer choices instead, the respondent fills in the answer. Forexample, in the clergy study you might ask: What is the most common way that you deal withdisputes arising between you and other clergy members? Open-ended questions have severaladvantages and disadvantages, summarized in the chart below:

Open-ended Question Advantages Open-ended Question Disadvantages

Helpful when researchers donthave enough knowledge todesign answer choices or thinkthey have missed an importantanswer choice

Demanding for respondents

Helpful when researchers seekstrong opinions or emotions

Can produce many differentanswers about the same topic

Good for obtaining explanationsfor answers to other close-ended questions

Tend to produce answers thatare imprecise and difficult tocompare

Helpful when respondents areunlikely to know an exactanswer and must estimateinstead

More difficult to code andprepare for analysis


29/57



Close-ended Questions

Close-ended questions come in several varieties, but they all include a number of answer choicesfrom which the respondent picks. These answer choices should be specific, cover all possibleanswers, and be mutually exclusive. For example, you might ask the clergy members:

How many disputes have arisen between you and another clergy member over the last year?

1) 0-22) 3-53) 6-84) 9-105) 11 or more

This is an example of a close-ended question with ordered answer choices the answer choices movein an ordered way across some spectrum of possible answers. This format is the least demandingfor respondents, and usually provides very specific information.

You can also write close-ended questions withunordered answer choices, usually when possibleanswer choices do not fall on a continuum and when you have enough information to provide acomplete list of possible answers. These can be particularly helpful for ranking questions. Forexample, in your research about evictions, you might ask the local housing advocates:

Please rank the most common reasons for tenant evictions in the City of Boston, starting with 1

for the most common and 5 for the least common.

Noise ComplaintsNon-payment of rentUse of unit for illegal purposeDamage to unit

Exceeded occupancy limits

A final format for close-ended questions arepartially close-ended questions, which include a seriesof answer choices and an other option that allows respondents to invent an answer. This formatcan be helpful for generating new information.

When youre using close-ended questions, it is worth noting that some research has suggested thatpeople tend to choose from among the first answer choices in written surveys, and among the lastanswer choices in oral surveys. This effect, called the category order effect, can introducemeasurement error. To reduce any such error, try systematically varying the order of answerchoices, and avoid listing too many of them.


30/57



Some Final Tips

Before we move on from the complex topic of question writing, wed like to give you a few finalpointers about common question writing pitfalls:

Avoid vaguely worded questions and responses, which usually produce useless information

Rank the dedication of your staff, on a scale of one to five (1 = very dedicated; 5 = totally

undedicated).

Problem

Who is the respondent thinking of when she thinks of her staff?

What is definition of dedicated does the respondent use to answer this question?

Avoid making questions too difficult to answer, like by requesting too much precision or bymaking respondents do calculations

Please give the percentage of sick days claimed by your staff that fell on Mondays or Fridays.Problem

To answer this question, your respondent will have to count up all sick days for all of her staff, and subsequently

divide that number by the number of total sick days that fell on a Monday or Friday. It is unlikely that yourrespondent will do this calculation accurately

Be cognizant of questions respondents may be not want to answer, like personal questions

Have you or anyone in your family ever had an abortion?

Problem

Who wants to answer this kind of personal question?

Avoid using language, such as jargon, abbreviations, and cryptic wording, that respondentsare unlikely to understand or can easily misunderstand.

Have you or your business ever resolved a case before the ICC ?

Problem

Does ICC refer to the International Criminal Court or the International Chamber of Commerce?

Avoid collapsing questions when they can be easily asked separately

Have you ever felt neglected or unfairly punished by this professor ?

Problem

These are actually two separate questions:

Have you ever felt neglected by this professor?

Have you ever felt unfairly punished by this professor?

Asking them in the aggregate gives you no clarity about which of the two questions respondents are answering.

Common sources of bias include:


31/57



o Making a certain behavior or attitude appear normal

When is it appropriate to torture suspects?

Problem

This question assumes that it is appropriate to torture suspects in some instances. Of course a respondent might

answer never, but the mere framing of this question indicates that there must be some instances where it is

appropriate.

o Using a subjective tone or making objectionable statements in questions

In 2006, only 32% (or less than one third) of respondents indicated that they found sexual

relations between two adults of the same sex to be not wrong. The rest of the population found

some fault with such relationships, despite the Supreme Courts narrow holding that such

relationships are legal. Do you personally object to two consenting adults of the same sex

being joined in marriage?

Problem

The supposedly factual introduction to this question biases the respondent, by framing opposition to gay marriage

as the societal norm, since even mere tolerance of homosexual relationships is seemingly a minority position

(according to this biased question). No information is given as to the source of these facts, or how we can be surethat these findings themselves rely on solid research methods.

o Answer choices weighted so that there are more choices reflecting one attitude ortendency than another

What factor will weigh most heavily in your choice for president in 2008?

a. The poor state of our economy.b. The Republican partys handling of the war in Iraq.c. The Republicans poor record when it comes to environmental protection.d. The lack of quality affordable healthcare in this country.e. This countrys disregard for international diplomacy.

f. National security.

Problem

While the six options listed all figure into the election, they include many more options pertaining to grievances

traditionally held by Democratic voters than grievances held by typically Republican constituencies.


32/57



G.Questionnaire DesignOnce you have your questions ready, youll need to consider how to present them so that they

encourage respondents to complete the survey and are clear and easy to answer. Here we give yousome general rules of thumb.

The First Question

Respondents often decide whether or not to complete a survey while reading or listening to thefirst question, so it should be easy to answer, non-threatening, and clearly related to the statedpurpose of the survey. Avoid starting with an open-ended question, and never start with personalquestions, like household incomes or demographics; respondents are much more likely to answerthis type of question later in the survey.

Question Order

Be careful about how one question can affect the next. Consider, for example, two questionspresented consecutively:

1. How satisfied are you with your supervisors dispute management skills?

a) Very satisfiedb) Somewhat satisfiedc) Neither satisfied nor dissatisfiedd) Somewhat dissatisfiede) Very dissatisfied

2. In general, how satisfied are you with your job?

a) Very satisfiedb) Somewhat satisfiedc) Neither satisfied nor dissatisfiedd) Somewhat dissatisfiede) Very dissatisfied

The first question gets the respondent thinking about one particular aspect of her job, and then thesecond question asks her to consider her job overall. If she feels negatively about her boss disputeresolution skills, this can prime her to answer the second question more negatively that sheotherwise would have.

While there is no one way to deal with this issue, it is helpful to be aware of it in ordering yourquestions. Testing out your survey design with friends and colleagues, and asking them tocomment on possible unwanted biases introduced by your question order, is an important way tominimize such interference.


33/57



Question Clusters

Generally speaking, clustering questions about the same topic makes it easier for the respondent tothink about that topic. It also helps to cluster questions of the same format together.

Narrative

Using a narrative tone can help respondents feel more comfortable about answering personalquestions, and can help signal transitions and changes in topic. For example, it can be helpful to saysomething like now wed like to ask you a few questions about yourself before asking personalquestions.

Oral vs. Written Surveys

Keep in mind that respondents process written questions differently from oral ones. In oral

surveys, questions need to be shorter so that they are easier to remember, and you should avoidlong lists of answer choices. Written surveys should be easy to read and navigate.


34/57



H.Analyzing ResultsOnce youve administered your survey, its time to analyze your results! This process involvespreparing surveys for analysis, analyzing using statistical methods, and then interpreting the

results. Here we give you an overview of these three steps.

Cleaning and Coding

The first thing youll need to do is clean and code your surveys to prepare them for analysis.Cleaning a survey simply means looking through it and erasing anything extraneous or clearlyerroneous. For example, you might have asked respondents for their year of birth, and one or twomay have accidentally written the current year instead. These answers should be discarded, alongwith any clear inconsistencies in data. Coding responses means assigning numbers to answerschoices so they are easy to compare. It is usually easiest to assign codes before administering thesurvey so that the appropriate numbers can be circled or otherwise marked by either the

respondent (in written surveys) or the interviewer (in oral surveys). While close-ended questionsare the most straightforward to code, open-ended questions require more work. First youll needto review the answers to develop categories that they all fit into, and then you can go back and codeanswers according to those categories. You might decide not to code open-ended questions thatwere mainly ignored, or that served a purely exploratory purpose.

Analysis

Once youve coded your surveys, youll probably want to enter the results into a computer databaseor software program. Its usually best to arrange answers in a chart so that the rows contain all ofthe answers of a particular respondent and the columns compare the responses to the samequestion. To protect confidentiality, avoid naming the rows according to respondents names.

A lot of computer programs will calculate measures of central tendency and dispersions for you,since these are the most commonly used statistics (theyre discussed above, in the statisticssection). Which statistic to use depends on what information youre looking for. For example, letssay youre interested in how most clergy members currently deal with disputes. A percentdistribution would probably be most helpful, enabling you to report that, say, 40% of clergymembers say they lump it when they have a dispute, while only 15% say they report the disputeto a superior. If youre interested in how many disputes arose among clergy members in the lastyear, a mean would probably be helpful, enabling you to say, for example on average, clergymembers reported being involved in 5 disputes last year. On the other hand, a dispersion wouldprobably be most helpful for comparing how many disputes arose on average in different churches,enabling you to say, for example, Church A reported 15 disputes last year, while Church B reportedonly 2. There are no strict guidelines on which statistic to use when, so we encourage you toconsult other resources for more guidance if you feel that you need it.


35/57



Interpretation

Once youve analyzed your survey results, its time to go back to the question that prompted yoursurvey in the first place. While there is no one way to interpret the results, we recommendstepping back to look at the big picture what big or surprising results popped up that tell yousomething about your research question? As always, we encourage you to consult with your

professors and other outside resources when interpreting your survey results.

I. EthicsThere is one other, extremely important topic to think about at every step of your survey designand implementation: ethics. First and foremost, you need to make sure to protect theconfidentiality of respondents. Remember, most surveys arent anonymous youre usuallyworking with a list of addresses, phone numbers, employees, and so on, all of which come withnames. Confidentiality means that although you can report the identities of respondents, you wont.Always make sure that you report your results in the aggregate, without ever mentioning a specificrespondent, and destroy anything that can link responses to particular respondents as soon as itspractical. Its also important to notify respondents that the surveys are confidential, notanonymous, and explain how you will protect their confidentiality.

Another ethical consideration that can arise with surveys relates to participation. With manysurveys, youll need to convince respondents to complete the survey and sometimes contact themmultiple times. There is a line between convincing and coercing, however, and you must alwaysrespect the right of respondents to decline to participate.

J.References1. Druckman, D. (2005). Doing Research: Methods of Inquiry for Conflict Analysis. Thousand

Oaks, CA: Sage. (Chapter 5)

2. Salant, P. and Dillman, D. (1994). How to Conduct Your Own Survey. London: New York:Wiley.


36/57


37/57



INTERVIEWS

In the course of your clinical project, you will likely conduct several stakeholder interviews. Prior tointerviewing individual stakeholders, you will need to draft an interview protocol, a list of questionsto be asked during the interview. As you conduct the interview, your primary goals will be to buildtrust between yourself and the stakeholder, to gather factual information about the conflict, and tolearn about the stakeholders interests. Following each interview, you should type up your notesand send them to the stakeholder to ensure their accuracy. Below are some guidelines forconstructing your protocol and conducting a stakeholder interview.

A.General GuidelinesIntroduction

Begin the interview by introducing yourself, giving the stakeholder a brief overview of HNMCP, anddescribing the project to which the interview relates. In general, it is good practice to send thestakeholder a letter of introduction, containing this information, prior to the interview. Doing sowill not only save time during the interview, it will give you the chance to provide a thoughtful,thorough, and consistent set of background information to each interviewee. If it is infeasible tosend a letter of introduction to each stakeholder, you should tailor the introductory portion of theinterview on a case-by-case basis. If a stakeholder already has a significant amount of b

2010 Research Methods Handbook

Documents

Transcript of 2010 Research Methods Handbook