IHTE-1800 Research methods: Surveys and interviews Sari Kujala, Spring 07.
-
date post
22-Dec-2015 -
Category
Documents
-
view
222 -
download
0
Transcript of IHTE-1800 Research methods: Surveys and interviews Sari Kujala, Spring 07.
http://www.cs.tut.fi/ihte
What is a survey? (Pfleeger& Kitchenham, 2001)
• A comprehensive system for collecting information to describe, compare or explain knowledge, attitudes and behavior
http://www.cs.tut.fi/ihte
Types of surveys
• Supervised- Telephone interviews
- Group surveys
• Unsupervised- Mailed questionnaire
- Electronic questionnaire
http://www.cs.tut.fi/ihte
Survey process (Pfleeger &Kitchenham, 2001)
1. Setting measurable objectives2. Planning and scheduling the survey3. Ensuring that appropriate resources are available4. Designing the survey5. Preparing the data collection instrument6. Validating the instrument (piloting)7. Selecting participants8. Administrating and scoring the instrument a9. Analyzing the data10. Reporting the results
http://www.cs.tut.fi/ihte
Setting measurable objectives
• Preferably research questions or hypotheses
• Statements of the survey’s expected outcomes- What information will be identified- Target population
• Definitions of all potentially ambiguous terms
http://www.cs.tut.fi/ihte
Designing the survey
• The goal is to provide the most effective means of obtaining the information- No bias
- Apparopriate (makes sense in the context of the population)
- Cost-effective
http://www.cs.tut.fi/ihte
Descriptive designs
• Cross sectional- Information at one fixed point in time
• Cohort- Information about changes in a specific
population
• Case control- Retrospective information about previous
circumstances to help explain a current phenomenon
http://www.cs.tut.fi/ihte
Experimental designs
• Concurrent control studies - participants are randomly assigned to groups - E.g. is training changing attitudes?
• Concurrent control studies – participants are not randomly assigned to groups
• Self-control studies - Pre- and post-treatment measures
• Historical control studies - E.g. comparison with previous surveys
• Combination of techniques
http://www.cs.tut.fi/ihte
Sample size
• Sample size should be big enough- If different groups are compared, each group
should have app. 30 data points in order to make statistical analyses
• Start sampling by defining a target population- May be a subset of a larger population,
inclusion or exclusion criteria may be used
http://www.cs.tut.fi/ihte
Sampling methods
• Probabilistic- Random
- Stratified random sample (subgroups)
- Systematic sampling (every nth member)
- Cluster-based sampling (belonging to a defined group)
• Non-probabilistic- Convenience (who is available), snowball
http://www.cs.tut.fi/ihte
Response rate
• A low response rate can destroy a good survey (range is from 10 % up to 90 %)
• The reasons for non-response should be known
http://www.cs.tut.fi/ihte
Improving response rate
• Over-sampling, reminders, rewards
• Ensuring that people are able, willing and motivated to answer the questions
• Respondents should see some clear benefit to answering the questions
http://www.cs.tut.fi/ihte
Data collection instrument
• Search the relevant literature- Existing instruments?
• Construct an instrument
• Evaluate/pilot the instrument
• Document the instrument
http://www.cs.tut.fi/ihte
Constructing an instrument: what to ask?
• Use survey objectives
• Consider the respondents- Questions should be easy and accurate to
answer, events not happened long in the past
- Respondents have sufficient knowledge and position to answer
• Remember background or demographic questions to identify the respondent
http://www.cs.tut.fi/ihte
Question types
• Open questions
• Standardized response format- Multiple-choice question
- Likert scale statements
http://www.cs.tut.fi/ihte
How to ask
• Don’t ask too many questions – too time demanding to answer –low response rate- Think every time: how you can use the answers, what
is your hypothesis• Don’t ask too many open questions
- Laborious to answer- Difficult to classify, time consuming analyze
• Standardize responses when appropriate - offer possibility to answer “other”
• Give the respondent enough instructions
http://www.cs.tut.fi/ihte
Information and instructions for respondents
• Include a cover letter providing a contact name and information
• Explain the purpose and relevance of the study• Describe who is sponsoring the study and how
confidentiality will be preserved• Explain how the respondents were chosen and
why• Explain how to return the questionnaire• Provide a realistic estimate of the time to
complete the questionnaire
http://www.cs.tut.fi/ihte
A good question is:
• Purposeful from the respondents’ point of view
• Clear• Neutral, not leading• Concrete• Concentrating on essentials• Asks only one issue• All in all easy to answer
http://www.cs.tut.fi/ihte
Example: Requirements quality questionnaire
Disagree Agree don’t 1 2 3 4 know
Requirements are completely defined.
1 2 3 4 [ ]
The requirements describe a system that meets user needs.
1 2 3 4 [ ]
The requirements are based on the information gained from users and customers.
1 2 3 4 [ ]
In all likelihood, there are moderately few errors in the requirements.
1 2 3 4 [ ]
http://www.cs.tut.fi/ihte
Piloting
• Is the most important way of improving the quality of the questionnaire- Use real people belonging to target group
- How is your questions and the used words understood?
- Is all options available?
- Iterate
http://www.cs.tut.fi/ihte
Types of reliability
• Test-retest reliability
• Internal consistency- Is a group of items forming a single scale?
- Statistical measures
• Inter-observer reliability when observers are completing a survey instrument- Statistical measures
http://www.cs.tut.fi/ihte
Types of validity
• Face validity• Content validity
- A group of reviewers with knowledge of the subject matter and target population members review the survey contents
• Criterion validity- A measure of how well one instrument compares with
another instrument or predictor• Construct validity
- The extent to which different data collection approaches produce similar results
http://www.cs.tut.fi/ihte
Analyzing the data
• Check the incomplete questionnaires
• Partition the responses according to subgroups
• Make figures of the results
• Statistical analysis depending on the scale type of replies- Frequency, mean, variance etc.
- Chi-squared test to measure associations among nominal scale variables
- Variance analysis, correlations
http://www.cs.tut.fi/ihte
Interviewing types
• Structured interviews- Questionnaire used
• Semi-structured or thematic interviews- Pre-defined themes used
- Additional questions in order to understand the answers and find new interesting issues
• Open interviews- More like free discussions
• Individual or group interviews, focus groups
http://www.cs.tut.fi/ihte
Literature• Bourque, L. and Fielder, E. (1995) How to conduct self-
administrated and mail surveys, Sage Publications Inc.• Fink, A. (1995). The Survey Handbook. Sage Publications.• Pfleeger and Kitchenham (2001). Principles of survey research.
Parts 1-5. Software Engineering Notes. • Straub, D.W. (1989) Validating Instruments in MIS Research, MIS
Quarterly, 13, 2, 147-169. • Andrews, D., Nonnecke, B., Preece. J. (2003) Electronic survey
methodology: A case study in reaching hard-to-involve internet users. International journal of human-computer interaction, 16, 2, 185-210.
• Ropponen, J. and Lyytinen, K. (2000) “Components of Software Development Risk: How to Address Them? A project manager survey”, IEEE Transactions on Software Engineering, 26, 2, 2000, 98-112. (good example)