Post on 24-Dec-2015
Using the Internet to Collect Primary Research Data
Postgraduate Research Methods Day Conference,
The Open University,
Camden Regional Centre
Claire Hewson,
Department of Psychology,
The Open University
claire.hewson@open.ac.uk
A (very) Brief History of Internet-mediated Research (IMR)
Pioneers started implementing online data collection methods around the mid-1990s Bordia (1996); Dillman (1991); Gaiser (1997); Hewson (1994).
Reports (e.g. validation studies) have since supported the quality of data that can be obtained in IMR Corley & Scheepers (2002); Hewson & Charlton (2005).
A new era of IMR (Web 2.0) The web as organic, collaborative, interactive, ever-changing (e.g. Wikipedia); the rise of social media (e.g. Facebook, Twitter, YouTube).
IMR Data Collection Methods
Obtrusive Approaches Surveys, experiments, interviews, participant and / or disclosed observation approaches.
Unobtrusive Approaches Observation using traces and archives, e.g. discussion group archives, browsing behaviours, social networking links (SNA); lurking?
'Big Data' Data sets which become so large as to be difficult to manage; presently a 'hot topic'.
The Visualisation of Complex Networks
Nike+ involves the placement of a sensor underneath the footbed of your Nike running shoe in order to collect data about where you've run, how long it took and where you can improve over time.
http://www.visualcomplexity.com/vc/
Visualisation of a year's worth of runs uploaded to the Nike+ website. It plays back runs throughout three cities: New York, London and Tokyo. The runs showed tens of thousands of peoples' runs animating the city and bringing it to life. The software visualizes and follows individual runs, as well as showing the collective energy of all the runners, defining the city by the constantly changing paths of the people running in it.
Web-based Surveys and Experiments
There is now a large range of software for helping implement these, particularly survey packages (which can also support simple experimental designs), e.g. 'Surveymonkey', 'Qualtrics'.
Dedicated packages to support experimental web-based studies include: WEXTOR (Reips, wextor.org/wextor/en) and WEBEXP (Edinburgh webexp.info).
Web-based Surveys and Experiments
New design issues and principles emerge in this novel context. Useful resources and guidelines include:
Exploring Online Research Methods: www.restore.ac.uk/orm
WebSurveyMethodology: websm.org
Couper (2008). Designing Effective Web Surveys. CUP: New York.
Reips & Krantz (2010). Conducting True Experiments on the Web. In Gosling & Johnson (eds) Advanced Methods for Conducting Online Behavioral Research. APA.
Fielding, Lee & Blank (2008). The Sage Handbook of Online Research Methods. Sage: London.
Recruiting Participants in IMR
Sample Bias Issues?, e.g. Bordia (1997).
True Volunteers, place adverts in public spaces, e.g. newsgroups, webpages, IMR study clearing houses, Facebook.
List-based Sampling Frame, e.g. email lists, online panel.
Probability Sampling Methods, e.g. random selection from known list, online probability panel.
General Principle: methods which allow some estimation of the sampling frame are preferable where measures of response rate, non-response bias, etc. are important to the research design.
Online Sampling Resources
● Online Study Clearing Houses, e.g. Online Psychology
Research: http://www.onlinepsychresearch.co.uk
● Newsgroups, SNSs, e.g. Facebook: facebook.com, Newsgroups (e.g. http://uk.groups.yahoo.com; https://www.groups.google.com), Mailing Lists (see the searchable Catalist database: http://www.lsoft.com/catalist.html)
● Online Panels, e.g. Knowledge Networks: http://www.knowledgenetworks.com/knpanel
Online workforces, e.g. Mechanical Turk: www.mturk.com
OU Participant Panel, Jim will talk more about this
OU ‘Virtual Participant Panel’
URL: http://www.open.ac.uk/socialsciences/psychology/cogpsypanel/
People can participate in studies ad hoc and/or… … can sign up to be contacted for future studies. Sign-ups give certain information (e.g. sex, age,
location, handedness) which can be used to filter if particular characteristics are needed in a sample.
Currently has 1,245 signed-up members. Has a built-in ‘Questionnaire Manager’ to create
surveys.
OU ‘Virtual Participant Panel’
Instruction text
Question text
Response text
OU ‘Virtual Participant Panel’
Stimulus image (‘question’)
Response images (‘answers’)
Response buttons
SurveyMonkey Example
● Computer Attitudes Surveyhttps://www.surveymonkey.com/s/computer_attitudes_study
● Qualtrics – site license for all OU social science faculty students / staff. Visit 'openss.qualtrics.com' and sign up with a valid OU (open.ac.uk) email address.
Martin discussed using Qualtrics in a radiology expertise study this morning, and Jim will demonstrate an online experiment implementation using Qualtrics.
Importing studies from SurveyMonkey into Qualtrics?
Qualtrics
Qualtrics
Stimulus image
Response images
Ethics in Online Research
Some ethics issues require special consideration in an IMR context, the complexities of which are not always obvious to researchers or participants.
e.g. Data security, traceability (confidentiality, anonymity).
Key feature of IMR: involves acquisition of data from or about individuals in absence of face-to-face co-presence
COUPLED WITH greater scope for carrying out quite complex interactive procedures
The New BPS Guidelines (November 2013)
Document aims to outline some of the key ethics principles for researchers and ethics committees to keep in mind when designing and / or evaluating an IMR study
Emphasis on how to apply existing ethics principles in an IMR context
“Thinking is not optional”
The New BPS Guidelines (November 2013)
● Centred around the four main principles of the BPS 'Code of Human Research Ethics' (2011).
1. Respect for the autonomy and dignity of persons
2. Scientific value
3. Social responsibility
4. Maximising benefits and minimising harm
Main Ethics Issues to Consider in IMR
Public-private domain distinction Confidentiality* Copyright Consent; withdrawal; debrief*● Levels of control*● Disruption of social structures
Valid Consent
Verifying characteristics (e.g. age, sex)
Engagement with consent process (check boxes?)
Adequate information – not all risks online are obvious, e.g. potential data leakage (balance between information and overload)
Risks will be higher in some contexts than others – key principle of proportionality of ethics procedures to levels of risk of harm re-emerges
Withdrawal and Debrief
Robust withdrawal procedures may be more difficult to implement online (e.g. volunteer web survey)
Detecting withdrawal may be harder (use of partial data?)
Ensuring debrief difficult (e.g. exit by closing browser)
Enhanced risk in cases of deception (no debrief)?
Principle of proportionality of ethics procedures to levels of risk (of harm) re-emerges, e.g. effects of withdrawal without debrief (such as in very sensitive research)
Confidentiality, Anonymity
● Enhanced traceability of data, dissemination implications, use of quotes.
Other data leakage risks, e.g. email.
Traces on personal computing equipment.
Sensitivity of data is a factor associated with levels of risk of harm.
Researchers should carefully assess risks, and perhaps inform participants of these risks.
Data Protection
UK Data Protection Act (1998).
Issue of storing data on non-UK/EU servers?
Popular software packages (e.g. SurveyMonkey, Qualtrics) are US-based.
'Safe Harbour Scheme'.
Office of National Statistics SurveyMonkey, Safe Harbour Scheme
Levels of control
Over procedures, who takes part, participant behaviours...
Implications for scientific value (validity)
Implications for levels of harm (monitoring participant reactions, presenting debrief info., etc.)
Maximising Benefits, Minimising Harm
These considerations permeate the other issues
Careful assessment needed; trade-offs will be inevitable (e.g. anonymity versus traceability/verifiability; disclosure versus social disruption/harm)
“Thinking is not optional”
Some Examples
● http://www.socialpsychology.org/expts.htm http://www.onlinepsychresearch.co.uk http://psych.hanover.edu/Research/exponnet.html http://www.wexlist.net