GDC17 A Practical Guide to Doing Ethical Player Testing
-
Upload
mia-consalvo -
Category
Education
-
view
292 -
download
0
Transcript of GDC17 A Practical Guide to Doing Ethical Player Testing
A Practical Guide to Doing Ethical Player Testing Mia Consalvo Concordia University
Who am I? ● Studying game players since 2000 ● Use a variety of methods to better
understand play ● Wrote Cheating, Players and their Pets,
Atari to Zelda ● Run the mLab at Concordia University
Outline for today’s talk ● A brief history of why ethical research became a
thing at universities ● Important concepts to keep in mind for testing ● Questions to ask yourself when prepping for
testing ● What about VR? ● Q&A
What this talk is NOT about ● Improving your game design through
player testing
What this talk IS about ● Ensuring play testers are getting the best
possible experience (and want to come back again)
● Improving the quality of testing feedback
A caveat ● “Following a code of ethics is not the
same as being ethical. A domain-specific ethics code … can never function as a substitute for ethical reasoning itself.”
A brief history of human subjects research
The Milgram experiment (1961) ● Studied obedience to authority figures via
willingness to give increasingly severe ‘shocks’ to others
● 65% of participants administered final massive 450-volt shock
● Study criticized for use of deception, other flaws
Stanford Prison Experiment (1971) ● A study of the psychological effects of
becoming a prisoner or prison guard ● Conducted by Philip Zimbardo with a
group of college students ● Prisoners ‘arrested’ at their homes, driven
to prison, processed, given jumpsuit and assigned an ID number
Stanford Prison Experiment ● Experiment abandoned after six days –
some prisoners had breakdowns ● Subjects could not leave voluntarily ● Zimbardo was not a neutral observer ● No debriefing was done to assess
potential harms done
The introduction of protocols and guidelines for doing federally funded research
Consent
What is consent? ● Free, informed and ongoing consent ● Free = voluntary
Voluntary Consent ● No Undue Influence or Coercion ● Is there a power relationship between the
Investigator and potential participants? ● Is the investigator relying on the trust or
dependency of particular people (ex: my students)?
● Threat of harm or punishment for failure to participate?
Informed Consent
● Participants know what is expected of them during the test
● How long the test will last ● How will they be observed/recorded ● It’s okay to change their mind
Informed Consent ● Told of foreseeable risks and potential
benefits ● Under no obligation to participate, free to
withdraw at any time ● Whether participants will be identified
directly or indirectly
Informed Consent ● Contact information if they have
questions after the fact ● Information about how data will be used,
who it (may be/will be) disclosed to ● Information about payments or incentives ● Time to consider all of this, ask questions
Ongoing Consent ● Consent is not a ‘once and you’re good’
decision ● Consent can be negotiated if there are
different aspects to the play test
Withdrawing Consent ● Testers should be told they can stop the
test at any time ● The environment should support this
possibility ● There should be multiple ways to
withdraw (verbal, nonverbal/written)
Using incentives to participate ● Incentives can attract more potential
participants ● If incentives are very large or valuable this
may encourage participants to disregard risks ● Economic circumstances of participant pool,
age and capacity, customs and practices
Use of deception ● Deception can be a part of ethical
research ● What is the risk involved in the
deception? ● Does the benefit of deception outweigh
potential risks?
Harm and Risk
Defining Harm and Risk ● Minimal risk: probability and magnitude
of possible harms implied by participation in the study is no greater than those encountered by participants in those aspects of their everyday life that relate to the research
Beyond minimal risk
Vulnerable Populations ● Children ● Diminished capacity ● Indigenous peoples
Children ● Consent usually obtained through parents/
guardians ● Consider the age range you are including and
why ● Children who are 7, 10 and 13 are very different
in terms of what they understand as risks, as feedback, as consent
Children ● During testing, nonverbal communication
can be more important than what is being said (for both testers and researchers)
Children ● For a detailed examination of kids and
play testing, check out Gareth Griffiths’ 2014 GDC talk “Child’s Play: Playtesting with Children in the World of Skylanders”
● http://www.gdcvault.com/play/1020348/Child-s-Play-Playtesting-with
Privacy and Confidentiality
Privacy and Confidentiality ● How are you collecting data? ● How are you storing data? ● How are participants identified? ● What are you doing with the data? ● What happens to the data when the study
is over?
Privacy and Confidentiality ● Levels of disclosure of participant identity
● Anonymous ● Pseudonyms/Confidential ● Identifiable
● What personal information are you collecting and how might it be linked to participants’ tests?
Special Considerations for VR
“VR poses risks that are novel, that go beyond the risks of traditional psychological experiments in isolated environments & go beyond risks of existing media technology for the general public” –Madary & Metzinger, 2016
“the virtual pit” ● Subject with HMD stands on ‘ledge’ and asked to
lean over and drop a beanbag into a deep pit ● Subject stands on wooden platform 1.5” from
the ground ● Showed increased signs of stress through
increases in heart rate, skin conductance
Remember the Milgram Experiment? ● Re-done in VR environment ● Asked participants to administer shocks
to a virtual human performing memory tests [they knew she was virtual]
--Slater M, Antley A, Davison A, Swapp D, Guger C, et al. (2006) A Virtual Reprise of the Stanley Milgram Obedience Experiments. PLoS ONE 1(1): e39. doi:10.1371/journal.pone.0000039
Virtual Reprise ● 2 participants emphasized correct answers while
reading words ‘in an attempt to help her’ ● 8 repeated the question after receiving no
response ● Voices of some participants showed increasing
frustration with wrong answers
Virtual Reprise ● Often behaved in a way that only made sense if
they were responding to the virtual character as if she were real
● Humans respond realistically at subjective, physiological, and behavioral levels in interactions with virtual characters notwithstanding their cognitive certainty that they are not real
Virtual Reprise ● Might be continued evidence that subjects will
be obedient to an authority figure ● May be a matter of participants being willing to
put up with their own discomfort for the sake of honoring their agreement to be a participant in the experiment
Virtual Reprise ● Might be continued evidence that subjects will
be obedient to an authority figure ● May be a matter of participants being willing to
put up with their own discomfort for the sake of honoring their agreement to be a participant in the experiment
Proteus Effect ● Subjects conform to the behavior that
they believe others would expect them to have based on the appearance of their avatar
● Behavior in VE can have lasting psychological impact after subjects return to the physical world
Guidelines for VR research ● Do no harm ● No real history we can use as a source for
insight ● Tautology?
Guidelines for VR Research ● Have an explicit statement (or explain) to
the effect that “immersive VR can have lasting behavioral influences on subjects, and some of these risks may presently be unknown”
Guidelines for VR Research ● Tell people that they may have powerful
emotional responses to game content whether or not they ‘believe’ it’s real
● Torture in a virtual environment “is still torture”
Guidelines for VR Research ● Many new, additional kinds of data being
collected ● Eye-movements, emotions, real-time reactions, bodily
movements (mo-cap) ● One’s kinematics may be uniquely connected to one’s
identity ● Consider what data you really need to keep and
what might be erased to preserve privacy
Broader considerations beyond testing
● We don’t know the psychological impact of long term immersion in VR (addiction, manipulation of agency, unnoticed psychological change, mental illness)
● The potential for abuse of avatars that look like their users
Creating a protocol for play testing – Questions to guide you
What am I testing for? ● Do players understand what to do in the
first level? ● Do my instructions make sense or are
players floundering around? ● How long do players take to get to the
end of the level?
Recruitment ● Who is my player base? How can I recruit
players from that group without exploiting trust/power relationship?
● Should I provide an incentive or reward for play testing?
● Where am I asking them to play test?
Informed Consent ● How will I handle obtaining consent? ● What do I say about withdrawing
consent? ● How can I ensure testers feel comfortable
enough to leave if they really want to quit testing?
Deception ● Is there anything in the game I want to be a
surprise to testers? ● Is there a way to let testers know there’s some
material they might be really bothered by without spoiling the content for everyone?
● Is this ‘surprise’ really worth it?
Harm and Risk – VR Edition ● Having a ‘kill switch’ ● Testers’ avatars have protective bubble ● Protected populations issues
● Find information or agencies that testers might want to contact if they are bothered by content in your game
Privacy and Confidentiality ● What should I record – gameplay, tester audio,
tester video, physiological responses? ● If I share data with anyone, am I keeping tester
identities anonymous or confidential? ● How am I storing data and keeping it secure? ● What am I doing with that data after testing is
done?
Questions for multiplayer testing ● Do players understand how they can interact
with other players? ● Do instructions for that make sense? ● Are players using anti-harassment tools we
created? ● What sorts of toxicity (if any) am I seeing during
play?
Harm and risk – an aside ● Be wary of defining what ‘harassment’
looks like/sounds like ● Ask what they might have seen done/said
about other players, not just this player ● ‘Abuse’ versus ‘Drama’
Putting it all together – Consent Form Language/Guidelines for testers to understand and agree to
Example Consent Form Text ● “I understand that I have been asked to
participate in play testing of Eksa: Isle of the Wisekind by Mia Consalvo of GAMBIT Studios, contact information HERE.”
Example Consent Form Text ● “I have been informed that the purpose
of the play testing is to determine [how clear directions are in the game, and how challenging puzzles are in the first few levels.]”
Example Consent Form Text ● “I understand that I am being asked to
play a game that isn’t finished, for about 30 minutes.”
● “I understand that the computer is video recording my gameplay and I am being observed by someone while I play, who is taking written notes.”
Example Consent Form Text ● “I understand that I am free to withdraw
my consent and discontinue my participation at any time without negative consequences.”
Example Consent Form Text ● “There might be certain risks in
participating in this play test. These risks include feeling frustration or anger while playing, or hearing other players saying negative things during gameplay which might upset me.”
Example Consent Form Text ● “I understand that my participation in this
study is… ● CONFIDENTIAL (the researcher will know, but
will not disclose my identity) ● NON-CONFIDENTIAL (my identity will be
revealed in study results)”
Example Consent Form Text ● “I have carefully studied the above and
understand this agreement. I freely consent and voluntarily agree to participate in this study.”
Additional Resources ● For US-based developers
● https://phrp.nihtraining.com/index.php ● The Belmont Report ● http://www.hhs.gov/ohrp/regulations-and-policy/belmont-
report/index.html
Additional Resources ● For Canadian-based developers
● http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/Default/
● For European-based developers ● http://www.eurecnet.org/index.html
Additional Resources ● Madary M and Metzinger TK (2016) Real
Virtuality: A Code of Ethical Conduct. Recommendations for Good Scientific Practice and the Consumers of VR-Technology. Frontiers in Robotics and AI. 3:3. doi: 10.3389/frobt.2016.00003
Questions?