PriBots: Conversational Privacy with Chatbots (at SOUPS 2016)
50
PriBots Hamza Harkous 1 , Kassem Fawaz 2 , Kang. G. Shin 2 , Karl Aberer 1 1 EPFL; 2 University of Michigan Conversational Privacy with Chatbots Workshop on the Future of Privacy Notices and Indicators, SOUPS 2016
-
Upload
hamza-harkous -
Category
Technology
-
view
74 -
download
0
Transcript of PriBots: Conversational Privacy with Chatbots (at SOUPS 2016)
PriBots
Hamza Harkous1, Kassem Fawaz2, Kang. G. Shin2, Karl Aberer1 1EPFL; 2University of Michigan
Conversational Privacy with Chatbots
Workshop on the Future of Privacy Notices and Indicators, SOUPS 2016
2
Privacy Notice: Current State
legally binding
human understandable
Dual Role:
2
Privacy Notice: Current State
Can we split these roles?
legally binding
human understandable
Dual Role:
2
Privacy Notice: Current State
Figure 1. An example of a standardized table is shown on the left, and a standardized short table on the right. The comparison highlights the rows
deleted to “shorten” this version. These deleted rows are listed directly below the table. While both formats contain the legend (bottom right), it is
displayed only on the right here due to space constraints.
have already been deployed by major corporations, makingthem a viable, real world summary format for privacy poli-cies. These policies were stripped of brand information, butthe formatting and styles were retained.
METHODOLOGYWe conducted an online user study in summer 2009 usingAmazon’s Mechanical Turk and a tool we developed calledSurveyor’s Point. Mechanical Turk offers workers the abil-ity to perform short tasks and get compensated. People canplace jobs through Mechanical Turk, specifying the numberof workers they are looking for, necessary qualifications, theamount they are willing to pay, and details about the task.Mechanical Turk payments are generally calibrated for thelength of the task. For our approximately 15-minute study,we paid $0.75 on successful completion.
We developed a custom survey-management tool called Sur-veyor’s Point to facilitate our data collection. Our implemen-tation allows us to show respondents a single question on thescreen along with links for switching back and forth betweentwo policies within a single browser window. This allowedus to track the number of users who looked at each policyand the number of times they switched between them. Addi-tionally, Surveyor’s Point allowed us to collect the amount oftime that users spent reading the policies, as well as informa-tion about whether they clicked through to opt-out forms, toadditional policy information links, or from a layered noticethrough to the full text policy.
In preparation for this study we first performed three smallerpilot tests of our survey framework. We ran our pilot studieswith approximately thirty users each, across 2-3 conditions.Our pilot studies helped us to finalize remaining design de-cisions surrounding the standardized short table, refine ourquestionnaire, and test the integration of Surveyor’s Pointwith Mechanical Turk.2
We then conducted our large-scale study and completed theanalysis with 764 participants (409 female, 355 male), ran-domly assigned to five conditions (see Table 1): full policytext, standardized table, standardized short table, standard-ized short text, and layered text. We dropped 25 additionalparticipants from the study prior to analysis due to incom-plete data or for completing the study in an amount of timethat indicated inadequate attention to the task (defined astime on task that was two standard deviations lower than themean). We chose a between-subjects design to remove learn-ing effects and ensure the study could be completed withinabout 15 minutes. Participants in each condition followedthe same protocol; only the policy format differed.
PoliciesWe selected policies for the study from companies that con-sumers would conceivably interact with. We narrowed our
2The two systems are linked using a shared key that Surveyor’sPoint generates on the completion of our survey, which a participantthen enters back into Mechanical Turk. This allows us to link anentry in Mechanical Turk with an entry in Surveyor’s Point andverify the worker completed the survey before payment.
3
Standardization
3
Figure 1. An example of a standardized table is shown on the left, and a standardized short table on the right. The comparison highlights the rows
deleted to “shorten” this version. These deleted rows are listed directly below the table. While both formats contain the legend (bottom right), it is
displayed only on the right here due to space constraints.
have already been deployed by major corporations, makingthem a viable, real world summary format for privacy poli-cies. These policies were stripped of brand information, butthe formatting and styles were retained.
METHODOLOGYWe conducted an online user study in summer 2009 usingAmazon’s Mechanical Turk and a tool we developed calledSurveyor’s Point. Mechanical Turk offers workers the abil-ity to perform short tasks and get compensated. People canplace jobs through Mechanical Turk, specifying the numberof workers they are looking for, necessary qualifications, theamount they are willing to pay, and details about the task.Mechanical Turk payments are generally calibrated for thelength of the task. For our approximately 15-minute study,we paid $0.75 on successful completion.
We developed a custom survey-management tool called Sur-veyor’s Point to facilitate our data collection. Our implemen-tation allows us to show respondents a single question on thescreen along with links for switching back and forth betweentwo policies within a single browser window. This allowedus to track the number of users who looked at each policyand the number of times they switched between them. Addi-tionally, Surveyor’s Point allowed us to collect the amount oftime that users spent reading the policies, as well as informa-tion about whether they clicked through to opt-out forms, toadditional policy information links, or from a layered noticethrough to the full text policy.
In preparation for this study we first performed three smallerpilot tests of our survey framework. We ran our pilot studieswith approximately thirty users each, across 2-3 conditions.Our pilot studies helped us to finalize remaining design de-cisions surrounding the standardized short table, refine ourquestionnaire, and test the integration of Surveyor’s Pointwith Mechanical Turk.2
We then conducted our large-scale study and completed theanalysis with 764 participants (409 female, 355 male), ran-domly assigned to five conditions (see Table 1): full policytext, standardized table, standardized short table, standard-ized short text, and layered text. We dropped 25 additionalparticipants from the study prior to analysis due to incom-plete data or for completing the study in an amount of timethat indicated inadequate attention to the task (defined astime on task that was two standard deviations lower than themean). We chose a between-subjects design to remove learn-ing effects and ensure the study could be completed withinabout 15 minutes. Participants in each condition followedthe same protocol; only the policy format differed.
PoliciesWe selected policies for the study from companies that con-sumers would conceivably interact with. We narrowed our
2The two systems are linked using a shared key that Surveyor’sPoint generates on the completion of our survey, which a participantthen enters back into Mechanical Turk. This allows us to link anentry in Mechanical Turk with an entry in Surveyor’s Point andverify the worker completed the survey before payment.
3
Standardization
3
Summarization
Figure 1. An example of a standardized table is shown on the left, and a standardized short table on the right. The comparison highlights the rows
deleted to “shorten” this version. These deleted rows are listed directly below the table. While both formats contain the legend (bottom right), it is
displayed only on the right here due to space constraints.
have already been deployed by major corporations, makingthem a viable, real world summary format for privacy poli-cies. These policies were stripped of brand information, butthe formatting and styles were retained.
METHODOLOGYWe conducted an online user study in summer 2009 usingAmazon’s Mechanical Turk and a tool we developed calledSurveyor’s Point. Mechanical Turk offers workers the abil-ity to perform short tasks and get compensated. People canplace jobs through Mechanical Turk, specifying the numberof workers they are looking for, necessary qualifications, theamount they are willing to pay, and details about the task.Mechanical Turk payments are generally calibrated for thelength of the task. For our approximately 15-minute study,we paid $0.75 on successful completion.
We developed a custom survey-management tool called Sur-veyor’s Point to facilitate our data collection. Our implemen-tation allows us to show respondents a single question on thescreen along with links for switching back and forth betweentwo policies within a single browser window. This allowedus to track the number of users who looked at each policyand the number of times they switched between them. Addi-tionally, Surveyor’s Point allowed us to collect the amount oftime that users spent reading the policies, as well as informa-tion about whether they clicked through to opt-out forms, toadditional policy information links, or from a layered noticethrough to the full text policy.
In preparation for this study we first performed three smallerpilot tests of our survey framework. We ran our pilot studieswith approximately thirty users each, across 2-3 conditions.Our pilot studies helped us to finalize remaining design de-cisions surrounding the standardized short table, refine ourquestionnaire, and test the integration of Surveyor’s Pointwith Mechanical Turk.2
We then conducted our large-scale study and completed theanalysis with 764 participants (409 female, 355 male), ran-domly assigned to five conditions (see Table 1): full policytext, standardized table, standardized short table, standard-ized short text, and layered text. We dropped 25 additionalparticipants from the study prior to analysis due to incom-plete data or for completing the study in an amount of timethat indicated inadequate attention to the task (defined astime on task that was two standard deviations lower than themean). We chose a between-subjects design to remove learn-ing effects and ensure the study could be completed withinabout 15 minutes. Participants in each condition followedthe same protocol; only the policy format differed.
PoliciesWe selected policies for the study from companies that con-sumers would conceivably interact with. We narrowed our
2The two systems are linked using a shared key that Surveyor’sPoint generates on the completion of our survey, which a participantthen enters back into Mechanical Turk. This allows us to link anentry in Mechanical Turk with an entry in Surveyor’s Point andverify the worker completed the survey before payment.
3
Standardization
3
Summarization
Challenges
one size fits all
user education
4
Privacy Choice: Current State
fragmented ecosystem
difficult to find
4
Privacy Choice: Current State
5
Conversation-first Interfaces
The Rise of Conversational UI
6
PriBots: Conversational Privacy Bots
Message|
7
PriBots: Conversational Privacy Bots
Message|
7
PriBots: Conversational Privacy Bots
Message|
Appeal to new tech adopters
7
PriBots: Conversational Privacy Bots
Message|
Appeal to new tech adopters
Appeal to existing users
7
PriBots: Conversational Privacy Bots
Message|
Appeal to new tech adopters
Appeal to existing users
An intuitive way to 1. communicate privacy policies 2. adjust privacy preferences
7
1- Communicating Privacy Policies
8
Channel
9
Channel
Primary9
Channel
Primary Secondary9
Timing
10
Timing
At-setup
10
Timing
At-setup On-demand
10
Feedback
implicit: sentiment analysis
explicit: structured messages
gathering users’ concerns
11
Voicing User Concerns
Providers traditionally
say what they want
12
Voicing User Concerns
Providers traditionally
say what they want
Users’ concerns might not
be covered
12
Voicing User Concerns
Providers traditionally
say what they want
Users’ concerns might not
be covered
PriBots activate the
two-way channel
12
2- Setting Privacy Preferences
13
14
Service and platform-dependent interface
14
Service and platform-dependent interface Tradeoffs for simplicity: try finding this setting on Mobile Web version
14
15
Unique interface with all functionalities Ability to suggest adjustments to the user (combining notice and choice/preferences )
15
User Input
Analysis & Classification
System Architecture
16
User Input
Analysis & Classification
Query Structured Query
Statement
Yes
No
System Architecture
16
User Input
Analysis & Classification
Query Structured Query
Statement
Yes
No
Retrieval Module
Result
Knowledge Base
System Architecture
16
User Input
Analysis & Classification
Query Structured Query
Statement
Yes
No
Confident?Answer
Formulation in NL
Fallback Answer
Generation
Yes
No
Retrieval Module
Result
Knowledge Base
System Architecture
16
PriBot Reply
User Input
Analysis & Classification
Query Structured Query
Statement
Yes
No
Confident?Answer
Formulation in NL
Fallback Answer
Generation
Yes
No
Retrieval Module
Result
Knowledge Base
System Architecture
16
PriBot Reply
User Input
Analysis & Classification
Query Structured Query
Statement
Yes
No
Confident?Answer
Formulation in NL
Fallback Answer
Generation
Yes
No
Retrieval Module
Result
Knowledge Base
System Architecture
16
Feedback DB
AnalyticsAmendments/ Improvements
Augment the Knowledge Base
• unanswered queries • frequent questions • user sentiments
Feedback DB
Challenges
17
Mature User Understanding Text processing
Question answering
Domain-specific datasets and ontologies
Graceful fallback
18
Legal Challenges
Inherently error prone: are they legally binding?
Accounting for false-positives and false-negatives
The case of 3rd party PriBots: defamation possibilities?
19
Trusting the Machine rule-based vs. AI-based
user backlash?
regulate the confidence level
20
PriBots’ Personality
positive tone → higher trust
diversified content → reduced habituation
21
Deployment
22
provider
3rd parties
Deployment
22
provider
3rd parties
Deployment
22
Suitable for Voice Assistants
Rule-based Prototype
System Implementation
User studies
What’s Next?
23
Privacy as a Dialogue
Image/Media Credits Zara Picken: slide 12 Egor Kosten: slide 24 Alex Prokhoda: slide 6 Freepik: slide 23 Geoff Keough: slide 14 Victor: slide 12
