Learning Adversarially Fair and Transferrable...
Transcript of Learning Adversarially Fair and Transferrable...
Learning Adversarially Fair and TransferrableRepresentations
David Madras\[| {z }me
Elliot Creager\[ Toniann Pitassi\[ Richard Zemel\[
\University of Toronto
[Vector Institute
July 13, 2018
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 1 / 18
Introduction
Classification: a tale of two parties
Example: targeted advertising: owner ! vendor ! prediction
Data owner Prediction vendor
[Dwork et al., 2012]Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 2 / 18
Why Fairness?
Want to minimize unfair targeting of disadvantaged groups byvendors
e.g. showing ads for worse lines of credit, lower paying jobs
We want fair predictions
Data owner Prediction vendor
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 3 / 18
Why Fair Representations?
Previous work emphasized the role of the vendor
Can we trust the vendor?
How can the owner ensure fairness?
Data owner Prediction vendor
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 4 / 18
The Data Owner
How should the data be represented?Feature selection? Measurement?
How can we choose a data representation that ensures fairclassifications downstream?
Let’s learn a fair representation!
Data owner!Representation learner
[Zemel et al., 2013]Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 5 / 18
Background: Fair Classification
Assume: data X 2 Rd , label Y 2 0, 1, sensitive attribute A 2 0, 1Goal: predict Y fairly with respect to A
Demographic parity
P(Y = 1|A = 0) = P(Y = 1|A = 1)
Equalized odds
P(Y 6= Y |A = 0,Y = y) = P(Y 6= Y |A = 1,Y = y) 8y 2 {0, 1}
Equal opportunity: equalized odds with only Y = 1
P(Y 6= Y |A = 0,Y = 1) = P(Y 6= Y |A = 1,Y = 1)
[Dwork et al., 2012] [Hardt et al., 2016]Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 6 / 18
Goals of Fair Representation Learning
Fair classification: learn X
f! Z
g! Y
encoder f , classifier g
Fair representation: learn X
f! Z
g! Y
Z = f (X ) should:Maintain useful information in X
Yield fair downstream classification for vendors g
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 7 / 18
Types of unfair vendors
Consider two types of unfair vendorsThe indi↵erent vendor: doesn’t care about fairness, only maximizesutilityThe malicious vendor: doesn’t care about utility, discriminatesmaximally
This suggests an adversarial learning scheme
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 8 / 18
Learning Adversarially Fair Representations
AZY
X
Encoderf (X )
Decoderk(Z ,A)
Classifierg(Z )
Adversaryh(Z )
The classifier is the indi↵erent vendor, forcing the encoder to makethe representations useful
The adversary is the malicious vendor, forcing the encoder to hide thesensitive attributes in the representations
[Edwards and Storkey, 2015]Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 9 / 18
Adversarial Learning in LAFTR
AZY
X
Encoderf (X )
Decoderk(Z ,A)
Classifierg(Z )
Adversaryh(Z )
Our game: encoder-decoder-classifier vs. adversary
Goal: learn a fair encoder
minimizef ,g ,k
maximizeh
EX ,Y ,A [L(f , g , h, k)] .
L(f , g , h, k) = ↵LClass + �LDec � �LAdv
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 10 / 18
Adversarial Objectives
AZY
X
Encoderf (X )
Decoderk(Z ,A)
Classifierg(Z )
Adversaryh(Z )
Choice of adversarial objective depends on fairness desideratum
Demographic parity: LDPAdv (h) =
Pi2{0,1}
1
|Di |P
(x ,a)2Di|h(f (x)) � a|
Equalized odds: LEOAdv (h) =
Pi ,j2{0,1}2
1
|Dji |
P(x ,a,y)2Dj
i|h(f (x), y)� a|
Equal Opportunity: LEOppAdv (h) =
Pi2{0,1}
1
|D1
i |P
(x ,a)2D1
i|h(f (x)) � a|
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 11 / 18
From Adversarial Objectives to Fairness Definitions
In general: pick the right adversarial loss, encourage the right conditionalindependencies
Demographic parity encourages Z ? A to fool adversary
Equalized odds encourages Z ? A | Y to fool adversary
Equal opportunity encourages Z ? A | Y = 1 to fool adversary
Note that independencies of Z = f (x) also hold for predictions Y = g(Z )
We show: In the adversarial limit, these objectives guarantee thesefairness metrics!
The key is to connect predictability of A by the adversary h(Z ) tounfairness in the classifier g(Z )
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 12 / 18
Theoretical Properties
Define �DP(g) , DP-unfairness of classifier g
Define LDPAdv (h) , adversarial loss (inv. weighted error)
We show: 8 classifier g(Z ), we can construct an adversary h(Z ) s.t.�LDP
Adv (h) = �DP(g)
Let h? be the optimal adversary. Then
�LDPAdv (h
?) � �LDPAdv (h) = �DP (1)
Takeaway: if �LDPAdv (h
?) is forced to be small, �DP will be too
Holds for EO as well, but with h as a function of Y also
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 13 / 18
Results - Fair Classification (Adult)
0.025 0.050 0.075 0.100 0.125 0.150 0.175�DP
0.80
0.81
0.82
0.83
0.84
0.85
Acc
urac
y
LAFTR-DP
LAFTR-EO
LAFTR-EOpp
DP-CE
MLP-Unfair
0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16�EO
0.838
0.840
0.842
0.844
0.846
0.848
0.850
Acc
urac
y
LAFTR-DP
LAFTR-EO
LAFTR-EOpp
MLP-Unfair
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08�EOpp
0.836
0.838
0.840
0.842
0.844
0.846
0.848
0.850
Acc
urac
y
LAFTR-DP
LAFTR-EO
LAFTR-EOpp
MLP-Unfair
Train with two-step method to simulate owner ! vendor framework
Tradeo↵s between accuracy and various fairness metrics yielded bydi↵erent LAFTR loss functions
Seems to work best for fairest solutions
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 14 / 18
Setup - Fair Transfer Learning
Downstream vendors will have unknown prediction tasks
Does fairness transfer?
We test this as follows:1 Train encoder f on data X , with label Y2 Freeze encoder f3 On new data X
0, train classifier on top of f (X 0), with new task label Y 0
4 Observe fairness and accuracy of this new classifier on new task Y
0
Compare LAFTR encoder f to other encoders
We use Heritage Health datasetY is Charlson comorbidity index > 0Y
0 is whether or not a certain type of insurance claim was madeCheck for fairness w.r.t. age
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 15 / 18
Results - Fair Transfer Learning
Transfer-Unf Transfer-Fair Transfer-Y-Adv LAFTR
�0.4
�0.3
�0.2
�0.1
0.0
0.1
Rel
ativ
eD
i�er
ence
toba
selin
e(T
arge
t-U
nfai
r)
Error�EO
Figure 2: Fair transfer learning on Health dataset. Down is better in both metrics.
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 16 / 18
Conclusion
Propose LAFTR: general model for fair representation learning
Connect common fairness metrics to adversarial objectives
Demonstrate that training with LAFTR improves transfer fairness
Open questions:Compare adversarial/non-adversarial methods?Transfer fairness: datasets, limitations, better methods?
Come check out our poster #44 tonight!
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 17 / 18
References
Dwork, C., M. Hardt, T. Pitassi, O. Reingold, and R. Zemel (2012).Fairness through awareness. In Proceedings of the 3rd innovations in
theoretical computer science conference, pp. 214–226. ACM.
Edwards, H. and A. Storkey (2015). Censoring representations with anadversary. arXiv preprint arXiv:1511.05897 .
Hardt, M., E. Price, N. Srebro, et al. (2016). Equality of opportunity insupervised learning. In Advances in neural information processing
systems, pp. 3315–3323.
Zemel, R., Y. Wu, K. Swersky, T. Pitassi, and C. Dwork (2013). Learningfair representations. In International Conference on Machine Learning,pp. 325–333.
Madras et al. 2017 (arxiv:1802.06309) LAFTR: Poster #44 July 13, 2018 18 / 18