Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew...
Transcript of Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew...
![Page 1: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/1.jpg)
Embeddings and UncertaintyOct. 4, 2019
Andrew Gallagher [email protected]
Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff, Kevin Murphy, Mike Mozer, Caroline Pantofaru, Michael Nechyba, Cusuh Ham, Zhou Zhou, Abinash Behera
![Page 2: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/2.jpg)
Andrew Gallagher
longing for certainty ... is in every human mind. But certainty generally is illusion.
Oliver Wendell Holmes
Bertrand Russell
![Page 3: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/3.jpg)
Andrew Gallagher
Would you recognize this person if you needed to?
![Page 4: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/4.jpg)
Andrew Gallagher
What about one of these?
Bulthoff and Edelman, Psychophysical support for a two-dimensional view interpolation theory of object recognition, 1992. (link)
![Page 5: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/5.jpg)
Andrew Gallagher
How about this?
Bulthoff, Object Recognition in Man and Machine (link)
![Page 6: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/6.jpg)
Andrew Gallagher
How about this?
Bulthoff, Object Recognition in Man and Machine (link)
![Page 7: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/7.jpg)
Andrew Gallagher
Introduction
Embeddings
Uncertainty Representations (2)
Discussion
Andrew Gallagher WNYIPW2019 Google
![Page 8: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/8.jpg)
Andrew Gallagher
Introduction
Embeddings
Uncertainty Representations (2)
Discussion
Andrew Gallagher WNYIPW2019 Google
![Page 9: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/9.jpg)
Anchor
Positive
Negative
Anchor
Positive
NegativeLearn
Margin
FaceNet
slide content: F. Schroff and D. Kalenichenko
Encourages a margin between the ||anchor - positive|| and ||anchor - negative||.
F. Schroff, D. Kalenichenko, J. Philbin, FaceNet: A Unified Embedding for Face Recognition and Clustering https://arxiv.org/abs/1503.03832
![Page 10: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/10.jpg)
Anchor
Positive
Negative
Anchor
Positive
NegativeLearn
Margin
FaceNet
slide content: F. Schroff and D. Kalenichenko
![Page 11: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/11.jpg)
Training
![Page 12: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/12.jpg)
Training
![Page 13: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/13.jpg)
?
Testing
![Page 14: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/14.jpg)
Testing
![Page 15: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/15.jpg)
Andrew Gallagher
Introduction
Embeddings
Uncertainty Representations (1)
Discussion
Andrew Gallagher WNYIPW2019 Google
![Page 16: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/16.jpg)
Andrew Gallagher
Are two images the same class or not?
We typically employ:
Seong Joon Oh, Kevin Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, Andrew Gallagher, Modeling Uncertainty with Hedged Instance Embedding, https://arxiv.org/abs/1810.00319
![Page 17: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/17.jpg)
Andrew Gallagher
Commonly done: Point embedding.
true class: 17
true class: 47
![Page 18: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/18.jpg)
Andrew Gallagher
Commonly done: Point embedding.
true class: 17
true class: 47
class: ?
![Page 19: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/19.jpg)
Andrew Gallagher
Commonly done: Point embedding.
true class: 17
true class: 47
class: 47?
![Page 20: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/20.jpg)
Andrew Gallagher
Commonly done: Point embedding.
true class: 17
true class: 47
class: ?
![Page 21: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/21.jpg)
Andrew Gallagher
Commonly done: Point embedding.
true class: 17
true class: 47
class: 17?
![Page 22: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/22.jpg)
Andrew Gallagher
Commonly done: Point embedding.
true class: 17
true class: 47
class: ?
?
![Page 23: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/23.jpg)
Andrew Gallagher
Commonly done: Point embedding.
true class: 17
true class: 47
class: ?
?
![Page 24: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/24.jpg)
Andrew Gallagher
Commonly done: Point embedding.
true class: 17
true class: 47
class: ?
?
![Page 25: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/25.jpg)
Andrew Gallagher
Hedged Instance Embeddings
true class: 17
true class: 47
class: ?
![Page 26: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/26.jpg)
Andrew Gallagher
true class: 17
true class: 47
class: ?
![Page 27: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/27.jpg)
Andrew Gallagher
Metric embedding as distributions.Formulate as discriminative task: given pair, predict match.
Introduce embedding as latent variable:
Embedding distributions must be:parameterizablesampleable
![Page 28: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/28.jpg)
Andrew Gallagher
Match probability estimation.
![Page 29: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/29.jpg)
Andrew Gallagher
Computing
![Page 30: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/30.jpg)
Andrew Gallagher
Computing
![Page 31: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/31.jpg)
Andrew Gallagher
Computing
![Page 32: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/32.jpg)
Andrew Gallagher
Computing
![Page 33: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/33.jpg)
Andrew Gallagher
sigmoid matchlikelihood
Computing
![Page 34: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/34.jpg)
Andrew Gallagher
Computing
sigmoid matchlikelihood
Scratch Demo
![Page 35: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/35.jpg)
Andrew Gallagher
End-to-end differentiable with reparametrization trick.
Shared params
![Page 36: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/36.jpg)
Andrew Gallagher
MoG embedding with multiple samples case.
Shared params
![Page 37: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/37.jpg)
Andrew Gallagher
Training objective - Variational Information Bottleneck
Derived from the Variational Information Bottleneck [from Bayesflow team]:
Log likelihood term: Binary cross-entropy loss for match prediction.
KL term: Controls the compression level for z. Unit Gaussian.
![Page 38: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/38.jpg)
Andrew Gallagher
We desire a measure that, given input , one can guess its performance on downstream tasks (e.g., verification, recognition).
Uncertainty measure: Self-mismatch.
![Page 39: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/39.jpg)
Andrew Gallagher
uncertain certainsomewhat uncertain
MoG-1 MoG-1MoG-2
Uncertainty measure: Self-mismatch.We desire a measure that, given input , one can guess its performance on downstream tasks (e.g., verification, recognition).
![Page 40: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/40.jpg)
Andrew Gallagher
Dataset and Experiments
Andrew Gallagher WNYIPW2019 Google
![Page 42: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/42.jpg)
Andrew Gallagher
Experimental setup.
![Page 43: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/43.jpg)
Andrew Gallagher
Clean images Corrupt images
2 Digit MNIST → 2 dimension Hedged Instance Embeddings (MoG-1)
![Page 44: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/44.jpg)
Andrew Gallagher
Clean images Corrupt images
3 Digit MNIST → 3 dimension Hedged Instance Embeddings (MoG-1)
![Page 45: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/45.jpg)
Andrew Gallagher
2 Digit → 2 Dim
Hedged Instance Embeddings (MoG-1)
![Page 46: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/46.jpg)
Andrew Gallagher
Task Performance
![Page 47: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/47.jpg)
Andrew Gallagher
Task Performance
![Page 48: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/48.jpg)
Andrew Gallagher
Most certain Least certain
![Page 49: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/49.jpg)
Andrew Gallagher
Uncertainty Measure
![Page 50: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/50.jpg)
Andrew Gallagher
PetNet with Uncertainty
20-dim embedding0.25 MobileNet
High Certainty Samples Low Certainty Samples
![Page 51: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/51.jpg)
Andrew Gallagher
Uncertain Pet *
roger901
Certain Pet *
unbunt
* Not the actual photos, but similar.
![Page 52: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/52.jpg)
Andrew Gallagher
Introduction
Embeddings
Uncertainty Representations (2)
Discussion
Andrew Gallagher WNYIPW2019 Google
![Page 53: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/53.jpg)
Andrew Gallagher
Deep Convolutional Neural Network Features and the Original Image
![Page 54: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/54.jpg)
Andrew Gallagher
Why is L2-Norm Useful?
![Page 55: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/55.jpg)
Andrew Gallagher
Probabilistic Face Embeddings, Shi & Jain, 2019, https://arxiv.org/abs/1904.09658.
![Page 56: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/56.jpg)
Fashion MNIST
MNIST
Datasets
![Page 57: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/57.jpg)
Architecture50
D
10normalization
![Page 58: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/58.jpg)
![Page 59: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/59.jpg)
D=3, normalized, cross-entropy loss
![Page 60: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/60.jpg)
D=4, normalized, triplet loss
![Page 61: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/61.jpg)
Andrew Gallagher
Staying close to home ...
![Page 62: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/62.jpg)
Andrew GallagherFa
ceN
et e
mbe
ddin
g
penu
ltim
ate
embe
ddin
g
feature extraction
norm
aliz
e
L2
sigmoid embedding confidence
Embedding
![Page 63: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/63.jpg)
Andrew GallagherFa
ceN
et e
mbe
ddin
g
penu
ltim
ate
embe
ddin
g
Embedding
feature extraction
norm
aliz
e
L2
sigmoid embedding confidence
Distilled Embedding Confidence
distilled embedding confidence
![Page 64: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/64.jpg)
Andrew Gallagher
Recognizable Not Recognizable0.123 0.0860.1640.2700.3200.3980.6180.630 0.4570.8650.935
![Page 65: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/65.jpg)
Andrew Gallagher
Introduction
Embeddings
Uncertainty Representations (2)
Discussion
Andrew Gallagher WNYIPW2019 Google
![Page 66: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/66.jpg)
Andrew Gallagher
Real-world applications
Andrew Gallagher WNYIPW2019 Google
![Page 67: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/67.jpg)
Andrew Gallagher
![Page 68: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/68.jpg)
Andrew Gallagher
A boring image.
![Page 69: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/69.jpg)
Andrew Gallagher
A corrupted image.
![Page 70: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/70.jpg)
Andrew Gallagher
![Page 71: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/71.jpg)
ApplicationsGallery Selection: Given a potentially large collection of images or a video. Select a subset of images/frames that will most reliably identify the individual.
Computational Efficiency: Prevent running the expensive models on images with high uncertainty.
Tracking: Use as the confidence score.
Offline Face Clustering: Confidence weighted cluster similarity instead of top-N or other heuristic based approaches.
![Page 72: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/72.jpg)
Andrew Gallagher
Open Questions
Andrew Gallagher WNYIPW2019 Google
![Page 73: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/73.jpg)
Andrew Gallagher
Open Questions
How best to quantify and measure uncertainty?
Andrew Gallagher WNYIPW2019 Google
![Page 74: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/74.jpg)
Andrew Gallagher
Open Questions
Tell us when you don’t know.
But only when absolutely necessary.
Andrew Gallagher WNYIPW2019 Google
![Page 75: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/75.jpg)
Andrew Gallagher
Open Questions
How best to compute, index, and match uncertain embeddings (at scale)?
Andrew Gallagher WNYIPW2019 Google
![Page 76: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/76.jpg)
Andrew Gallagher
Open Questions
What network architectures encourage learning uncertainty?
Andrew Gallagher WNYIPW2019 Google
![Page 77: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/77.jpg)
Andrew Gallagher
Open Questions
Open world (epistemic) uncertainty.
Andrew Gallagher WNYIPW2019 Google
![Page 80: Embeddings and Uncertainty · 2019-10-07 · Embeddings and Uncertainty Oct. 4, 2019 Andrew Gallagher agallagher@google.com Seong Joon Oh, Joseph Roth, Jiyan Pan, Florian Schroff,](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f49b6960ada2537ae6b8364/html5/thumbnails/80.jpg)
Andrew Gallagher
Thank you
Andrew Gallagher WNYIPW2019 Google