Probabilistic Meta-Representations Of Neural Networksbalaji/udl-camera-ready/UDL-13.pdf ·...

Probabilistic Meta-Representations Of Neural Networks

Theofanis Karaletsos1, Peter Dayan1,2,Zoubin Ghahramani1,3

1 Uber AI Labs, San Francisco, CA, USA2 University College London, United Kingdom

3 Department of Engineering, University of Cambridge, United [email protected], [email protected],

[email protected]

Abstract

Existing Bayesian treatments of neural net-works are typically characterized by weak priorand approximate posterior distributions accord-ing to which all the weights are drawn indepen-dently. Here, we consider a richer prior distri-bution in which units in the network are repre-sented by latent variables, and the weights be-tween units are drawn conditionally on the val-ues of the collection of those variables. This al-lows rich correlations between related weights,and can be seen as realizing a function priorwith a Bayesian complexity regularizer ensur-ing simple solutions. We illustrate the resultingmeta-representations and representations, eluci-dating the power of this prior.

1 IntroductionWe propose a higher-level abstraction of neural networkweight representations in which the units of the networkare probabilistically embedded into a shared, structured,meta-representational space (generalizing topographic lo-cation), with weights and biases being derived conditionalon these embeddings. Rich structural patterns can thusbe induced into the weight distributions. Our model cap-tures uncertainty on three levels: meta-representationaluncertainty, function uncertainty given the embeddings,and observation (or irreducible output) uncertainty. Thishierarchical decomposition is flexible, and is broadlyapplicable to modeling task-appropriate weight priors,weight-correlations, and weight uncertainty. It can alsobe beneficially used in the context of various modern ap-plications involving out of sample generalization, wherethe ability to perform structured weight manipulationsonline is beneficial 1 .

1Due to space concerns we provide more background andrelated work in the Appendix Sec. 8

2 Meta-Representations of Units

We suggest abandoning direct characterizations ofweights or distributions over weights, in which weightsare individually independently tunable. Instead, we cou-ple weights using meta-representations (so called, sincethey determine the parameters of the underlying NN thatthemselves govern the input-output function representedby the NN). These treat the units as the primary objectsof interest and embed them into a shared space, derivingweights as secondary structures.

Consider a code zl,u ∈ <D that uniquely describes eachunit u (visible or hidden) in layer l in the network. Suchcodes could for example be one-hot codes or Euclideanembeddings of units in a real space <K . A generalizationis to use an inferred latent representation which embedsunits l, u in a D-dimensional vector space. Note that thiscode encodes the unit itself, not its activation.

Weighs wl,i,j linking two units can then be recast in termsof those units’ codes zl,i and zl−1,j for instance by con-catenation zw(l, i, j) =

[zl,i, zl−1,j

]. We call the collec-

tion of all such weight codes Zw (which can be deter-ministically derived from the collection of unit codes Z).Biases b can be constructed similarly, for instance using0’s as the second code; thus we do not distinguish thembelow. Weight codes then form a conditional prior distri-bution P (wl,i,j |zw(l, i, j)), parameterized by a functiong(zw(l, i, j), ξ) shared across the entire network. Func-tion g, which may itself have parameters ξ, acts as aconditional hyperprior that gives rise to a prior over theweights of the original network:

P (W|Zw; ξ)=

L∏l=1

Vl∏i=1

Vl−1∏j=1

P (wl,i,j |zw(l, i, j); ξ).

There remain various choices: we commonly use eitherGaussian or implicit models for the weight prior and neu-ral networks as hyperpriors (though clustering and Gaus-sian Processes merit exploration). Further, the weight

mailto:[email protected]

[email protected]

mailto:[email protected]

code can be augmented with a global state variable zs(making zw(l, i, j) =

[zl,i, zl−1,j , zs

]) which can coor-

dinate all the weights or add conditioning knowledge.

3 MetaPrior: A Generative Meta-Modelof Neural Network Weights

Figure 1: A graphical modelshowing the MetaPrior. Note theadditional plate over the meta-variables Z indicating the distri-bution over meta-representations.

The meta-representationcan be used as aprior for BayesianNNs. Consider acase with latentcodes for unitsbeing sampledfrom P (Z) =∏

l,i P (zl,i), andthe weights of theunderlying NNbeing sampledaccording to theconditional dis-tribution for theweights P (W|Zw).Put together thisyields the model:

P (y|x) =

∫Z

P (Z)

∫WP (y|x,W)P (W|Z)dWdZ

P (Z) =∏

l,iP (zl,i) =

∏l,i

N (0,1).

Crucially, the conditional distribution over weights de-pends on more than one unit representation. This canbe seen as a structural form of weight-sharing or a func-tion prior and is made more explicit using the plate no-tation in Fig 1. Conditioned on a set of sampled vari-ables Z, our model defines a particular space of functionsfZ : X → Y. We can recast the predictive distributiongiven training data D∗ as:

P (y|x,D∗) =

∫Z

Q(Z|D∗)∫fZ

P (y|x, fZ)P (fZ|Z)dfZdZ.

Uncertainty about the unit embeddings affords the modelthe flexibility to represent diverse functions by couplingweight distributions with meta-variables.

The learning task for training MetaPriors for a particulardataset D consists of inferring the posterior distributionof the latent variables P (Z|D). Compared with the typ-ical training loop in Bayesian Neural Networks, whichinvolves learning a posterior distribution over weights,the posterior distribution that needs to be inferred forMetaPriors is the approximate distribution Q(Z; Φ) over

the collection of unit variables as the model builds meta-models of neural networks. We train by maximizing theevidence lower bound (ELBO):

logP (D; ξ) ≥ EQ(Z)

[log

P (D|Z; ξ)P (Z)

Q(Z; Φ)

](1)

with logP (D|Z) ≈ 1S

S∑s=1

P (y|x,Ws,m) and Ws,m ∼

P (W|Zm). In practice, we apply the reparametrizationtrick (15; 33; 39) and its variants (14) and subsampleZm ∼ Q(Z; Φ) to maximize the objective:

ELBO(Φ, ξ) =1

M

M∑m=1

[ 1

S

S∑s=1

logP (y|x,Ws,m)]

− KL(Q(Z; Φ)||P (Z)

).

(2)

However, pure posterior inference over Φ withoutgradient steps to update ξ results in learning meta-representations which best explain the data given thecurrent hypernetwork with parameters ξ. We call theprocess of inferring representations P (Z|D) illation. In-tuitively, inferring the meta-representations for a datasetinduces a functional alignment to its input-output pairsand thus reduces variance in the marginal representationsfor a particular collection of data points.

We can also recast this as a two-stage learning proce-dure, similar to Expectation Maximization (EM), if wewant to update the function parameters ξ and the meta-variables independently. First, we approximate P (Z|D; ξ)by Q(Z;φ) by maximizing ELBO(φ; ξ). Then, we canmaximize of ELBO(ξ;φ) to update the hyperprior func-tion. In practice, we find that Eq. 2 performs well with asmall amount of samples for learning, but using EM canhelp reduce gradient variance in the small-data setting.

4 ExperimentsWe illustrate the properties of MetaPriors with a seriesof tasks of increasing complexity, starting with simpleregression and classification, and then graduating to few-shot learning.

4.1 Toy Example: RegressionHere, we consider the toy regression task popularizedin (10) ( y = x3 + ε with ε ∼ N (0, 3)). We use neu-ral networks with a fixed observation noise given by themodel, and seek to learn suitably uncertain functions. Forall networks, we use 100 hidden units, and 2-dimensionallatent codes and 32 hidden units in the hyper-prior net-work f .

In Fig. 2 we show two function fits to this example: ourmodel and a mean field network. We observe that both

models increase uncertainty away from the data, as wouldbe expected. We also illustrate function draws whichshow how each model uses its respective weight repre-sentation differently to encode functions and uncertainty.We sample functions in both cases by clamping latentvariables to their posterior mean and allowing a singlelatent variable to be sampled from its prior. Our model,the MetaPrior-based Bayesian NN, learns a global formof function uncertainty and has global diversity over sam-pled functions for sampling just a single latent variable.It uses the composition of these functions through themeta-representational uncertainty to model the functionspace. This can be attributed to the strong complexitycontrol enacted by the pull of the posterior fitting mech-anism on the meta-variables. Maximum a posteriori fitsof this model yielded just the mean line directly fittingthe data. The mean field example shows dramatically lessdiversity in function samples, we were forced to sample alarge amount of weights to elicit the diversity we got assingle weight samples only induced small local changes.This suggests that the MetaPrior may be capturing inter-esting properties of the weights beyond what the meanfield approximation does, as single variable perturbationshave much more impact on the function space.

4.2 Toy Example: ClassificationWe illustrate the model’s function fit to the half-moontwo class classification task in Fig. 3, also visualizing thelearned weight correlations by sampling from the repre-sentation. The model reaches 95.5% accuracy, on parwith a mean field BNN and an MLP. Interestingly, meta-representations induce intra- and inter-layer correlationsof weights, amounting to a form of soft weight-sharingwith long-range correlations. This visualizes the mecha-nisms by which complex function draws as observed inSec. 4.1 are feasible with only a single variable chang-ing. The model captures structured weight correlationswhich enable global weight changes subject to a low-dimensional parametrization. This is a conceptual differ-ence to networks with individually tunable weights.

4.3 MNIST50k-ClassificationWe use NNs with one hidden layer and 100 hidden units totest the simplest model possible for MNIST-classification.We train and compare the deterministic one-hot embed-dings (Gaussian-OH), with the latent variable embeddings(Gaussian-LV) used elsewhere (with 10 latent dimen-sions); along with mean field NNs with a unit Gaussianprior (Gaussian-ML). We visualize the learned Zs (Fig. 4)by producing a T-SNE embedding of their means whichreveal relationships among the units in the network. Thefigure shows that the model infers semantic structure inthe input units, as it compresses boundary units to a simi-lar value. This is representationally efficient as no capac-

mean<latexit sha1_base64="2CX5lI2t6HimQe/54buIJp8wl0I=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe7SKFYBG8sI5gOTEPY2c8mS3b1jd08IR/6FjYUitv4bO/+Nm+QKTXww8Hhvhpl5YSK4sb7/7RU2Nre2d4q7pb39g8Oj8vFJy8SpZthksYh1J6QGBVfYtNwK7CQaqQwFtsPJ7dxvP6E2PFYPdppgX9KR4hFn1DrpMeuFEZFI1WxQrvhVfwGyToKcVCBHY1D+6g1jlkpUlglqTDfwE9vPqLacCZyVeqnBhLIJHWHXUUUlmn62uHhGLpwyJFGsXSlLFurviYxKY6YydJ2S2rFZ9ebif143tdF1P+MqSS0qtlwUpYLYmMzfJ0OukVkxdYQyzd2thI2ppsy6kEouhGD15XXSqlUDvxrc1yr1mzyOIpzBOVxCAFdQhztoQBMYKHiGV3jzjPfivXsfy9aCl8+cwh94nz9l8pCx</latexit><latexit sha1_base64="2CX5lI2t6HimQe/54buIJp8wl0I=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe7SKFYBG8sI5gOTEPY2c8mS3b1jd08IR/6FjYUitv4bO/+Nm+QKTXww8Hhvhpl5YSK4sb7/7RU2Nre2d4q7pb39g8Oj8vFJy8SpZthksYh1J6QGBVfYtNwK7CQaqQwFtsPJ7dxvP6E2PFYPdppgX9KR4hFn1DrpMeuFEZFI1WxQrvhVfwGyToKcVCBHY1D+6g1jlkpUlglqTDfwE9vPqLacCZyVeqnBhLIJHWHXUUUlmn62uHhGLpwyJFGsXSlLFurviYxKY6YydJ2S2rFZ9ebif143tdF1P+MqSS0qtlwUpYLYmMzfJ0OukVkxdYQyzd2thI2ppsy6kEouhGD15XXSqlUDvxrc1yr1mzyOIpzBOVxCAFdQhztoQBMYKHiGV3jzjPfivXsfy9aCl8+cwh94nz9l8pCx</latexit><latexit sha1_base64="2CX5lI2t6HimQe/54buIJp8wl0I=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe7SKFYBG8sI5gOTEPY2c8mS3b1jd08IR/6FjYUitv4bO/+Nm+QKTXww8Hhvhpl5YSK4sb7/7RU2Nre2d4q7pb39g8Oj8vFJy8SpZthksYh1J6QGBVfYtNwK7CQaqQwFtsPJ7dxvP6E2PFYPdppgX9KR4hFn1DrpMeuFEZFI1WxQrvhVfwGyToKcVCBHY1D+6g1jlkpUlglqTDfwE9vPqLacCZyVeqnBhLIJHWHXUUUlmn62uHhGLpwyJFGsXSlLFurviYxKY6YydJ2S2rFZ9ebif143tdF1P+MqSS0qtlwUpYLYmMzfJ0OukVkxdYQyzd2thI2ppsy6kEouhGD15XXSqlUDvxrc1yr1mzyOIpzBOVxCAFdQhztoQBMYKHiGV3jzjPfivXsfy9aCl8+cwh94nz9l8pCx</latexit><latexit sha1_base64="2CX5lI2t6HimQe/54buIJp8wl0I=">AAAB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe7SKFYBG8sI5gOTEPY2c8mS3b1jd08IR/6FjYUitv4bO/+Nm+QKTXww8Hhvhpl5YSK4sb7/7RU2Nre2d4q7pb39g8Oj8vFJy8SpZthksYh1J6QGBVfYtNwK7CQaqQwFtsPJ7dxvP6E2PFYPdppgX9KR4hFn1DrpMeuFEZFI1WxQrvhVfwGyToKcVCBHY1D+6g1jlkpUlglqTDfwE9vPqLacCZyVeqnBhLIJHWHXUUUlmn62uHhGLpwyJFGsXSlLFurviYxKY6YydJ2S2rFZ9ebif143tdF1P+MqSS0qtlwUpYLYmMzfJ0OukVkxdYQyzd2thI2ppsy6kEouhGD15XXSqlUDvxrc1yr1mzyOIpzBOVxCAFdQhztoQBMYKHiGV3jzjPfivXsfy9aCl8+cwh94nz9l8pCx</latexit>

MetaPrior<latexit sha1_base64="Rjn0MvSfNCm3QiTyngGfLDwd0dE=">AAAB+HicbVA9SwNBEJ2LXzF+JGppsxgEq3CXRrEK2NgIEcwHJEfY28wlS/Zuj909IR75JTYWitj6U+z8N26SKzTxwcDjvRlm5gWJ4Nq47rdT2Njc2t4p7pb29g8Oy5Wj47aWqWLYYlJI1Q2oRsFjbBluBHYThTQKBHaCyc3c7zyi0lzGD2aaoB/RUcxDzqix0qBSzvpBSO7Q0KbiUs0Glapbcxcg68TLSRVyNAeVr/5QsjTC2DBBte55bmL8jCrDmcBZqZ9qTCib0BH2LI1phNrPFofPyLlVhiSUylZsyEL9PZHRSOtpFNjOiJqxXvXm4n9eLzXhlZ/xOEkNxmy5KEwFMZLMUyBDrpAZMbWEMsXtrYSNqaLM2KxKNgRv9eV10q7XPLfm3derjes8jiKcwhlcgAeX0IBbaEILGKTwDK/w5jw5L86787FsLTj5zAn8gfP5A59nkwY=</latexit><latexit sha1_base64="Rjn0MvSfNCm3QiTyngGfLDwd0dE=">AAAB+HicbVA9SwNBEJ2LXzF+JGppsxgEq3CXRrEK2NgIEcwHJEfY28wlS/Zuj909IR75JTYWitj6U+z8N26SKzTxwcDjvRlm5gWJ4Nq47rdT2Njc2t4p7pb29g8Oy5Wj47aWqWLYYlJI1Q2oRsFjbBluBHYThTQKBHaCyc3c7zyi0lzGD2aaoB/RUcxDzqix0qBSzvpBSO7Q0KbiUs0Glapbcxcg68TLSRVyNAeVr/5QsjTC2DBBte55bmL8jCrDmcBZqZ9qTCib0BH2LI1phNrPFofPyLlVhiSUylZsyEL9PZHRSOtpFNjOiJqxXvXm4n9eLzXhlZ/xOEkNxmy5KEwFMZLMUyBDrpAZMbWEMsXtrYSNqaLM2KxKNgRv9eV10q7XPLfm3derjes8jiKcwhlcgAeX0IBbaEILGKTwDK/w5jw5L86787FsLTj5zAn8gfP5A59nkwY=</latexit><latexit sha1_base64="Rjn0MvSfNCm3QiTyngGfLDwd0dE=">AAAB+HicbVA9SwNBEJ2LXzF+JGppsxgEq3CXRrEK2NgIEcwHJEfY28wlS/Zuj909IR75JTYWitj6U+z8N26SKzTxwcDjvRlm5gWJ4Nq47rdT2Njc2t4p7pb29g8Oy5Wj47aWqWLYYlJI1Q2oRsFjbBluBHYThTQKBHaCyc3c7zyi0lzGD2aaoB/RUcxDzqix0qBSzvpBSO7Q0KbiUs0Glapbcxcg68TLSRVyNAeVr/5QsjTC2DBBte55bmL8jCrDmcBZqZ9qTCib0BH2LI1phNrPFofPyLlVhiSUylZsyEL9PZHRSOtpFNjOiJqxXvXm4n9eLzXhlZ/xOEkNxmy5KEwFMZLMUyBDrpAZMbWEMsXtrYSNqaLM2KxKNgRv9eV10q7XPLfm3derjes8jiKcwhlcgAeX0IBbaEILGKTwDK/w5jw5L86787FsLTj5zAn8gfP5A59nkwY=</latexit><latexit sha1_base64="Rjn0MvSfNCm3QiTyngGfLDwd0dE=">AAAB+HicbVA9SwNBEJ2LXzF+JGppsxgEq3CXRrEK2NgIEcwHJEfY28wlS/Zuj909IR75JTYWitj6U+z8N26SKzTxwcDjvRlm5gWJ4Nq47rdT2Njc2t4p7pb29g8Oy5Wj47aWqWLYYlJI1Q2oRsFjbBluBHYThTQKBHaCyc3c7zyi0lzGD2aaoB/RUcxDzqix0qBSzvpBSO7Q0KbiUs0Glapbcxcg68TLSRVyNAeVr/5QsjTC2DBBte55bmL8jCrDmcBZqZ9qTCib0BH2LI1phNrPFofPyLlVhiSUylZsyEL9PZHRSOtpFNjOiJqxXvXm4n9eLzXhlZ/xOEkNxmy5KEwFMZLMUyBDrpAZMbWEMsXtrYSNqaLM2KxKNgRv9eV10q7XPLfm3derjes8jiKcwhlcgAeX0IBbaEILGKTwDK/w5jw5L86787FsLTj5zAn8gfP5A59nkwY=</latexit>

MF-SVI<latexit sha1_base64="ctML9YmhATWPmVyBeW4LsKK9nQk=">AAAB83icbVBNSwMxEJ3Ur1q/qh69BIvgxbLbi+KpIIgehIr2A7pLyabZNjSbXZKsUJb+DS8eFPHqn/HmvzFt96CtDwYe780wMy9IBNfGcb5RYWV1bX2juFna2t7Z3SvvH7R0nCrKmjQWseoERDPBJWsabgTrJIqRKBCsHYyupn77iSnNY/loxgnzIzKQPOSUGCt5mReE+O767KF1O+mVK07VmQEvEzcnFcjR6JW/vH5M04hJQwXRuus6ifEzogyngk1KXqpZQuiIDFjXUkkipv1sdvMEn1ilj8NY2ZIGz9TfExmJtB5Hge2MiBnqRW8q/ud1UxNe+BmXSWqYpPNFYSqwifE0ANznilEjxpYQqri9FdMhUYQaG1PJhuAuvrxMWrWq61Td+1qlfpnHUYQjOIZTcOEc6nADDWgChQSe4RXeUIpe0Dv6mLcWUD5zCH+APn8Azt2Q1g==</latexit><latexit sha1_base64="ctML9YmhATWPmVyBeW4LsKK9nQk=">AAAB83icbVBNSwMxEJ3Ur1q/qh69BIvgxbLbi+KpIIgehIr2A7pLyabZNjSbXZKsUJb+DS8eFPHqn/HmvzFt96CtDwYe780wMy9IBNfGcb5RYWV1bX2juFna2t7Z3SvvH7R0nCrKmjQWseoERDPBJWsabgTrJIqRKBCsHYyupn77iSnNY/loxgnzIzKQPOSUGCt5mReE+O767KF1O+mVK07VmQEvEzcnFcjR6JW/vH5M04hJQwXRuus6ifEzogyngk1KXqpZQuiIDFjXUkkipv1sdvMEn1ilj8NY2ZIGz9TfExmJtB5Hge2MiBnqRW8q/ud1UxNe+BmXSWqYpPNFYSqwifE0ANznilEjxpYQqri9FdMhUYQaG1PJhuAuvrxMWrWq61Td+1qlfpnHUYQjOIZTcOEc6nADDWgChQSe4RXeUIpe0Dv6mLcWUD5zCH+APn8Azt2Q1g==</latexit><latexit sha1_base64="ctML9YmhATWPmVyBeW4LsKK9nQk=">AAAB83icbVBNSwMxEJ3Ur1q/qh69BIvgxbLbi+KpIIgehIr2A7pLyabZNjSbXZKsUJb+DS8eFPHqn/HmvzFt96CtDwYe780wMy9IBNfGcb5RYWV1bX2juFna2t7Z3SvvH7R0nCrKmjQWseoERDPBJWsabgTrJIqRKBCsHYyupn77iSnNY/loxgnzIzKQPOSUGCt5mReE+O767KF1O+mVK07VmQEvEzcnFcjR6JW/vH5M04hJQwXRuus6ifEzogyngk1KXqpZQuiIDFjXUkkipv1sdvMEn1ilj8NY2ZIGz9TfExmJtB5Hge2MiBnqRW8q/ud1UxNe+BmXSWqYpPNFYSqwifE0ANznilEjxpYQqri9FdMhUYQaG1PJhuAuvrxMWrWq61Td+1qlfpnHUYQjOIZTcOEc6nADDWgChQSe4RXeUIpe0Dv6mLcWUD5zCH+APn8Azt2Q1g==</latexit><latexit sha1_base64="ctML9YmhATWPmVyBeW4LsKK9nQk=">AAAB83icbVBNSwMxEJ3Ur1q/qh69BIvgxbLbi+KpIIgehIr2A7pLyabZNjSbXZKsUJb+DS8eFPHqn/HmvzFt96CtDwYe780wMy9IBNfGcb5RYWV1bX2juFna2t7Z3SvvH7R0nCrKmjQWseoERDPBJWsabgTrJIqRKBCsHYyupn77iSnNY/loxgnzIzKQPOSUGCt5mReE+O767KF1O+mVK07VmQEvEzcnFcjR6JW/vH5M04hJQwXRuus6ifEzogyngk1KXqpZQuiIDFjXUkkipv1sdvMEn1ilj8NY2ZIGz9TfExmJtB5Hge2MiBnqRW8q/ud1UxNe+BmXSWqYpPNFYSqwifE0ANznilEjxpYQqri9FdMhUYQaG1PJhuAuvrxMWrWq61Td+1qlfpnHUYQjOIZTcOEc6nADDWgChQSe4RXeUIpe0Dv6mLcWUD5zCH+APn8Azt2Q1g==</latexit>

a<latexit sha1_base64="+8O6dzOh7RYlxYInflBJMh8cOdo=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTOhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/CQ+PUw==</latexit><latexit sha1_base64="+8O6dzOh7RYlxYInflBJMh8cOdo=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTOhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/CQ+PUw==</latexit><latexit sha1_base64="+8O6dzOh7RYlxYInflBJMh8cOdo=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTOhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/CQ+PUw==</latexit><latexit sha1_base64="+8O6dzOh7RYlxYInflBJMh8cOdo=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTOhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/CQ+PUw==</latexit> b

<latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit>

Figure 2: MetaPrior as a Function Prior: (a) Means ofthe meta-variables Z of MetaPrior model embedded inshared 2d space at the onset of training and at the endshow how the variables perform a structured topographicmapping over the embedding space. (b) Left: MetaPriorfit and function draws fZ are visualized. Functions aredrawn by keeping all meta-variables clamped to the meanexcept a random one among the hidden layer units whichis then perturbed according to the offset indicated by thecolor-legend. Changes in a single meta-variable induceglobal changes across the entire function. The functionspace itself is interesting as the model appears to havegeneralized to a variety of smoothly varying functions aswe change the sampling offset, the mean of which is thecubic function. This is a natural task for our model, asall units of the hidden layer are embedded in the samespace and sampling means exploring that space and allthe functions this induces. Right: A MF-VI BNN withfixed observation noise function fit to the toy exampleand function draws are shown. The function draws areperformed by picking 40 weights at random and samplingfrom their priors, while keeping the rest clamped to theirposterior means. Single weight changes induce barely per-ceptible function fluctuations and in order to explore therepresentational ability of the model we sample multipleweights at once.

ity is wasted on modeling empty units repeatedly.

4.4 Few-Shot and Multi-Task LearningThe model allows task-specific latent variables Zt to beused to learn representations for separate, but related,tasks, effectively aligning the model to the current inputdata. In particular, the hierarchical structure of the modelfacilitates few-shot learning on a new task t by inferringP (Zt|Dt), starting from the Z produced by the originaltraining, and keeping fixed the mapping P (W|Zt; ξ). Wetested this ability using the MNIST network. After learn-ing the previous model for clean MNIST, we evaluate themodel’s performance on the MNIST test data in both aclean and a permuted version. We sample random per-mutations π for input pixels and classes and apply themto the test data, visualized in Fig. 5. This yields threedatasets, with permutations of input, output or both. We

mean

Epoch 0

Epoch 1000

Layer 1 Layer 2

Figure 3: Classification and Weight Correlation Ex-ample: (a) We illustrate correlation structures in themarginal weight distributions for the Half Moon classifica-tion example. All units Z are clamped to their mean. Wesample a single unit z0 repeatedly from its 2-dimensionaldistribution and sample P (W|Z). We track how partic-ular ancestral samples generate weights by color-codingthe resulting sampled weights by the offset of the ances-tral sample from its mean. Similar colors indicate similarsamples. Horizontal axes denote value for weight 1, ver-tical weight for 2, so that each point describes a state ofa weight column in a layer. Left: We show the sampledweights from layer 1 of the neural network connectedto unit 0. Right: The sampled weights from the outputlayer 2 connected to unit 0. Top: At the onset of train-ing the effect of varying the unit meta-variable zu is notobservable. Bottom: After learning the function shownin SubFig. d we can see that the effect of varying theunit meta-variable induces correlations within each layerand also across both layers. (b) Sketch of the NN usedwith highlighted unit whose meta-variable we perturb andthe affected weights. (c) Legend encoding shifts fromthe mean in the meta-variable to colors. (d) Predictiveuncertainty of the classifier and the predicted labels ofthe datapoints, demonstrating that the model learns morethan a point estimate for the class boundary.

then proceed to illate P (Zt|Dt) on progressively grow-ing numbers of scrambled observations (shots) Dt. Foreach such attempt, we reset the model’s representationsto the original ones from clean training. As a baseline,we train a mean field neural network. We also keep trackof performance on the clean dataset as reported in Fig. 5.We examine a related setting of generalization in the Ap-pendix Sec. 6.

5 DiscussionWe proposed a meta-representation of neural networks.This is based on the idea of characterizing neurons interms of pre-determined or learned latent variables, andusing a shared hypernetwork to generate weights andbiases from conditional distributions based on those vari-

b<latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="zQFmq8S0y+sDrDYgVPnXJXJ9W30=">AAAB43icbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZwT6gHUomvdOGZjJDcqdQhv4INy4U8T+589+YPhbaeiDwcU5C7j1RpqQl3//2Sju7e/sH5cPKUfX45LR2Vm3bNDcCWyJVqelG3KKSGlskSWE3M8iTSGEnmjws8s4UjZWpfqZZhmHCR1rGUnByVqfoRzGL5oNa3W/4S7FtCNZQh7Wag9pXf5iKPEFNQnFre4GfUVhwQ1IonFf6ucWMiwkfYc+h5gnasFiOO2dXzhmyODXuaGJL9/eLgifWzpLI3Uw4je1mtjD/y3o5xXdhIXWWE2qx+ijOFaOULXZnQ2lQkJo54MJINysTY264INdQxZUQbK68De2bRuA3gicfynABl3ANAdzCPTxCE1ogYAIv8AbvXua9eh+rukreurdz+CPv8wffxY4F</latexit><latexit sha1_base64="zQFmq8S0y+sDrDYgVPnXJXJ9W30=">AAAB43icbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZwT6gHUomvdOGZjJDcqdQhv4INy4U8T+589+YPhbaeiDwcU5C7j1RpqQl3//2Sju7e/sH5cPKUfX45LR2Vm3bNDcCWyJVqelG3KKSGlskSWE3M8iTSGEnmjws8s4UjZWpfqZZhmHCR1rGUnByVqfoRzGL5oNa3W/4S7FtCNZQh7Wag9pXf5iKPEFNQnFre4GfUVhwQ1IonFf6ucWMiwkfYc+h5gnasFiOO2dXzhmyODXuaGJL9/eLgifWzpLI3Uw4je1mtjD/y3o5xXdhIXWWE2qx+ijOFaOULXZnQ2lQkJo54MJINysTY264INdQxZUQbK68De2bRuA3gicfynABl3ANAdzCPTxCE1ogYAIv8AbvXua9eh+rukreurdz+CPv8wffxY4F</latexit><latexit sha1_base64="YlyWyW3pm+e2ejTB5mxunPYmoMI=">AAAB7nicbVA9SwNBEJ2LXzF+RS1tFoNgFe5sFKuAjWUE8wHJEfY2c8mS3b1jd08IR36EjYUitv4eO/+Nm+QKTXww8Hhvhpl5USq4sb7/7ZU2Nre2d8q7lb39g8Oj6vFJ2ySZZthiiUh0N6IGBVfYstwK7KYaqYwEdqLJ3dzvPKE2PFGPdppiKOlI8Zgzap3UyftRTKLZoFrz6/4CZJ0EBalBgeag+tUfJiyTqCwT1Jhe4Kc2zKm2nAmcVfqZwZSyCR1hz1FFJZowX5w7IxdOGZI40a6UJQv190ROpTFTGblOSe3YrHpz8T+vl9n4Jsy5SjOLii0XxZkgNiHz38mQa2RWTB2hTHN3K2FjqimzLqGKCyFYfXmdtK/qgV8PHvxa47aIowxncA6XEMA1NOAemtACBhN4hld481LvxXv3PpatJa+YOYU/8D5/AAn0j1I=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit>

a<latexit sha1_base64="+8O6dzOh7RYlxYInflBJMh8cOdo=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTOhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/CQ+PUw==</latexit><latexit sha1_base64="+8O6dzOh7RYlxYInflBJMh8cOdo=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTOhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/CQ+PUw==</latexit><latexit sha1_base64="+8O6dzOh7RYlxYInflBJMh8cOdo=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTOhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/CQ+PUw==</latexit><latexit sha1_base64="+8O6dzOh7RYlxYInflBJMh8cOdo=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTOhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/CQ+PUw==</latexit>

c<latexit sha1_base64="SzrBuFPMaZUYMA6Iz11SK9Yu52k=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTNhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/DBmPVQ==</latexit><latexit sha1_base64="SzrBuFPMaZUYMA6Iz11SK9Yu52k=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTNhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/DBmPVQ==</latexit><latexit sha1_base64="SzrBuFPMaZUYMA6Iz11SK9Yu52k=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTNhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/DBmPVQ==</latexit><latexit sha1_base64="SzrBuFPMaZUYMA6Iz11SK9Yu52k=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhL3NXrJkb/fYnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWZelEph0fe/vY3Nre2d3dJeef/g8Oi4cnLatjozjLeYltp0I2q5FIq3UKDk3dRwmkSSd6LJ3dzvPHFjhVaPOE15mNCRErFgFJ3UyftRTNhsUKn6NX8Bsk6CglShQHNQ+eoPNcsSrpBJam0v8FMMc2pQMMln5X5meUrZhI54z1FFE27DfHHujFw6ZUhibVwpJAv190ROE2unSeQ6E4pju+rNxf+8XobxTZgLlWbIFVsuijNJUJP572QoDGcop45QZoS7lbAxNZShS6jsQghWX14n7Xot8GvBQ73auC3iKME5XMAVBHANDbiHJrSAwQSe4RXevNR78d69j2XrhlfMnMEfeJ8/DBmPVQ==</latexit>

d<latexit sha1_base64="nYMwtCnBEzCXPWXLbp0z6Uk/PMc=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0l6sXgqePFYwX5AG8pmM2mXbjZhdyOU0B/hxYMiXv093vw3btsctPXBwOO9GWbmBang2rjut1Pa2t7Z3SvvVw4Oj45PqqdnXZ1kimGHJSJR/YBqFFxix3AjsJ8qpHEgsBdM7xZ+7wmV5ol8NLMU/ZiOJY84o8ZKvXwYRCScj6o1t+4uQTaJV5AaFGiPql/DMGFZjNIwQbUeeG5q/Jwqw5nAeWWYaUwpm9IxDiyVNEbt58tz5+TKKiGJEmVLGrJUf0/kNNZ6Fge2M6Zmote9hfifN8hM1PRzLtPMoGSrRVEmiEnI4ncScoXMiJkllClubyVsQhVlxiZUsSF46y9vkm6j7rl176FRa90WcZThAi7hGjy4gRbcQxs6wGAKz/AKb07qvDjvzseqteQUM+fwB87nDw2ej1Y=</latexit><latexit sha1_base64="nYMwtCnBEzCXPWXLbp0z6Uk/PMc=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0l6sXgqePFYwX5AG8pmM2mXbjZhdyOU0B/hxYMiXv093vw3btsctPXBwOO9GWbmBang2rjut1Pa2t7Z3SvvVw4Oj45PqqdnXZ1kimGHJSJR/YBqFFxix3AjsJ8qpHEgsBdM7xZ+7wmV5ol8NLMU/ZiOJY84o8ZKvXwYRCScj6o1t+4uQTaJV5AaFGiPql/DMGFZjNIwQbUeeG5q/Jwqw5nAeWWYaUwpm9IxDiyVNEbt58tz5+TKKiGJEmVLGrJUf0/kNNZ6Fge2M6Zmote9hfifN8hM1PRzLtPMoGSrRVEmiEnI4ncScoXMiJkllClubyVsQhVlxiZUsSF46y9vkm6j7rl176FRa90WcZThAi7hGjy4gRbcQxs6wGAKz/AKb07qvDjvzseqteQUM+fwB87nDw2ej1Y=</latexit><latexit sha1_base64="nYMwtCnBEzCXPWXLbp0z6Uk/PMc=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0l6sXgqePFYwX5AG8pmM2mXbjZhdyOU0B/hxYMiXv093vw3btsctPXBwOO9GWbmBang2rjut1Pa2t7Z3SvvVw4Oj45PqqdnXZ1kimGHJSJR/YBqFFxix3AjsJ8qpHEgsBdM7xZ+7wmV5ol8NLMU/ZiOJY84o8ZKvXwYRCScj6o1t+4uQTaJV5AaFGiPql/DMGFZjNIwQbUeeG5q/Jwqw5nAeWWYaUwpm9IxDiyVNEbt58tz5+TKKiGJEmVLGrJUf0/kNNZ6Fge2M6Zmote9hfifN8hM1PRzLtPMoGSrRVEmiEnI4ncScoXMiJkllClubyVsQhVlxiZUsSF46y9vkm6j7rl176FRa90WcZThAi7hGjy4gRbcQxs6wGAKz/AKb07qvDjvzseqteQUM+fwB87nDw2ej1Y=</latexit><latexit sha1_base64="nYMwtCnBEzCXPWXLbp0z6Uk/PMc=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0l6sXgqePFYwX5AG8pmM2mXbjZhdyOU0B/hxYMiXv093vw3btsctPXBwOO9GWbmBang2rjut1Pa2t7Z3SvvVw4Oj45PqqdnXZ1kimGHJSJR/YBqFFxix3AjsJ8qpHEgsBdM7xZ+7wmV5ol8NLMU/ZiOJY84o8ZKvXwYRCScj6o1t+4uQTaJV5AaFGiPql/DMGFZjNIwQbUeeG5q/Jwqw5nAeWWYaUwpm9IxDiyVNEbt58tz5+TKKiGJEmVLGrJUf0/kNNZ6Fge2M6Zmote9hfifN8hM1PRzLtPMoGSrRVEmiEnI4ncScoXMiJkllClubyVsQhVlxiZUsSF46y9vkm6j7rl176FRa90WcZThAi7hGjy4gRbcQxs6wGAKz/AKb07qvDjvzseqteQUM+fwB87nDw2ej1Y=</latexit>

Figure 4: MNIST Study: (a) T-SNE visualization of theunits in an MNIST network before training reveals ran-dom structure of the units of various layers. (b) Traininginduces structured embeddings of input, hidden and classunits. (c) Input units color coded by coordinates of corre-sponding unit embeddings are visualized. Interestingly,many units in the input layer appear to cluster accordingto their marginal occupancy of digits. This is an expectedproperty for the boundary pixels in MNIST which arealways empty. The model represents those input pixelswith similar latent units Z and effectively compresses theweight space. (d) Performance table of MNIST models.

π


b<latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit>



Label Task<latexit sha1_base64="e5gqjsKAvV99KRWq12vcM31DfFQ=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtylUawCNhYWEfIFSQh7m7lkyd7esbsXCEf+iY2FIrb+Ezv/jZvkCk18MPB4b4aZeUEiuDae9+0UtrZ3dveK+6WDw6PjE/f0rKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaDyf3Cb09RaR7Lhpkl2I/oSPKQM2qsNHDdrBeE5JEGKEiD6sl84Ja9ircE2SR+TsqQoz5wv3rDmKURSsME1brre4npZ1QZzgTOS71UY0LZhI6wa6mkEep+trx8Tq6sMiRhrGxJQ5bq74mMRlrPosB2RtSM9bq3EP/zuqkJb/sZl0lqULLVojAVxMRkEQMZcoXMiJkllClubyVsTBVlxoZVsiH46y9vkla14nsV/6lart3lcRThAi7hGny4gRo8QB2awGAKz/AKb07mvDjvzseqteDkM+fwB87nD8gzkxA=</latexit><latexit sha1_base64="e5gqjsKAvV99KRWq12vcM31DfFQ=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtylUawCNhYWEfIFSQh7m7lkyd7esbsXCEf+iY2FIrb+Ezv/jZvkCk18MPB4b4aZeUEiuDae9+0UtrZ3dveK+6WDw6PjE/f0rKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaDyf3Cb09RaR7Lhpkl2I/oSPKQM2qsNHDdrBeE5JEGKEiD6sl84Ja9ircE2SR+TsqQoz5wv3rDmKURSsME1brre4npZ1QZzgTOS71UY0LZhI6wa6mkEep+trx8Tq6sMiRhrGxJQ5bq74mMRlrPosB2RtSM9bq3EP/zuqkJb/sZl0lqULLVojAVxMRkEQMZcoXMiJkllClubyVsTBVlxoZVsiH46y9vkla14nsV/6lart3lcRThAi7hGny4gRo8QB2awGAKz/AKb07mvDjvzseqteDkM+fwB87nD8gzkxA=</latexit><latexit sha1_base64="e5gqjsKAvV99KRWq12vcM31DfFQ=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtylUawCNhYWEfIFSQh7m7lkyd7esbsXCEf+iY2FIrb+Ezv/jZvkCk18MPB4b4aZeUEiuDae9+0UtrZ3dveK+6WDw6PjE/f0rKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaDyf3Cb09RaR7Lhpkl2I/oSPKQM2qsNHDdrBeE5JEGKEiD6sl84Ja9ircE2SR+TsqQoz5wv3rDmKURSsME1brre4npZ1QZzgTOS71UY0LZhI6wa6mkEep+trx8Tq6sMiRhrGxJQ5bq74mMRlrPosB2RtSM9bq3EP/zuqkJb/sZl0lqULLVojAVxMRkEQMZcoXMiJkllClubyVsTBVlxoZVsiH46y9vkla14nsV/6lart3lcRThAi7hGny4gRo8QB2awGAKz/AKb07mvDjvzseqteDkM+fwB87nD8gzkxA=</latexit><latexit sha1_base64="e5gqjsKAvV99KRWq12vcM31DfFQ=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtylUawCNhYWEfIFSQh7m7lkyd7esbsXCEf+iY2FIrb+Ezv/jZvkCk18MPB4b4aZeUEiuDae9+0UtrZ3dveK+6WDw6PjE/f0rKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaDyf3Cb09RaR7Lhpkl2I/oSPKQM2qsNHDdrBeE5JEGKEiD6sl84Ja9ircE2SR+TsqQoz5wv3rDmKURSsME1brre4npZ1QZzgTOS71UY0LZhI6wa6mkEep+trx8Tq6sMiRhrGxJQ5bq74mMRlrPosB2RtSM9bq3EP/zuqkJb/sZl0lqULLVojAVxMRkEQMZcoXMiJkllClubyVsTBVlxoZVsiH46y9vkla14nsV/6lart3lcRThAi7hGny4gRo8QB2awGAKz/AKb07mvDjvzseqteDkM+fwB87nD8gzkxA=</latexit>

Pixel Task<latexit sha1_base64="umKQHqEaWb1zcw0Cjw8cSMBy1DE=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtylUawCNpYR8gVJCHubuWTJ3t6xuxcMR/6JjYUitv4TO/+Nm+QKTXww8Hhvhpl5QSK4Np737RS2tnd294r7pYPDo+MT9/SspeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/cJvT1FpHsuGmSXYj+hI8pAzaqw0cN2sF4Skzp9QkAbVk/nALXsVbwmySfyclCFHfeB+9YYxSyOUhgmqddf3EtPPqDKcCZyXeqnGhLIJHWHXUkkj1P1sefmcXFllSMJY2ZKGLNXfExmNtJ5Fge2MqBnrdW8h/ud1UxPe9jMuk9SgZKtFYSqIickiBjLkCpkRM0soU9zeStiYKsqMDatkQ/DXX94krWrF9yr+Y7Vcu8vjKMIFXMI1+HADNXiAOjSBwRSe4RXenMx5cd6dj1VrwclnzuEPnM8f/NuTMg==</latexit><latexit sha1_base64="umKQHqEaWb1zcw0Cjw8cSMBy1DE=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtylUawCNpYR8gVJCHubuWTJ3t6xuxcMR/6JjYUitv4TO/+Nm+QKTXww8Hhvhpl5QSK4Np737RS2tnd294r7pYPDo+MT9/SspeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/cJvT1FpHsuGmSXYj+hI8pAzaqw0cN2sF4Skzp9QkAbVk/nALXsVbwmySfyclCFHfeB+9YYxSyOUhgmqddf3EtPPqDKcCZyXeqnGhLIJHWHXUkkj1P1sefmcXFllSMJY2ZKGLNXfExmNtJ5Fge2MqBnrdW8h/ud1UxPe9jMuk9SgZKtFYSqIickiBjLkCpkRM0soU9zeStiYKsqMDatkQ/DXX94krWrF9yr+Y7Vcu8vjKMIFXMI1+HADNXiAOjSBwRSe4RXenMx5cd6dj1VrwclnzuEPnM8f/NuTMg==</latexit><latexit sha1_base64="umKQHqEaWb1zcw0Cjw8cSMBy1DE=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtylUawCNpYR8gVJCHubuWTJ3t6xuxcMR/6JjYUitv4TO/+Nm+QKTXww8Hhvhpl5QSK4Np737RS2tnd294r7pYPDo+MT9/SspeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/cJvT1FpHsuGmSXYj+hI8pAzaqw0cN2sF4Skzp9QkAbVk/nALXsVbwmySfyclCFHfeB+9YYxSyOUhgmqddf3EtPPqDKcCZyXeqnGhLIJHWHXUkkj1P1sefmcXFllSMJY2ZKGLNXfExmNtJ5Fge2MqBnrdW8h/ud1UxPe9jMuk9SgZKtFYSqIickiBjLkCpkRM0soU9zeStiYKsqMDatkQ/DXX94krWrF9yr+Y7Vcu8vjKMIFXMI1+HADNXiAOjSBwRSe4RXenMx5cd6dj1VrwclnzuEPnM8f/NuTMg==</latexit><latexit sha1_base64="umKQHqEaWb1zcw0Cjw8cSMBy1DE=">AAAB+XicbVA9SwNBEJ2LXzF+nVraLAbBKtylUawCNpYR8gVJCHubuWTJ3t6xuxcMR/6JjYUitv4TO/+Nm+QKTXww8Hhvhpl5QSK4Np737RS2tnd294r7pYPDo+MT9/SspeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/cJvT1FpHsuGmSXYj+hI8pAzaqw0cN2sF4Skzp9QkAbVk/nALXsVbwmySfyclCFHfeB+9YYxSyOUhgmqddf3EtPPqDKcCZyXeqnGhLIJHWHXUkkj1P1sefmcXFllSMJY2ZKGLNXfExmNtJ5Fge2MqBnrdW8h/ud1UxPe9jMuk9SgZKtFYSqIickiBjLkCpkRM0soU9zeStiYKsqMDatkQ/DXX94krWrF9yr+Y7Vcu8vjKMIFXMI1+HADNXiAOjSBwRSe4RXenMx5cd6dj1VrwclnzuEPnM8f/NuTMg==</latexit>

Pixel-Label Task<latexit sha1_base64="hTySN1OZqGqYPfK97GUJRwcVOKI=">AAAB/3icbVA9SwNBEN2LXzF+nQo2NotBsDHcpVGsAjYWFhHyBUkIc5u5ZMneB7t7YjhT+FdsLBSx9W/Y+W/cJFdo4oOBx3szzMzzYsGVdpxvK7eyura+kd8sbG3v7O7Z+wcNFSWSYZ1FIpItDxQKHmJdcy2wFUuEwBPY9EbXU795j1LxKKzpcYzdAAYh9zkDbaSefZR2PJ9W+QOK81vwUNAaqNGkZxedkjMDXSZuRookQ7Vnf3X6EUsCDDUToFTbdWLdTUFqzgROCp1EYQxsBANsGxpCgKqbzu6f0FOj9KkfSVOhpjP190QKgVLjwDOdAeihWvSm4n9eO9H+ZTflYZxoDNl8kZ8IqiM6DYP2uUSmxdgQYJKbWykbggSmTWQFE4K7+PIyaZRLrlNy78rFylUWR54ckxNyRlxyQSrkhlRJnTDySJ7JK3mznqwX6936mLfmrGzmkPyB9fkDHu+Vew==</latexit><latexit sha1_base64="hTySN1OZqGqYPfK97GUJRwcVOKI=">AAAB/3icbVA9SwNBEN2LXzF+nQo2NotBsDHcpVGsAjYWFhHyBUkIc5u5ZMneB7t7YjhT+FdsLBSx9W/Y+W/cJFdo4oOBx3szzMzzYsGVdpxvK7eyura+kd8sbG3v7O7Z+wcNFSWSYZ1FIpItDxQKHmJdcy2wFUuEwBPY9EbXU795j1LxKKzpcYzdAAYh9zkDbaSefZR2PJ9W+QOK81vwUNAaqNGkZxedkjMDXSZuRookQ7Vnf3X6EUsCDDUToFTbdWLdTUFqzgROCp1EYQxsBANsGxpCgKqbzu6f0FOj9KkfSVOhpjP190QKgVLjwDOdAeihWvSm4n9eO9H+ZTflYZxoDNl8kZ8IqiM6DYP2uUSmxdgQYJKbWykbggSmTWQFE4K7+PIyaZRLrlNy78rFylUWR54ckxNyRlxyQSrkhlRJnTDySJ7JK3mznqwX6936mLfmrGzmkPyB9fkDHu+Vew==</latexit><latexit sha1_base64="hTySN1OZqGqYPfK97GUJRwcVOKI=">AAAB/3icbVA9SwNBEN2LXzF+nQo2NotBsDHcpVGsAjYWFhHyBUkIc5u5ZMneB7t7YjhT+FdsLBSx9W/Y+W/cJFdo4oOBx3szzMzzYsGVdpxvK7eyura+kd8sbG3v7O7Z+wcNFSWSYZ1FIpItDxQKHmJdcy2wFUuEwBPY9EbXU795j1LxKKzpcYzdAAYh9zkDbaSefZR2PJ9W+QOK81vwUNAaqNGkZxedkjMDXSZuRookQ7Vnf3X6EUsCDDUToFTbdWLdTUFqzgROCp1EYQxsBANsGxpCgKqbzu6f0FOj9KkfSVOhpjP190QKgVLjwDOdAeihWvSm4n9eO9H+ZTflYZxoDNl8kZ8IqiM6DYP2uUSmxdgQYJKbWykbggSmTWQFE4K7+PIyaZRLrlNy78rFylUWR54ckxNyRlxyQSrkhlRJnTDySJ7JK3mznqwX6936mLfmrGzmkPyB9fkDHu+Vew==</latexit><latexit sha1_base64="hTySN1OZqGqYPfK97GUJRwcVOKI=">AAAB/3icbVA9SwNBEN2LXzF+nQo2NotBsDHcpVGsAjYWFhHyBUkIc5u5ZMneB7t7YjhT+FdsLBSx9W/Y+W/cJFdo4oOBx3szzMzzYsGVdpxvK7eyura+kd8sbG3v7O7Z+wcNFSWSYZ1FIpItDxQKHmJdcy2wFUuEwBPY9EbXU795j1LxKKzpcYzdAAYh9zkDbaSefZR2PJ9W+QOK81vwUNAaqNGkZxedkjMDXSZuRookQ7Vnf3X6EUsCDDUToFTbdWLdTUFqzgROCp1EYQxsBANsGxpCgKqbzu6f0FOj9KkfSVOhpjP190QKgVLjwDOdAeihWvSm4n9eO9H+ZTflYZxoDNl8kZ8IqiM6DYP2uUSmxdgQYJKbWykbggSmTWQFE4K7+PIyaZRLrlNy78rFylUWR54ckxNyRlxyQSrkhlRJnTDySJ7JK3mznqwX6936mLfmrGzmkPyB9fkDHu+Vew==</latexit>

Figure 5: Few-shot Disentangling Task (a)-(c) Left:Performance of MetaPrior when inferring Q(Z|D) as afunction of inference work (color coding) and revealedtest data (shots divided by 20) for adaptation to a newtask. Right: Performance of the baseline MF-BNN modeldecays on the clean task. (d) Illustration of the pixel per-mutation we stress the model with.

ables. We used this meta-representation as a functionprior, and showed its advantageous properties as a learned,adaptive, weight regularizer that can perform complexitycontrol in function space. We also showed the complexcorrelation structures in the input and output weights ofhidden units that arise from this meta-representation, anddemonstrated how the combination of hypernetwork andnetwork can adapt to out-of-task generalization settingsand distribution shift by re-aligning the networks to thenew data. Our model handles a variety of tasks with-out requiring task-dependent manually-imposed structure,as it benefits from the blessing of abstraction (9) whicharises when rich structured representations emerge fromhierarchical modeling.

We believe this type of modeling jointly capturing repre-sentational uncertainty, function uncertainty and observa-tion uncertainty can be beneficially applied to many differ-ent neural network architectures and generalized furtherwith more interestingly structured meta-representations.

References

D. F. Andrews and C. L. Mallows. Scale mixtures ofnormal distributions. Journal of the Royal StatisticalSociety. Series B (Methodological), pages 99–102, 1974.C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wier-stra. Weight uncertainty in neural networks. arXivpreprint arXiv:1505.05424, 2015.J. Clune, K. O. Stanley, R. T. Pennock, and C. Ofria. Onthe performance of indirect encoding across the contin-uum of regularity. IEEE Transactions on EvolutionaryComputation, 15(3):346–367, 2011.A. Damianou and N. Lawrence. Deep gaussian pro-cesses. In Artificial Intelligence and Statistics, pages207–215, 2013.P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel.The helmholtz machine. Neural computation, 7(5):889–904, 1995.L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density esti-mation using real nvp. arXiv preprint arXiv:1605.08803,2016.J. R. Finkel and C. D. Manning. Hierarchical bayesiandomain adaptation. In Proceedings of Human LanguageTechnologies: The 2009 Annual Conference of the NorthAmerican Chapter of the Association for ComputationalLinguistics, pages 602–610. Association for Computa-tional Linguistics, 2009.J. Gauci and K. O. Stanley. Autonomous evolutionof topographic regularities in artificial neural networks.Neural computation, 22(7):1860–1898, 2010.N. D. Goodman, T. D. Ullman, and J. B. Tenenbaum.Learning a theory of causality. Psychological review,118(1):110, 2011.J. M. Hernandez-Lobato and R. Adams. Probabilisticbackpropagation for scalable learning of bayesian neu-ral networks. In International Conference on MachineLearning, pages 1861–1869, 2015.G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal. The”wake-sleep” algorithm for unsupervised neural networks.Science, 268(5214):1158–1161, 1995.G. E. Hinton and D. Van Camp. Keeping the neuralnetworks simple by minimizing the description lengthof the weights. In Proceedings of the sixth annual con-ference on Computational learning theory, pages 5–13.ACM, 1993.A. Joshi, S. Ghosh, M. Betke, and H. Pfister. Hierar-chical bayesian neural networks for personalized clas-sification. In Neural Information Processing SystemsWorkshop on Bayesian Deep Learning, 2016.D. P. Kingma, T. Salimans, and M. Welling. Variationaldropout and the local reparameterization trick. In Ad-vances in Neural Information Processing Systems, pages2575–2583, 2015.D. P. Kingma and M. Welling. Stochastic gradient vb and

the variational auto-encoder. In Second InternationalConference on Learning Representations, ICLR, 2014.D. Krueger, C.-W. Huang, R. Islam, R. Turner, A. La-coste, and A. Courville. Bayesian hypernetworks. arXivpreprint arXiv:1710.04759, 2017.B. M. Lake, N. D. Lawrence, and J. B. Tenenbaum.The emergence of organizing structure in conceptualrepresentation. Cognitive science, 2018.B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum.Human-level concept learning through probabilistic pro-gram induction. Science, 350(6266):1332–1338, 2015.B. Lakshminarayanan, A. Pritzel, and C. Blundell. Sim-ple and scalable predictive uncertainty estimation usingdeep ensembles. In Advances in Neural InformationProcessing Systems, pages 6405–6416, 2017.N. D. Lawrence. Gaussian process latent variable modelsfor visualisation of high dimensional data. In Advancesin neural information processing systems, pages 329–336, 2004.P. Liang, M. I. Jordan, and D. Klein. Learning programs:A hierarchical bayesian approach. In Proceedings ofthe 27th International Conference on Machine Learning(ICML-10), pages 639–646, 2010.C. Louizos and M. Welling. Multiplicative normalizingflows for variational bayesian neural networks. arXivpreprint arXiv:1703.01961, 2017.D. J. MacKay. Bayesian interpolation. Neural computa-tion, 4(3):415–447, 1992.D. J. MacKay. A practical bayesian framework for back-propagation networks. Neural computation, 4(3):448–472, 1992.D. J. MacKay. Bayesian neural networks and densitynetworks. Nuclear Instruments and Methods in PhysicsResearch Section A: Accelerators, Spectrometers, Detec-tors and Associated Equipment, 354(1):73–80, 1995.R. M. Neal. Priors for infinite networks. In BayesianLearning for Neural Networks, pages 29–53. Springer,1996.R. M. Neal. Bayesian learning for neural networks,volume 118. Springer Science & Business Media, 2012.G. Papamakarios, I. Murray, and T. Pavlakou. Maskedautoregressive flow for density estimation. In Advancesin Neural Information Processing Systems, pages 2335–2344, 2017.N. Pawlowski, M. Rajchl, and B. Glocker. Implicitweight uncertainty in neural networks. arXiv preprintarXiv:1711.01297, 2017.Y. N. Perov and F. D. Wood. Learning probabilisticprograms. arXiv preprint arXiv:1407.2646, 2014.J. Quinonero-Candela and C. E. Rasmussen. A unifyingview of sparse approximate gaussian process regression.Journal of Machine Learning Research, 6(Dec):1939–1959, 2005.D. J. Rezende and S. Mohamed. Variational in-

ference with normalizing flows. arXiv preprintarXiv:1505.05770, 2015.D. J. Rezende, S. Mohamed, and D. Wierstra. Stochas-tic backpropagation and variational inference in deeplatent gaussian models. In International Conference onMachine Learning, volume 2, 2014.S. Risi and K. O. Stanley. An enhanced hypercube-based encoding for evolving the placement, density, andconnectivity of neurons. Artificial life, 18(4):331–363,2012.E. Snelson and Z. Ghahramani. Sparse gaussian pro-cesses using pseudo-inputs. In Advances in neural infor-mation processing systems, pages 1257–1264, 2006.K. O. Stanley. Compositional pattern producing net-works: A novel abstraction of development. Geneticprogramming and evolvable machines, 8(2):131–162,2007.M. Titsias. Variational learning of inducing variables insparse gaussian processes. In Artificial Intelligence andStatistics, pages 567–574, 2009.M. Titsias and N. D. Lawrence. Bayesian gaussian pro-cess latent variable model. In Proceedings of the Thir-teenth International Conference on Artificial Intelligenceand Statistics, pages 844–851, 2010.M. K. Titsias and M. Lazaro-Gredilla. Local expecta-tion gradients for black box variational inference. InProceedings of the 28th International Conference onNeural Information Processing Systems-Volume 2, pages2638–2646. MIT Press, 2015.


b<latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit><latexit sha1_base64="a/KccsW9NF2kxVLY8ZuJj2kezio=">AAAB7nicbVA9SwNBEJ3zM8avqKXNYhCswl0axSpgYxnBfEByhN3NXrJkb+/YnRPCkR9hY6GIrb/Hzn/jJrlCEx8MPN6bYWYeS5W06Pvf3sbm1vbObmmvvH9weHRcOTlt2yQzXLR4ohLTZdQKJbVooUQluqkRNGZKdNjkbu53noSxMtGPOE1FGNORlpHkFJ3UyfssImw2qFT9mr8AWSdBQapQoDmofPWHCc9ioZEram0v8FMMc2pQciVm5X5mRUr5hI5Ez1FNY2HDfHHujFw6ZUiixLjSSBbq74mcxtZOY+Y6Y4pju+rNxf+8XobRTZhLnWYoNF8uijJFMCHz38lQGsFRTR2h3Eh3K+FjaihHl1DZhRCsvrxO2vVa4NeCh3q1cVvEUYJzuIArCOAaGnAPTWgBhwk8wyu8ean34r17H8vWDa+YOYM/8D5/AAqUj1Q=</latexit>



Pixel-Task Zinput<latexit sha1_base64="Pfs0kz15mXgrraDCSsf27rGNYQM=">AAACBXicbVC7TsMwFL0pr1JeAUYYLFokFqqkC4ipEgtjkfoSbRQ5rtNadR6yHUQVZWHhV1gYQIiVf2Djb3DbDNByJEtH59yr63O8mDOpLOvbKKysrq1vFDdLW9s7u3vm/kFbRokgtEUiHomuhyXlLKQtxRSn3VhQHHicdrzx9dTv3FMhWRQ21SSmToCHIfMZwUpLrnmc9j0fNdgD5edNLMeocuemLIwTlVUy1yxbVWsGtEzsnJQhR8M1v/qDiCQBDRXhWMqebcXKSbFQjHCalfqJpDEmYzykPU1DHFDppLMUGTrVygD5kdAvVGim/t5IcSDlJPD0ZIDVSC56U/E/r5co/9KZh6IhmR/yE45UhKaVoAETlCg+0QQTwfRfERlhgYnSxZV0CfZi5GXSrlVtq2rf1sr1q7yOIhzBCZyBDRdQhxtoQAsIPMIzvMKb8WS8GO/Gx3y0YOQ7h/AHxucPqceYAA==</latexit><latexit sha1_base64="Pfs0kz15mXgrraDCSsf27rGNYQM=">AAACBXicbVC7TsMwFL0pr1JeAUYYLFokFqqkC4ipEgtjkfoSbRQ5rtNadR6yHUQVZWHhV1gYQIiVf2Djb3DbDNByJEtH59yr63O8mDOpLOvbKKysrq1vFDdLW9s7u3vm/kFbRokgtEUiHomuhyXlLKQtxRSn3VhQHHicdrzx9dTv3FMhWRQ21SSmToCHIfMZwUpLrnmc9j0fNdgD5edNLMeocuemLIwTlVUy1yxbVWsGtEzsnJQhR8M1v/qDiCQBDRXhWMqebcXKSbFQjHCalfqJpDEmYzykPU1DHFDppLMUGTrVygD5kdAvVGim/t5IcSDlJPD0ZIDVSC56U/E/r5co/9KZh6IhmR/yE45UhKaVoAETlCg+0QQTwfRfERlhgYnSxZV0CfZi5GXSrlVtq2rf1sr1q7yOIhzBCZyBDRdQhxtoQAsIPMIzvMKb8WS8GO/Gx3y0YOQ7h/AHxucPqceYAA==</latexit><latexit sha1_base64="Pfs0kz15mXgrraDCSsf27rGNYQM=">AAACBXicbVC7TsMwFL0pr1JeAUYYLFokFqqkC4ipEgtjkfoSbRQ5rtNadR6yHUQVZWHhV1gYQIiVf2Djb3DbDNByJEtH59yr63O8mDOpLOvbKKysrq1vFDdLW9s7u3vm/kFbRokgtEUiHomuhyXlLKQtxRSn3VhQHHicdrzx9dTv3FMhWRQ21SSmToCHIfMZwUpLrnmc9j0fNdgD5edNLMeocuemLIwTlVUy1yxbVWsGtEzsnJQhR8M1v/qDiCQBDRXhWMqebcXKSbFQjHCalfqJpDEmYzykPU1DHFDppLMUGTrVygD5kdAvVGim/t5IcSDlJPD0ZIDVSC56U/E/r5co/9KZh6IhmR/yE45UhKaVoAETlCg+0QQTwfRfERlhgYnSxZV0CfZi5GXSrlVtq2rf1sr1q7yOIhzBCZyBDRdQhxtoQAsIPMIzvMKb8WS8GO/Gx3y0YOQ7h/AHxucPqceYAA==</latexit><latexit sha1_base64="Pfs0kz15mXgrraDCSsf27rGNYQM=">AAACBXicbVC7TsMwFL0pr1JeAUYYLFokFqqkC4ipEgtjkfoSbRQ5rtNadR6yHUQVZWHhV1gYQIiVf2Djb3DbDNByJEtH59yr63O8mDOpLOvbKKysrq1vFDdLW9s7u3vm/kFbRokgtEUiHomuhyXlLKQtxRSn3VhQHHicdrzx9dTv3FMhWRQ21SSmToCHIfMZwUpLrnmc9j0fNdgD5edNLMeocuemLIwTlVUy1yxbVWsGtEzsnJQhR8M1v/qDiCQBDRXhWMqebcXKSbFQjHCalfqJpDEmYzykPU1DHFDppLMUGTrVygD5kdAvVGim/t5IcSDlJPD0ZIDVSC56U/E/r5co/9KZh6IhmR/yE45UhKaVoAETlCg+0QQTwfRfERlhgYnSxZV0CfZi5GXSrlVtq2rf1sr1q7yOIhzBCZyBDRdQhxtoQAsIPMIzvMKb8WS8GO/Gx3y0YOQ7h/AHxucPqceYAA==</latexit>

Pixel-Task Zoutput<latexit sha1_base64="J348xKJkjrrhiZJzQeeiByyGdbs=">AAACBnicbVDLSgMxFM3UV62vUZciBFvBjWWmG8VVwY3LCn1hOwyZNNOGZpIhyYhlmJUbf8WNC0Xc+g3u/BvTdhbaeiBwOOdebs4JYkaVdpxvq7Cyura+UdwsbW3v7O7Z+wdtJRKJSQsLJmQ3QIowyklLU81IN5YERQEjnWB8PfU790QqKnhTT2LiRWjIaUgx0kby7eO0H4SwQR8IO28iNYaVOz8ViY4TnVUy3y47VWcGuEzcnJRBjoZvf/UHAicR4RozpFTPdWLtpUhqihnJSv1EkRjhMRqSnqEcRUR56SxGBk+NMoChkOZxDWfq740URUpNosBMRkiP1KI3Ff/zeokOL72UchOKcDw/FCYMagGnncABlQRrNjEEYUnNXyEeIYmwNs2VTAnuYuRl0q5VXafq3tbK9au8jiI4AifgDLjgAtTBDWiAFsDgETyDV/BmPVkv1rv1MR8tWPnOIfgD6/MHnXyYiw==</latexit><latexit sha1_base64="J348xKJkjrrhiZJzQeeiByyGdbs=">AAACBnicbVDLSgMxFM3UV62vUZciBFvBjWWmG8VVwY3LCn1hOwyZNNOGZpIhyYhlmJUbf8WNC0Xc+g3u/BvTdhbaeiBwOOdebs4JYkaVdpxvq7Cyura+UdwsbW3v7O7Z+wdtJRKJSQsLJmQ3QIowyklLU81IN5YERQEjnWB8PfU790QqKnhTT2LiRWjIaUgx0kby7eO0H4SwQR8IO28iNYaVOz8ViY4TnVUy3y47VWcGuEzcnJRBjoZvf/UHAicR4RozpFTPdWLtpUhqihnJSv1EkRjhMRqSnqEcRUR56SxGBk+NMoChkOZxDWfq740URUpNosBMRkiP1KI3Ff/zeokOL72UchOKcDw/FCYMagGnncABlQRrNjEEYUnNXyEeIYmwNs2VTAnuYuRl0q5VXafq3tbK9au8jiI4AifgDLjgAtTBDWiAFsDgETyDV/BmPVkv1rv1MR8tWPnOIfgD6/MHnXyYiw==</latexit><latexit sha1_base64="J348xKJkjrrhiZJzQeeiByyGdbs=">AAACBnicbVDLSgMxFM3UV62vUZciBFvBjWWmG8VVwY3LCn1hOwyZNNOGZpIhyYhlmJUbf8WNC0Xc+g3u/BvTdhbaeiBwOOdebs4JYkaVdpxvq7Cyura+UdwsbW3v7O7Z+wdtJRKJSQsLJmQ3QIowyklLU81IN5YERQEjnWB8PfU790QqKnhTT2LiRWjIaUgx0kby7eO0H4SwQR8IO28iNYaVOz8ViY4TnVUy3y47VWcGuEzcnJRBjoZvf/UHAicR4RozpFTPdWLtpUhqihnJSv1EkRjhMRqSnqEcRUR56SxGBk+NMoChkOZxDWfq740URUpNosBMRkiP1KI3Ff/zeokOL72UchOKcDw/FCYMagGnncABlQRrNjEEYUnNXyEeIYmwNs2VTAnuYuRl0q5VXafq3tbK9au8jiI4AifgDLjgAtTBDWiAFsDgETyDV/BmPVkv1rv1MR8tWPnOIfgD6/MHnXyYiw==</latexit><latexit sha1_base64="J348xKJkjrrhiZJzQeeiByyGdbs=">AAACBnicbVDLSgMxFM3UV62vUZciBFvBjWWmG8VVwY3LCn1hOwyZNNOGZpIhyYhlmJUbf8WNC0Xc+g3u/BvTdhbaeiBwOOdebs4JYkaVdpxvq7Cyura+UdwsbW3v7O7Z+wdtJRKJSQsLJmQ3QIowyklLU81IN5YERQEjnWB8PfU790QqKnhTT2LiRWjIaUgx0kby7eO0H4SwQR8IO28iNYaVOz8ViY4TnVUy3y47VWcGuEzcnJRBjoZvf/UHAicR4RozpFTPdWLtpUhqihnJSv1EkRjhMRqSnqEcRUR56SxGBk+NMoChkOZxDWfq740URUpNosBMRkiP1KI3Ff/zeokOL72UchOKcDw/FCYMagGnncABlQRrNjEEYUnNXyEeIYmwNs2VTAnuYuRl0q5VXafq3tbK9au8jiI4AifgDLjgAtTBDWiAFsDgETyDV/BmPVkv1rv1MR8tWPnOIfgD6/MHnXyYiw==</latexit>

Label-Task Zoutput<latexit sha1_base64="RG6vxZR1+3YcXlYbkhkYqgYEO2c=">AAACBnicbVDLSgMxFM3UV62vUZciBFvBjWWmG8VVwY0LFxX6wnYYMmmmDc0kQ5IRyjArN/6KGxeKuPUb3Pk3pu0stPVA4HDOvdycE8SMKu0431ZhZXVtfaO4Wdra3tnds/cP2kokEpMWFkzIboAUYZSTlqaakW4sCYoCRjrB+Hrqdx6IVFTwpp7ExIvQkNOQYqSN5NvHaT8I4S0KCDtvIjWGlXs/FYmOE51VMt8uO1VnBrhM3JyUQY6Gb3/1BwInEeEaM6RUz3Vi7aVIaooZyUr9RJEY4TEakp6hHEVEeeksRgZPjTKAoZDmcQ1n6u+NFEVKTaLATEZIj9SiNxX/83qJDi+9lHITinA8PxQmDGoBp53AAZUEazYxBGFJzV8hHiGJsDbNlUwJ7mLkZdKuVV2n6t7VyvWrvI4iOAIn4Ay44ALUwQ1ogBbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB2camGk=</latexit><latexit sha1_base64="RG6vxZR1+3YcXlYbkhkYqgYEO2c=">AAACBnicbVDLSgMxFM3UV62vUZciBFvBjWWmG8VVwY0LFxX6wnYYMmmmDc0kQ5IRyjArN/6KGxeKuPUb3Pk3pu0stPVA4HDOvdycE8SMKu0431ZhZXVtfaO4Wdra3tnds/cP2kokEpMWFkzIboAUYZSTlqaakW4sCYoCRjrB+Hrqdx6IVFTwpp7ExIvQkNOQYqSN5NvHaT8I4S0KCDtvIjWGlXs/FYmOE51VMt8uO1VnBrhM3JyUQY6Gb3/1BwInEeEaM6RUz3Vi7aVIaooZyUr9RJEY4TEakp6hHEVEeeksRgZPjTKAoZDmcQ1n6u+NFEVKTaLATEZIj9SiNxX/83qJDi+9lHITinA8PxQmDGoBp53AAZUEazYxBGFJzV8hHiGJsDbNlUwJ7mLkZdKuVV2n6t7VyvWrvI4iOAIn4Ay44ALUwQ1ogBbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB2camGk=</latexit><latexit sha1_base64="RG6vxZR1+3YcXlYbkhkYqgYEO2c=">AAACBnicbVDLSgMxFM3UV62vUZciBFvBjWWmG8VVwY0LFxX6wnYYMmmmDc0kQ5IRyjArN/6KGxeKuPUb3Pk3pu0stPVA4HDOvdycE8SMKu0431ZhZXVtfaO4Wdra3tnds/cP2kokEpMWFkzIboAUYZSTlqaakW4sCYoCRjrB+Hrqdx6IVFTwpp7ExIvQkNOQYqSN5NvHaT8I4S0KCDtvIjWGlXs/FYmOE51VMt8uO1VnBrhM3JyUQY6Gb3/1BwInEeEaM6RUz3Vi7aVIaooZyUr9RJEY4TEakp6hHEVEeeksRgZPjTKAoZDmcQ1n6u+NFEVKTaLATEZIj9SiNxX/83qJDi+9lHITinA8PxQmDGoBp53AAZUEazYxBGFJzV8hHiGJsDbNlUwJ7mLkZdKuVV2n6t7VyvWrvI4iOAIn4Ay44ALUwQ1ogBbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB2camGk=</latexit><latexit sha1_base64="RG6vxZR1+3YcXlYbkhkYqgYEO2c=">AAACBnicbVDLSgMxFM3UV62vUZciBFvBjWWmG8VVwY0LFxX6wnYYMmmmDc0kQ5IRyjArN/6KGxeKuPUb3Pk3pu0stPVA4HDOvdycE8SMKu0431ZhZXVtfaO4Wdra3tnds/cP2kokEpMWFkzIboAUYZSTlqaakW4sCYoCRjrB+Hrqdx6IVFTwpp7ExIvQkNOQYqSN5NvHaT8I4S0KCDtvIjWGlXs/FYmOE51VMt8uO1VnBrhM3JyUQY6Gb3/1BwInEeEaM6RUz3Vi7aVIaooZyUr9RJEY4TEakp6hHEVEeeksRgZPjTKAoZDmcQ1n6u+NFEVKTaLATEZIj9SiNxX/83qJDi+9lHITinA8PxQmDGoBp53AAZUEazYxBGFJzV8hHiGJsDbNlUwJ7mLkZdKuVV2n6t7VyvWrvI4iOAIn4Ay44ALUwQ1ogBbA4BE8g1fwZj1ZL9a79TEfLVj5ziH4A+vzB2camGk=</latexit>

Label-Task Zinput<latexit sha1_base64="DdrZGW4wJWilGEeS/9giiJ4vOtM=">AAACBXicbVC7SgNBFJ2Nrxhfq5ZaDCaCjWE3jWIVsLGwiJAXJstydzKbDJl9MDMrhGUbG3/FxkIRW//Bzr9xkmyhiQcGDufcy51zvJgzqSzr2yisrK6tbxQ3S1vbO7t75v5BW0aJILRFIh6JrgeSchbSlmKK024sKAQepx1vfD31Ow9USBaFTTWJqRPAMGQ+I6C05JrHad/z8S14lJ83QY5x5d5NWRgnKqtkrlm2qtYMeJnYOSmjHA3X/OoPIpIENFSEg5Q924qVk4JQjHCalfqJpDGQMQxpT9MQAiqddJYiw6daGWA/EvqFCs/U3xspBFJOAk9PBqBGctGbiv95vUT5l848FA3J/JCfcKwiPK0ED5igRPGJJkAE03/FZAQCiNLFlXQJ9mLkZdKuVW2rat/VyvWrvI4iOkIn6AzZ6ALV0Q1qoBYi6BE9o1f0ZjwZL8a78TEfLRj5ziH6A+PzB3OHl94=</latexit><latexit sha1_base64="DdrZGW4wJWilGEeS/9giiJ4vOtM=">AAACBXicbVC7SgNBFJ2Nrxhfq5ZaDCaCjWE3jWIVsLGwiJAXJstydzKbDJl9MDMrhGUbG3/FxkIRW//Bzr9xkmyhiQcGDufcy51zvJgzqSzr2yisrK6tbxQ3S1vbO7t75v5BW0aJILRFIh6JrgeSchbSlmKK024sKAQepx1vfD31Ow9USBaFTTWJqRPAMGQ+I6C05JrHad/z8S14lJ83QY5x5d5NWRgnKqtkrlm2qtYMeJnYOSmjHA3X/OoPIpIENFSEg5Q924qVk4JQjHCalfqJpDGQMQxpT9MQAiqddJYiw6daGWA/EvqFCs/U3xspBFJOAk9PBqBGctGbiv95vUT5l848FA3J/JCfcKwiPK0ED5igRPGJJkAE03/FZAQCiNLFlXQJ9mLkZdKuVW2rat/VyvWrvI4iOkIn6AzZ6ALV0Q1qoBYi6BE9o1f0ZjwZL8a78TEfLRj5ziH6A+PzB3OHl94=</latexit><latexit sha1_base64="DdrZGW4wJWilGEeS/9giiJ4vOtM=">AAACBXicbVC7SgNBFJ2Nrxhfq5ZaDCaCjWE3jWIVsLGwiJAXJstydzKbDJl9MDMrhGUbG3/FxkIRW//Bzr9xkmyhiQcGDufcy51zvJgzqSzr2yisrK6tbxQ3S1vbO7t75v5BW0aJILRFIh6JrgeSchbSlmKK024sKAQepx1vfD31Ow9USBaFTTWJqRPAMGQ+I6C05JrHad/z8S14lJ83QY5x5d5NWRgnKqtkrlm2qtYMeJnYOSmjHA3X/OoPIpIENFSEg5Q924qVk4JQjHCalfqJpDGQMQxpT9MQAiqddJYiw6daGWA/EvqFCs/U3xspBFJOAk9PBqBGctGbiv95vUT5l848FA3J/JCfcKwiPK0ED5igRPGJJkAE03/FZAQCiNLFlXQJ9mLkZdKuVW2rat/VyvWrvI4iOkIn6AzZ6ALV0Q1qoBYi6BE9o1f0ZjwZL8a78TEfLRj5ziH6A+PzB3OHl94=</latexit><latexit sha1_base64="DdrZGW4wJWilGEeS/9giiJ4vOtM=">AAACBXicbVC7SgNBFJ2Nrxhfq5ZaDCaCjWE3jWIVsLGwiJAXJstydzKbDJl9MDMrhGUbG3/FxkIRW//Bzr9xkmyhiQcGDufcy51zvJgzqSzr2yisrK6tbxQ3S1vbO7t75v5BW0aJILRFIh6JrgeSchbSlmKK024sKAQepx1vfD31Ow9USBaFTTWJqRPAMGQ+I6C05JrHad/z8S14lJ83QY5x5d5NWRgnKqtkrlm2qtYMeJnYOSmjHA3X/OoPIpIENFSEg5Q924qVk4JQjHCalfqJpDGQMQxpT9MQAiqddJYiw6daGWA/EvqFCs/U3xspBFJOAk9PBqBGctGbiv95vUT5l848FA3J/JCfcKwiPK0ED5igRPGJJkAE03/FZAQCiNLFlXQJ9mLkZdKuVW2rat/VyvWrvI4iOkIn6AzZ6ALV0Q1qoBYi6BE9o1f0ZjwZL8a78TEfLRj5ziH6A+PzB3OHl94=</latexit>

Figure 6: Structured Surgery Task (a)-(d) Left: Perfor-mance of MetaPrior when inferringQ(Z·|D) as a functionof inference work (color coding) and revealed test data(shots divided by 20) for adaptation to a new task. Right:Performance of the baseline MF-BNN model decays onthe clean task.

6 Appendix: Structured Surgery

In the multi-task experiments in Sec. 4.4, we studiedthe model’s ability to update representations holisti-cally and generalize to unseen variations in the train-ing data. Here, we manipulate the meta-variables in astructured and targeted way (see Fig. 6). Since Z ={Zinput,Zhidden,Zoutput}, we can elect to perform illa-tion only on a subset of variables. Instead of learning anew set of task-dependent Zt variables, we only updateinput or output variables per task to demonstrate how themodel can disentangle the representations it models andcan generalize in highly structured fashion. When up-dating only the input variables, the model reasons aboutpixel transformations, as it can move the representationof each input pixel around its latent feature space. Themodel appears to solve for an input permutation by search-ing in representation space for a program approximatingZinput = π(Zinput). This search is ambiguous, givenlittle data and the sparsity underlying MNIST. This pro-cess demonstrates the alignment the model automaticallyperforms only of its inputs in order to change the weightdistribution it applies to datasets, while keeping the restof the features intact. Similarly, we observe that updatingonly the class-label units leads to rapid and effective gener-alization for class shifts or, in this case, class permutation,since only 10 variables need to be updated. The modelcould also easily generalize to new subclasses smoothlyexisting between current classes. These demonstrate theability of the model to react in differentiated ways toshifts, either by adapting to changes in input-geometry,target semantics or actual features in a targeted way whilekeeping the rest of the representations constant.

7 Appendix: Probabilistic NeuralNetworks

Let D be a dataset of n tuples {(x1, y1), ..., (xn, yn)}where x are inputs and y are targets for supervised learn-ing. Take a neural network (NN) with L layers, Vl

units in layer l (we drop l where this is clear), an over-all collection of weights and biases W = {Wl}1:L,and fixed nonlinear activation functions. In Bayesianterms, the NN realizes the likelihood p(y|x,W); togetherwith a prior P (W) over W , this generates the condi-tional distribution (also known as the marginal likeli-hood) P (y|x) =

∫P (y|x,W)P (W)dW , where x de-

notes (x1, . . . , xn) and y denotes (y1, . . . , yn). One com-mon assumption for the prior is that the weights are drawniid from a zero-mean common-variance normal distribu-tion, leading to a prior which factorizes across layers andunits:

P (W) =

L∏l=1

Vl∏i=1

Vl−1∏j=1

P (wl,i,j)

=

L∏l=1

Vl∏i=1

Vl−1∏j=1

N (wl,i,j |0, λ),

where i and j index units in adjacent layers of the network,and λ is the prior weight variance. Maximum-a-posterioriinference with this prior famously results in an objectiveidentical to L2 weight regularization with regularizationconstant 1/λ. Other suggestions for priors have beenmade, such as (26)’s proposal that the precision of theprior should be scaled according to the number of hiddenunits in a layer, yielding a contribution to the prior ofN (0, 1

Vl).

Bayesian learning requires us to infer the posterior dis-tribution of the weights given the data D: P (W|D) =P (W)

∏ni=1 P (yi|xi,W)/P (y|x). Unfortunately, the

marginal likelihood and posterior are intractable as theyinvolve integrating over a high-dimensional space definedby the weight priors. A common step is therefore to per-form approximate inference, for instance by varying theparameters Φ of an approximating distribution Q(W; Φ)to make it close to the true posterior. For instance, inMean Field Variational Inference (MF-VI), we considerthe factorized posterior:

Q(W; Φ) =

L∏l=1

Vl∏i=1

Vl−1∏j=1

Q(wl,i,j ;φφφl,i,j).

Commonly, Q is Gaussian for each weight,Q(wl,i,j ;φφφl,i,j) = N (wl,i,j |µl,i,j , σ

2l,i,j), with vari-

ational parameters φφφl,i,j = {µl,i,j , σ2l,i,j}. The

parameters Φ are adjusted to maximize a lower boundL(Φ) to the marginal likelihood given by the EvidenceLower Bound (ELBO) (a relative of the free energy):

L(Φ) = EQ(W)

[logP (y|x,W)+logP (W)−logQ(W)

].

(3)The mean field factorization assumption renders max-imizing the ELBO tractable for many models. The

predictive distribution of a Bayesian Neural Networkcan be approximated utilizing a mixture distributionP (y|x,D) ≈ 1

S

∑Ss=1 P (y|x,Ws) over S sampled in-

stances of weightsWs ∼ Q(W).

8 Appendix: Related Work

Our work brings together two themes in the literature.One is the probabilistic interpretation of weights andactivations of neural networks, which has been a com-mon approach to regularization and complexity con-trol (2; 5; 10; 11; 12; 23; 24; 25; 27). The second theme isto consider the structure and weights of neural networksas arising from embeddings in other spaces. This idea hasbeen explored in evolutionary computation (3; 8; 34; 36)and beyond, and applied to recurrent and convolutionalNNs and more. Our learned hierarchical probabilistic rep-resentation of units, which we call a meta-representationbecause of the way it generates the weights, is inspiredby this work. It can thus also be considered as a richlystructured hierarchical Bayesian Neural network (7; 13).In important recent work training ensembles of neuralnetworks (19) was proposed. This captures uncertaintywell; but ensembles are a departure from a single, self-contained model.

Our work is most closely related to two sets of recentstudies. One considers reweighting activation patternsto improve posterior inference (16; 22; 29). The use ofparametric weights and normalizing flows (6; 28; 32) tomodel scalar changes to those weights offers a probabilis-tic patina around forms of batch normalization. However,our work is not aimed at capturing posterior uncertaintyfor given weight priors, but rather as a novel weight priorin its own right. Our method is also representationallymore flexible, as it provides embeddings for the weightsas a whole.

Equally, our meta-representations of units is reminiscentof the inducing points that are used to simplify GaussianProcess (GP) inference (31; 35; 37), and that are key com-ponents in GP latent variable models (20; 38). Rather likeinducing points, our units control the modeled functionand regularize its complexity. However, unlike inducingpoints, the latent variables we use do not even occupy thesame space as the input data, and so offer the blessing ofextra abstraction. The meta-representational aspects ofthe model can be related to Deep GPs, as proposed by (4).

Finally, as is clearest in the permuted MNIST example,the hypernetwork can be cast as an interpreter, turning onerepresentation of a program (the unit embeddings) into arealized method for mapping inputs to outputs (the NN).Thus, our method can be seen in terms of program induc-tion, a field of recent interest in various fields, including

concept learning (17; 18; 21; 30).

Probabilistic Meta-Representations Of Neural Networksbalaji/udl-camera-ready/UDL-13.pdf ·...

Documents

Transcript of Probabilistic Meta-Representations Of Neural Networksbalaji/udl-camera-ready/UDL-13.pdf ·...