Learning a Domain-Invariant Embedding for Unsupervised DomainAdaptation Using Class-Conditioned Distribution Alignment

Alexander J. Gabourie1, Mohammad Rostami2, Philip E. Pope3,Soheil Kolouri3, and Kuyngnam Kim3

Abstract— We address the problem of unsupervised do-main adaptation (UDA) by learning a cross-domain agnosticembedding space, where the distance between the probabilitydistributions of the two source and target visual domainsis minimized. We use the output space of a shared cross-domain deep encoder to model the embedding space anduse the Sliced-Wasserstein Distance (SWD) to measure andminimize the distance between the embedded distributions oftwo source and target domains to enforce the embedding tobe domain-agnostic. Additionally, we use the source domainlabeled data to train a deep classifier from the embeddingspace to the label space to enforce the embedding space tobe discriminative. As a result of this training scheme, weprovide an effective solution to train the deep classificationnetwork on the source domain such that it will generalize wellon the target domain, where only unlabeled training data isaccessible. To mitigate the challenge of class matching, wealso align corresponding classes in the embedding space byusing high confidence pseudo-labels for the target domain,i.e. assigning the class for which the source classifier has ahigh prediction probability. We provide experimental resultson UDA benchmark tasks to demonstrate that our methodis effective and leads to state-of-the-art performance.

I. IntroductionDeep learning classification algorithms have surpassed

performance of humans for a wide range of computervision applications. However, this achievement is con-ditioned on availability of high-quality labeled datasetsto supervise training deep neural networks. Unfortu-nately, preparing huge labeled datasets is not feasiblefor many situations as data labeling and annotationcan be expensive [29]. Domain adaptation [12] is aparadigm to address the problem of labeled data scarcityin computer vision, where the goal is to improve learningspeed and model generalization as well as to avoidexpensive redundant model retraining. The major idea isto overcome labeled data scarcity in a target domain bytransferring knowledge from a related auxiliary sourcedomain, where labeled data is easy and cheap to obtain.

A common technique in domain adaptation literatureis to embed data from the two source and targetvisual domains in an intermediate embedding space such

1Alexander J. Gabourie is with the Department of Electri-cal Engineering, Stanford University, Stanford, CA, USA aj-gabourie@hrl.com

2Mohammad Rostami is with the Department of Electrical andSystems Engineering, University of Pennsylvania, Philadelphia,PA, US. mrostami@seas.upenn.com

Philip Pope, Soheil Kolouri, and Kyungnam Kim are withInformation and Systems Laboratory, HRL Laboratories, LLC,Malibu, CA, USA pepope, skolouri, kkim @hrl.com

that common cross-domain discriminative relations arecaptured in the embedding space. For example, if thedata from source and target domains have similar class-conditioned probability distributions in the embeddingspace, then a classifier trained solely using labeleddata from the source domain will generalize well ondata points that are drawn from the target domaindistribution [28], [30].

In this paper, we propose a novel unsupervised adap-tation (UDA) algorithm following the above explainedprocedure. Our approach is a simpler, yet effective,alternative for adversarial learning techniques that havebeen more dominant to address probability matchingindirectly for UDA [41], [43], [23]. Our contribution is twofolds. First, we train the shared encoder by minimizingthe Sliced-Wasserstein Distance (SWD) [26] between thesource and the target distributions in the embeddingspace. We also train a classifier network simultaneouslyusing the source domain labeled data. A major benefit ofSWD over alternative probability metrics is that it canbe computed efficiently. Additionally, SWD is knownto be suitable for gradient-based optimization which isessential for deep learning [28]. Our second contributionis to circumvent the class matching challenge [34] byminimizing SWD between conditional distributions insequential iterations for better performance comparedto prior UDA methods that match probabilities explic-itly. At each iteration, we assign pseudo-labels only tothe target domain data that the classifier predicts theassigned class label with high probability and use thisportion of target data to minimize the SWD betweenconditional distributions. As more learning iterationsare performed, the number of target data points withcorrect pseudo-labels grows and progressively enforcesdistributions to align class-conditionally. We provideexperimental results on benchmark problems, includingablation and sensitivity studies, to demonstrate that ourmethod is effective.

II. Background and Related WorkThere are two major approaches in the literature to

address domain adaption. The approach for a group ofmethods is based on preprocessing the target domaindata points. The target data is mapped from the targetdomain to the source domain such that the targetdata structure is preserved in the source [33]. Anothercommon approach is to map data from both domains

2019 57th Annual Allerton Conference on Communication, Control, and Computing(Allerton)Allerton Park and Retreat CenterMonticello, IL, USA, September 24-27, 2019

978-1-7281-3151-1/19/$31.00 ©2019 IEEE 352

to a latent domain invariant space [7]. Early methodswithin the second approach learn a linear subspace asthe invariant space [15], [13] where the target domaindata points distribute similar to the source domain datapoints. A linear subspace is not suitable for capturingcomplex distributions. For this reason, recently deepneural networks have been used to model the interme-diate space as the output of the network. The networkis trained such that the source and the target domaindistributions in its output possess minimal discrepancy.Training procedure can be done both by adversariallearning [14] or directly minimizing the distance betweenthe two distributions [4].

Several important UDA methods use adversarial learn-ing. Ganin et al. [10] pioneered and developed aneffective method to match two distributions indirectlyby using adversarial learning. Liu et al. [21] and Tzenget al. [41] use the Generative Adversarial Networks(GAN) structure [14] to tackle domain adaptation. Theidea is to train two competing (i.e., adversarial) deepneural networks to match the source and the targetdistributions. A generator network maps data pointsfrom both domains to the domain-invariant space anda binary discriminator network is trained to classify thedata points, with each domain considered as a class,based on the representations of the target and the sourcedata points. The generator network is trained such thateventually the discriminator cannot distinguish betweenthe two domains, i.e. classification rate becomes 50%.

A second group of domain adaptation algorithmsmatch the distributions directly in the embedding byusing a shared cross-domain mapping such that thedistance between the two distributions is minimized withrespect to a distance metric [4]. Early methods use simplemetrics such the Maximum Mean Discrepancy (MMD)for this purpose [16]. MMD measures the distancesbetween the distance between distributions simply asthe distance between the mean of embedding features.In contrast, more recent techniques that use a shareddeep encoder, employ the Wasserstein metric [42] toaddress UDA [4], [6]. Wasserstein metric is shown to be amore accurate probability metric and can be minimizedeffectively by deep learning first-order optimization tech-niques. A major benefit of matching distributions directlyis existence of theoretical guarantees. In particular,Redko et al. [28] provided theoretical guarantees for us-ing a Wasserstein metric to address domain adaptation.Additionally, adversarial learning often requires deliber-ate architecture engineering, optimization initialization,and selection of hyper-parameters to be stable [32].In some cases, adversarial learning also suffers from aphenomenon known as mode collapse [22]. That is, if thedata distribution is a multi-modal distribution, which isthe case for most classification problems, the generatornetwork might not generate samples from some modes ofthe distribution. These challenges are easier to addresswhen the distributions are matched directly.

As Wasserstein distance is finding more applicationsin deep learning, efficient computation of Wassersteindistance has become an active area of research. Thereason is that Wasserstein distance is defined in formof a linear programming optimization and solving thisoptimization problem is computationally expensive forhigh-dimensional data. Although computationally effi-cient variations and approximations of the Wassersteindistance have been recently proposed [5], [40], [25], thesevariations still require an additional optimization ineach iteration of the stochastic gradient descent (SGD)steps to match distributions. Courty et al. [4] used aregularized version of the optimal transport for domainadaptation. Seguy et al. [36] used a dual stochasticgradient algorithm for solving the regularized optimaltransport problem. Alternatively, we propose to addressthe above challenges using Sliced Wasserstein Distance(SWD). Definition of SWD is motivated by the fact thatin contrast to higher dimensions, the Wasserstein dis-tance for one-dimensional distributions has a closed formsolution which can be computed efficiently. This factis used to approximate Wasserstein distance by SWD,which is a computationally efficient approximation andhas recently drawn interest from the machine learningand computer vision communities [26], [1], [3], [8], [39].

III. Problem FormulationConsider a source domain, DS = (XS ,YS), with

N labeled samples, i.e. labeled images, where XS =[xs1, . . . ,x

sN ] ∈ X ⊂ Rd×N denotes the samples and YS =

[ys1, ...,ysN ] ∈ Y ⊂ Rk×N contains the corresponding

labels. Note that label ysn identifies the membershipof xsn to one or multiple of the k classes (e.g. digits1, ..., 10 for hand-written digit recognition). We assumethat the source samples are drawn i.i.d. from the sourcejoint probability distribution, i.e. (xsi ,yi) ∼ p(xS , y). Wedenote the source marginal distribution over xS withpS . Additionally, we have a related target domain (e.g.machine-typed digit recognition) with M unlabeled datapoints XT = [xt1, . . . ,xtM ] ∈ Rd×M . Following existingUDA methods, we assume that the same type of labelsin the source domain holds for the target domain. Thetarget samples are drawn from the target marginal dis-tribution xti ∼ pT . We also know that despite similaritybetween these domains, distribution discrepancy existsbetween these two domains, i.e. pS ̸= pT . Our goalis to classify the unlabeled target data points throughknowledge transfer from the source domain. Learning agood classifier for the source data points is a straightforward problem as given a large enough number ofsource samples, N , a parametric function fθ : Rd → Y,e.g., a deep neural network with concatenated learnableparameters θ, can be trained to map samples to theircorresponding labels using standard supervised learningsolely in the source domain. The training is conductedvia minimizing the empirical risk, θ̂ = argminθ êθ =argminθ

∑i L(fθ(xsi ),ysi ), with respect to a proper loss

353

Fig. 1: Architecture of the proposed unsupervised do-main adaptation framework.

function, L(·) (e.g., cross entropy loss). The learnedclassifier fθ̂ generalizes well on testing data points if theyare drawn from the training data point’s distributions.Only then, the empirical risk is a suitable surrogate forthe real risk function, e = E(x,y)∼p(xS ,yS)(L(fθ(x), y)).Hence the naive approach of using fθ̂ on the targetdomain might not be effective as given the discrepancybetween the source and target distributions, fθ̂ mightnot generalize well on the target domain. Therefore,there is a need for adapting the training procedure offθ̂ by incorporating unlabeled target data points suchthat the learned knowledge from the source domain couldbe transferred and used for classification in the targetdomain using only the unlabeled samples.

The main challenge is to circumvent the problem ofdiscrepancy between the source and the target domaindistributions. To that end, the mapping fθ(·) can bedecomposed into a feature extractor ϕv(·) and a classifierhw(·), such that fθ = hw ◦ ϕv, where w and v arethe corresponding learnable parameters, i.e. θ = (w,v).The core idea is to learn the feature extractor function,ϕv, for both domains such that the domain specificdistribution of the extracted features to be similar toone another. The feature extracting function ϕv : X →Z, maps the data points from both domains to anintermediate embedding space Z ⊂ Rf (i.e., featurespace) and the classifier hw : Z → Y maps the datapoints representations in the embedding space to thelabel set. Note that, as a deterministic function, thefeature extractor function ϕv can change the distributionof the data in the embedding. Therefore, if ϕv is learnedsuch that the discrepancy between the source and targetdistributions is minimized in the embedding space, i.e.,discrepancy between pS(ϕ(xs)) and pT (ϕ(xt)) (i.e., theembedding is domain agnostic), then the classifier wouldgeneralize well on the target domain and could be usedto label the target domain data points. This is the coreidea behind various prior domain adaptation approachesin the literature [24], [11].

IV. Proposed MethodWe consider the case where the feature extractor,

ϕv(·), is a deep convolutional encoder with weights v

and the classifier hw(·) is a shallow fully connectedneural network with weights w. The last layer of theclassifier network is a softmax layer that assigns amembership probability distribution to any given datapoint. It is often the case that the labels of data points areassigned according to the class with maximum predictedprobability. In short, the encoder network is learned tomix both domains such that the extracted features inthe embedding are: 1) domain agnostic in terms of datadistributions, and 2) discriminative for the source domainto make learning hw feasible. Figure 1 demonstratessystem level presentation of our framework. Followingthis framework, the UDA reduces to solving the followingoptimization problem to solve for v and w:

minv,w

N∑i=1

L(hw(ϕv(x

si )),y

si

)+ λD

(pS(ϕv(XS)), pT (ϕv(XT ))

),

(1)

where D(·, ·) is a discrepancy measure between theprobabilities and λ is a trade-off parameter. The firstterm in Eq. (1) is empirical risk for classifying the sourcelabeled data points from the embedding space and thesecond term is the cross-domain probability matchingloss. The encoder’s learnable parameters are learnedusing data points from both domains and the classifierparameters are simultaneously learned using the sourcedomain labeled data.

A major remaining question is to select a proper met-ric. First, note that the actual distributions pS(ϕ(XS))and pT (ϕ(XT )) are unknown and we can rely only onobserved samples from these distributions. Therefore,a sensible discrepancy measure, D(·, ·), should be ableto measure the dissimilarity between these distributionsonly based on the drawn samples. In this work, we usethe SWD [27] as it is computationally efficient to com-pute SWD from drawn samples from the correspondingdistributions. More importantly, the SWD is a goodapproximation for the optimal transport [2] which hasgained interest in deep learning community as it is aneffective distribution metric and its gradient is non-vanishing.

The idea behind the SWD is to project two d-dimensional probability distributions into their marginalone-dimensional distributions, i.e., slicing the high-dimensional distributions, and to approximate theWasserstein distance by integrating the Wassersteindistances between the resulting marginal probabilitydistributions over all possible one-dimensional subspaces.For the distribution pS , a one-dimensional slice of thedistribution is defined as:

RpS(t; γ) =∫SpS(x)δ(t− ⟨γ,x⟩)dx, (2)

where δ(·) denotes the Kronecker delta function, ⟨·, ·⟩denotes the vector dot product, Sd−1 is the d-dimensionalunit sphere and γ is the projection direction. In otherwords, RpS(·; γ) is a marginal distribution of pS obtained

354

from integrating pS over the hyperplanes orthogonal toγ. The SWD then can be computed by integrating theWasserstein distance between sliced distributions over allγ:

SW (pS , pT ) =

∫Sd−1

W (RpS(·; γ),RpT (·; γ))dγ (3)

where W (·) denotes the Wasserstein distance. The mainadvantage of using the SWD is that, unlike the Wasser-stein distance, calculation of the SWD does not requirea numerically expensive optimization. This is due tothe fact that the Wasserstein distance between two one-dimensional probability distributions has a closed formsolution and is equal to the ℓp-distance between theinverse of their cumulative distribution functions Sinceonly samples from distributions are available, the one-dimensional Wasserstein distance can be approximatedas the ℓp-distance between the sorted samples [31]. Theintegral in Eq. (3) is approximated using a Monte Carlostyle numerical integration. Doing so, the SWD betweenf -dimensional samples {ϕ(xSi ) ∈ Rf ∼ pS}Mi=1 and{ϕ(xTi ) ∈ Rf ∼ pT }Mj=1 can be approximated as thefollowing sum:

SW 2(pS , pT ) ≈1

L

L∑l=1

M∑i=1

|⟨γl, ϕ(xSsl[i]⟩)−⟨γl, ϕ(xTtl[i]

)⟩|2 (4)

where γl ∈ Sf−1 is uniformly drawn random sample fromthe unit f -dimensional ball Sf−1, and sl[i] and tl[i] arethe sorted indices of {γl ·ϕ(xi)}Mi=1 for source and targetdomains, respectively. Note that for a fixed dimensiond, Monte Carlo approximation error is proportional toO( 1√

L). We utilize the SWD as the discrepancy measure

between the probability distributions to match themin the embedding space. Next, we discuss a majordeficiency in Eq. (1) and our remedy to tackle it. Weutilize the SWD as the discrepancy measure between theprobability densities, pS(ϕ(xS)|Cj)

)and pT (ϕ(xT )|Cj)

).

A. Class-conditional Alignment of DistributionsA main shortcoming of Eq. (1) is that minimizing

the discrepancy between pS(ϕ(XS)) and pT (ϕ(XT ))does not guarantee semantic consistency between thetwo domains. To clarify this point, consider the sourceand target domains to be images corresponding toprinted digits and handwritten digits. While the featuredistributions in the embedding space could have lowdiscrepancy, the classes might not be correctly aligned inthis space, e.g. digits from a class in the target domaincould be matched to a wrong class of the source domainor, even digits from multiple classes in the target domaincould be matched to the cluster of a single digit of thesource domain. In such cases, the source classifier will notgeneralize well on the target domain. In other words, theshared embedding space, Z, might not be a semanticallymeaningful space for the target domain if we solelyminimize SWD between pS(ϕ(XS)) and pT (ϕ(XT )).To solve this challenge, the encoder function shouldbe learned such that the class-conditioned probabilitiesof both domains in the embedding space are similar,

Algorithm 1 DACAD(L, η, λ)1: Input: data DS = (XS ,YS);DT = (XS),2: Pre-training:3: θ̂0 = (w0,v0) = argminθ

∑i L(fθ(x

si ),y

si )

4: for itr = 1, . . . , ITR do5: DPL = {(xti, ŷti)|ŷti = fθ̂(x

ti), p(ŷ

ti |xti) > τ}

6: for alt = 1, . . . , ALT do7: Update encoder parameters using pseudo-labels:8: v̂ =

∑j D

(pS(ϕv(xS)|Cj), pSL(ϕv(xT )|Cj)

)9: Update entire model:

10: v̂, ŵ = argminw,v∑N

i=1 L(hw(ϕ̂̂v(x

si )),y

si

)11: end for12: end for

i.e. pS(ϕ(xS)|Cj) ≈ pT (ϕ(xT )|Cj), where Cj denotesa particular class. Given this, we can mitigate the classmatching problem by using an adapted version of Eq. (1)as:

minv,w

N∑i=1

L(hw(ϕv(x

si )),y

si

)+ λ

k∑j=1

D(pS(ϕv(xS)|Cj), pT (ϕv(xT )|Cj)

),

(5)

where discrepancy between distributions is minimizedconditioned on classes, to enforce semantic alignment inthe embedding space. Solving Eq. (5), however, is nottractable as the labels for the target domain are not avail-able and the conditional distribution, pT (ϕ(xT )|Cj)

), is

not known.To tackle the above issue, we compute a surrogate

of the objective in Eq. (5). Our idea is to approximatepT (ϕ(xT )|Cj)

)by generating pseudo-labels for the tar-

get data points. The pseudo-labels are obtained from thesource classifier prediction, but only for the portion oftarget data points that the the source classifier providesconfident prediction. More specifically, we solve Eq. (5)in incremental gradient descent iterations. In particular,we first initialize the classifier network by training it onthe source data. We then alternate between optimizingthe classification loss for the source data and SWD lossterm at each iteration. At each iteration, we pass thetarget domain data points into the classifier learned onthe source data and analyze the label probability distri-bution on the softmax layer of the classifier. We choose athreshold τ and assign pseudo-labels only to those targetdata points that the classifier predicts the pseudo-labelswith high confidence, i.e. p(yi|xti) > τ . Since the sourceand the target domains are related, it is sensible thatthe source classifier can classify a subset of target datapoints correctly and with high confidence. We use thesedata points to approximate pT (ϕ(xT )|Cj)

)in Eq. (5)

and update the encoder parameters, v, accordingly.In our empirical experiments, we have observed thatbecause the domains are related, as more optimizationiterations are performed, the number of data points withconfident pseudo-labels increases and our approximation

355

for Eq. (5) improves and becomes more stable, enforcingthe source and the target distributions to align classconditionally in the embedding space. As a side benefit,since we math the distributions class-conditionally, aproblem similar to mode collapse is unlikely to occur.Figure 2 visualizes this process using real data. Ourproposed framework, named Domain Adaptation withConditional Alignment of Distributions (DACAD) issummarized in Algorithm 1.

V. Experimental ValidationWe evaluate our algorithm using standard benchmark

UDA tasks and compare against several UDA methods.Datasets: We investigate the empirical performance of

our proposed method on five commonly used benchmarkdatasets in UDA, namely: MNIST (M) [19], USPS(U) [20], Street View House Numbers, i.e., SVHN (S),CIFAR (CI), and STL (ST ). The first three datasetsare 10 class (0-9) digit classification datasets. MNISTand USPS are collection of hand written digits whereasSVHN is a collection of real world RGB images of housenumbers. STL and CIFAR contain RGB images thatshare 9 object classes: airplane, car, bird, cat, deer, dog,horse, ship, and truck. For the digit datasets, while sixdomain adaptation problems can be defined among thesedatasets, prior works often consider four of these sixcases, as knowledge transfer from simple MNIST andUSPS datasets to a more challenging SVHN domain doesnot seem to be tractable. Following the literature, weuse 2000 randomly selected images from MNIST and1800 images from USPS in our experiments for the caseof U → M and S → M [23]. In the remaining cases,we used full datasets. All datasets have their imagesscaled to 32 × 32 pixels and the SVHN images areconverted to grayscale as the encoder network is sharedbetween the domains. CIFAR and STL maintain theirRGB components. We report the target classificationaccuracy across the tasks.

Pre-training: Our experiments involve a pre-trainingstage to initialize the encoder and the classifier networkssolely using the source data. This is an essential stepbecause the combined deep network can generate confi-dent pseudo-labels on the target domain only if initiallytrained on the related source domain. In other words, thisinitially learned network can be served as a naive modelon the target domain. We then boost the performanceon the target domain using our proposed algorithm,demonstrating that our algorithm is indeed effectivefor transferring knowledge. Doing so, we investigatea less-explored issue in the UDA literature. DifferentUDA approaches use considerably different networks,both in terms of complexity, e.g. number of layers andconvolution filters, and the structure, e.g. using an auto-encoder. Consequently, it is ambiguous whether perfor-mance of a particular UDA algorithm is due to successfulknowledge transfer from the source domain or just agood baseline network that performs well on the target

domain even without considerable knowledge transferfrom the source domain. To highlight that our algorithmcan indeed transfer knowledge, we use two different net-work architectures: DRCN [11] and VGG [38]. We thenshow that our algorithm can effectively boost base-lineperformance (statistically significant) regardless of theunderlying network. In most of the domain adaptationtasks, we demonstrate that this boost indeed stems fromtransferring knowledge from the source domain. In ourexperiments we used Adam optimizer [18] and set thepseudo-labeling threshold to tr = 0.99.

Data Augmentation: Following the literature, we usedata augmentation to create additional training databy applying reasonable transformations to input datain an effort to improve generalization [37]. Confirmingthe reported result in [11], we also found that geometrictransformations and noise, applied to appropriate inputs,greatly improves performance and transferability of thesource model to the target data. Data augmentationcan help to reduce the domain shift between the twodomains. The augmentations in this work are limitedto translation, rotation, skew, zoom, Gaussian noise,Binomial noise, and inverted pixels.

A. ResultsFigure 2 demonstrates how our algorithm successfully

learns an embedding with class-conditional alignment ofdistributions of both domains. This figure presents thetwo-dimensional t_SNE visualization of the source andtarget domain data points in the shared embedding spacefor the S → U task. The horizontal axis demonstratesthe optimization iterations where each cell presents datavisualization after a particular optimization iteration isperformed. The top sub-figures visualize the source datapoints, where each color represents a particular class.The bottom sub-figures visualize the target data points,where the colored data points represent the pseudo-labeled data points at each iteration and the black pointsrepresent the rest of the target domain data points.We can see that, due to pre-training initialization, theembedding space is discriminative for the source domainin the beginning, but the target distribution differsfrom the source distributions. However, the classifieris confident about a portion of target data points. Asmore optimization iterations are performed, since thenetwork becomes a better classifier for the target domain,the number of the target pseudo-labeled data pointsincrease, improving our approximate of Eq. 5. As aresult, the discrepancy between the two distributionsprogressively decreases. Over time, our algorithm learnsa shared embedding which is discriminative for bothdomains, making pseudo-labels a good prediction for theoriginal labels, bottom, right-most sub-figure. This resultempirically validates applicability of our algorithm toaddress UDA.

We also compare our results against several recentUDA algorithms in Table I. In particular, we compare

356

SharedEncoder,!

SharedEmbedding,"SourceClassifier, ℎ

Inputs$%& $'(

SourceDomain

LabeledDataset

TargetDomain

UnlabeledDataset

Labels Pseudo-Labels

…

…

SourceDomain

TargetDomain

GroundTruthFinalpredictionConfidentpseudolabelsUnlabeledsamples

Fig. 2: The high-level system architecture, shown on the left, illustrates the data paths used during UDA training.On the right, t_SNE visualizations demonstrate how the embedding space evolves during training for the S → Utask. In the target domain, colored points are examples with assigned pseudo-labels, which increase in number withthe confidence of the classifier.

Method M → U U → M S → M S → U ST → CI CI → STGtA 92.8±0.9 90.8±1.3 92.4±0.9 - - -

CoGAN 91.2±0.8 89.1±0.8 - - - -ADDA 89.4±0.2 90.1±0.8 76.0±1.8 - - -

CyCADA 95.6±0.2† 96.5±0.1† 90.4±0.4 - - -I2I-Adapt 92.1† 87.2† 80.3 - - -

FADA 89.1 81.1 72.8 78.3 - -RevGrad 77.1±1.8‡ 73.0±2.0‡ 73.9 - - -DRCN 91.8±0.1† 73.7±0.04† 82.0±0.2 - 58.9±0.1 66.4±0.1AUDA - - 86.0 - - -OPDA 70.0 60.2 - - - -MML 77.9† 60.5† 62.9 - - -

Target (FS) 96.8±0.2 98.5±0.2 98.5±0.2 96.8±0.2 81.5±1.0 64.8±1.7VGG 90.1±2.6 80.2±5.7 67.3±2.6 66.7±2.7 53.8±1.4 63.4±1.2

DACAD 92.4±1.2 91.1±3.0 80.0±2.7 79.6±3.3 54.4±1.9 66.5±1.0DACAD 93.6±1.0 95.8±1.3 82.0±3.4 78.0±2.0 44.4±0.4 65.7±1.0

DRCN (Ours) 88.6±1.3 89.6±1.3 74.3±2.8 54.9±1.8 50.0±1.5 64.2±1.7DACAD 94.5±0.7 97.7±0.3 88.2±2.8 82.6±2.9 55.0±1.4 65.9±1.4

TABLE I: Classification accuracy for UDA between MINIST, USPS, SVHN, CIFAR, and STL datasets. † indicatesuse of full MNIST and USPS datasets as opposed to the subset described in the paper. ‡ indicates results fromreimplementation in [41].

against the recent adversarial learning algorithms: Gen-erate to Adapt (GtA) [35], CoGAN [21], ADDA [41],CyCADA [17], and I2I-Adapt [24]. We also includeFADA [23], which is originally a few-shot learningtechnique. For FADA, we list the reported one-shotaccuracy, which is very close to the UDA setting (butit is arguably a simpler problem). Additionally, we haveincluded results for RevGrad [9], DRCN [11], AUDA [34],OPDA [4], MML [36]. The latter methods are similar toour method because these methods learn an embeddingspace to couple the domains. OPDA and MML aremore similar as they match distributions explicitly inthe learned embedding. Finally, we have included theperformance of fully-supervised (FS) learning on thetarget domain as an upper-bound for UDA. In our ownresults, we include the baseline target performance thatwe obtain by naively employing a DRCN network aswell as target performance from VGG network that arelearned solely on the source domain. We notice that inTable I, our baseline performance is better than some of

the UDA algorithms for some tasks. This is a very crucialobservation as it demonstrates that, in some cases, atrained deep network with good data augmentation canextract domain invariant features that make domainadaptation feasible even without any further transferlearning procedure. The last row demonstrates that ourmethod is effective in transferring knowledge to boost thebaseline performance. In other words, Table I serves asan ablation study to demonstrate that that effectivenessof our algorithm stems from successful cross-domainknowledge transfer. We can see that our algorithm leadsto near- or the state-of-the-art performance across thetasks. Additionally, an important observation is thatour method significantly outperforms the methods thatmatch distributions directly and is competent againstmethods that use adversarial learning. This can beexplained as the result of matching distributions class-conditionally and suggests our second contribution canpotentially boost performance of these methods. Finally,we note that our proposed method provide a statistically

357

significant boost in all but two of the cases (shown ingray in Table I).

VI. Conclusions and DiscussionWe developed a new UDA algorithm based on learning

a domain-invariant embedding space. We map datapoints from two related domains to the embedding spacesuch that discrepancy between the transformed distri-butions is minimized. We used the sliced Wassersteindistance metric as a measure to match the distributionsin the embedding space. As a result, our method is com-putationally more efficient. Additionally, we matcheddistributions class-conditionally by assigning pseudo-labels to the target domain data. As a result, our methodis more robust and outperforms prior UDA methods thatmatch distributions directly. We provided experimentalvalidations to demonstrate that our method is competentagainst SOA recent UDA methods.

References[1] N. Bonneel, J. Rabin, G. Peyré, and H. Pfister. Sliced

and Radon Wasserstein barycenters of measures. Journal ofMathematical Imaging and Vision, 51(1):22–45, 2015.

[2] N. Bonnotte. Unidimensional and evolution methods foroptimal transportation. PhD thesis, Paris 11, 2013.

[3] Mathieu Carriere, Marco Cuturi, and Steve Oudot. Slicedwasserstein kernel for persistence diagrams. arXiv preprintarXiv:1706.03358, 2017.

[4] N. Courty, R. Flamary, D. Tuia, and A. Rakotomamonjy.Optimal transport for domain adaptation. IEEE TPAMI,39(9):1853–1865, 2017.

[5] M. Cuturi. Sinkhorn distances: Lightspeed computation of op-timal transport. In Advances in neural information processingsystems, pages 2292–2300, 2013.

[6] B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, andN. Courty. Deepjdot: Deep joint distribution optimal trans-port for unsupervised domain adaptation. arXiv preprintarXiv:1803.10081, 2018.

[7] H. Daumé III. Frustratingly easy domain adaptation. arXivpreprint arXiv:0907.1815, 2009.

[8] I. Deshpande, Z. Zhang, and A. Schwing. Generative modelingusing the sliced wasserstein distance. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recogni-tion, pages 3483–3491, 2018.

[9] Y. Ganin and V. Lempitsky. Unsupervised domain adaptationby backpropagation. In ICML, 2014.

[10] Y. Ganin, E. Ustinova, H. Ajakan, P.l Germain, H. Larochelle,F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. The Journal ofMachine Learning Research, 17(1):2096–2030, 2016.

[11] M. Ghifary, B. Kleijn, M. Zhang, D. Balduzzi, and W. Li.Deep reconstruction-classification networks for unsuperviseddomain adaptation. In European Conference on ComputerVision, pages 597–613. Springer, 2016.

[12] X. Glorot, A. Bordes, and Y. Bengio. Domain adaptation forlarge-scale sentiment classification: A deep learning approach.In ICML, pages 513–520, 2011.

[13] B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow ker-nel for unsupervised domain adaptation. In Computer Visionand Pattern Recognition (CVPR), 2012 IEEE Conference on,pages 2066–2073. IEEE, 2012.

[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generativeadversarial nets. In Advances in neural information processingsystems, pages 2672–2680, 2014.

[15] R. Gopalan, R. Li, and R. Chellappa. Domain adaptation forobject recognition: An unsupervised approach. In ComputerVision (ICCV), 2011 IEEE International Conference on, pages999–1006. IEEE, 2011.

[16] A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borg-wardt, and B. Schölkopf. Covariate shift by kernel meanmatching, 2009.

[17] J. Hoffman, E. Tzeng, T. Park, J. Zhu, P. Isola, K. Saenko,A. A. Efros, and T. Darrell. Cycada: Cycle-consistent adver-sarial domain adaptation. In ICML, 2018.

[18] D. P. Kingma and J. Ba. Adam: A method for stochasticoptimization. arXiv preprint arXiv:1412.6980, 2014.

[19] Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E.Howard, W. E. Hubbard, and L. D. Jackel. Handwritten digitrecognition with a back-propagation network. In Advances inneural information processing systems, pages 396–404, 1990.

[20] Y. LeCun, L. D. Jackel, L. Bottou, A. Brunot, C. Cortes,J. S. Denker, H. Drucker, I. Guyon, U. A. Muller, E. Sackinger,et al. Comparison of learning algorithms for handwritten digitrecognition. In International conference on artificial neuralnetworks, volume 60, pages 53–60. Perth, Australia, 1995.

[21] M. Liu and O. Tuzel. Coupled generative adversarial networks.In Advances in neural information processing systems, pages469–477, 2016.

[22] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein. Un-rolled generative adversarial networks. arXiv preprintarXiv:1611.02163, 2016.

[23] S. Motiian, Q. Jones, S. Iranmanesh, and G. Doretto. Few-shot adversarial domain adaptation. In Advances in NeuralInformation Processing Systems, pages 6670–6680, 2017.

[24] Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, andK. Kim. Image to image translation for domain adaptation.arXiv preprint arXiv:1712.00479, 2017.

[25] A. Oberman and Y. Ruan. An efficient linear program-ming method for optimal transportation. arXiv preprintarXiv:1509.03668, 2015.

[26] J. Rabin, G. Peyré, J. Delon, and M. Bernot. Wassersteinbarycenter and its application to texture mixing. In Interna-tional Conference on Scale Space and Variational Methods inComputer Vision, pages 435–446. Springer, 2011.

[27] J. Rabin, G. Peyré, J. Delon, and M. Bernot. Wassersteinbarycenter and its application to texture mixing. In Interna-tional Conference on Scale Space and Variational Methods inComputer Vision, pages 435–446. Springer, 2011.

[28] A. Redko, I.and Habrard and M. Sebban. Theoretical analysisof domain adaptation with optimal transport. In JointEuropean Conference on Machine Learning and KnowledgeDiscovery in Databases, pages 737–753. Springer, 2017.

[29] M. Rostami, D. Huber, and T. Lu. A crowdsourcing triagealgorithm for geopolitical event forecasting. In Proceedings ofthe 12th ACM Conference on Recommender Systems, pages377–381. ACM, 2018.

[30] M. Rostami, S. Kolouri, E. Eaton, and K. Kim. Deep transferlearning for few-shot sar image classification. Remote Sensing,11(11):1374, 2019.

[31] M. Rostami, S. Kolouri, E. Eaton, and K. Kim. Sar imageclassification using few-shot cross-domain transfer learning.In Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition Workshops, pages 0–0, 2019.

[32] K. Roth, A. Lucchi, S. Nowozin, and T. Hofmann. Stabilizingtraining of generative adversarial networks through regulariza-tion. In Advances in Neural Information Processing Systems,pages 2018–2028, 2017.

[33] K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visualcategory models to new domains. In European conference oncomputer vision, pages 213–226. Springer, 2010.

[34] K. Saito, Y. Ushiku, and T. Harada. Asymmetric tri-trainingfor unsupervised domain adaptation. In ICML, 2018.

[35] S. Sankaranarayanan, Y. Balaji, C. D Castillo, and R. Chel-lappa. Generate to adapt: Aligning domains using generativeadversarial networks. In CVPR, 2018.

[36] V. Seguy, B. B. Damodaran, R. Flamary, N. Courty, A. Rolet,and M. Blondel. Large-scale optimal transport and mappingestimation. In ICLR, 2018.

[37] P. Y. Simard, D. Steinkraus, and J. C. Platt. Best practicesfor convolutional neural networks applied to visual documentanalysis. In Seventh International Conference on DocumentAnalysis and Recognition, 2003. Proceedings., pages 958–963,Aug 2003.

358

[38] K. Simonyan and A. Zisserman. Very deep convolutionalnetworks for large-scale image recognition. arXiv preprintarXiv:1409.1556, 2014.

[39] Umut Şimşekli, Antoine Liutkus, Szymon Majewski, and AlainDurmus. Sliced-wasserstein flows: Nonparametric generativemodeling via optimal transport and diffusions. arXiv preprintarXiv:1806.08141, 2018.

[40] J Solomon, F. De Goes, G. Peyré, M. Cuturi, A. Butscher,A. Nguyen, T. Du, and L. Guibas. Convolutional wasser-stein distances: Efficient optimal transportation on geometricdomains. ACM Transactions on Graphics (TOG), 34(4):66,2015.

[41] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarialdiscriminative domain adaptation. In Computer Vision andPattern Recognition (CVPR), volume 1, page 4, 2017.

[42] C. Villani. Optimal transport: old and new, volume 338.Springer Science & Business Media, 2008.

[43] Z. Yi, H. Zhang, P. Tan, and M. Gong. Dualgan: Unsuperviseddual learning for image-to-image translation. In ICCV, pages2868–2876, 2017.

359