Parts4Feature: Learning 3D Global Features from Generally ... · Parts4Feature: Learning 3D Global...

Parts4Feature: Learning 3D Global Features from Generally Semantic Parts inMultiple Views

Zhizhong Han1,3 , Xinhai Liu1,2 , Yu-Shen Liu1,2∗ and Matthias Zwicker31School of Software, Tsinghua University, Beijing, China

2Beijing National Research Center for Information Science and Technology (BNRist)3Department of Computer Science, University of Maryland, College Park, USA

[email protected], [email protected], [email protected], [email protected]

AbstractDeep learning has achieved remarkable results in3D shape analysis by learning global shape featuresfrom the pixel-level over multiple views. Previ-ous methods, however, compute low-level featuresfor entire views without considering part-level in-formation. In contrast, we propose a deep neuralnetwork, called Parts4Feature, to learn 3D glob-al features from part-level information in multipleviews. We introduce a novel definition of gener-ally semantic parts, which Parts4Feature learns todetect in multiple views from different 3D shapesegmentation benchmarks. A key idea of our ar-chitecture is that it transfers the ability to detec-t semantically meaningful parts in multiple viewsto learn 3D global features. Parts4Feature achievesthis by combining a local part detection branch anda global feature learning branch with a shared re-gion proposal module. The global feature learn-ing branch aggregates the detected parts in terms oflearned part patterns with a novel multi-attentionmechanism, while the region proposal module en-ables locally and globally discriminative informa-tion to be promoted by each other. We demonstratethat Parts4Feature outperforms the state-of-the-artunder three large-scale 3D shape benchmarks.

1 IntroductionLearning 3D global features from multiple views is an effec-tive approach for 3D shape understanding. A widely adoptedstrategy is to leverage deep neural networks to aggregate fea-tures hierarchically extracted from pixel-level information ineach view. However, current approaches can not employ part-level information. In this paper, we show for the first timehow extracting part-level information over multiple views canbe leveraged to learn 3D global features. We demonstratethat this approach further increases the discriminability of 3Dglobal features and outperforms the state-of-the-art methodson large scale 3D shape benchmarks.It is intuitive that learning to detect and localize semantic

parts could help classify shapes more accurately. Previous∗Corresponding Author

studies on fine-grained image recognition also employ thisintuition by combining local part detection and global featurelearning together. To learn highly discriminative features todistinguish subordinate categories, these methods try to firstdetect important parts, such as heads, wings and tails of bird-s, and then collect these part features into a global feature.However, these methods do not tackle the challenges that weare facing in the 3D domain. First, these methods requireground truth parts with specified semantic labels, while 3Dshape classification benchmarks do not provide such kind oflabels. Second, the part detection knowledge learned by thesemethods cannot be transferred for general purpose use, suchas non-fine-grained image classification, since it is specifiedfor particular shape classes. Third, these methods are not de-signed to aggregate part information from multiple images,corresponding to multiple views of a 3D shape in our scenari-o. Therefore, simultaneously learning part detection and fur-ther aggregating part-level information from multiple viewsbecome a unique challenge in 3D global feature learning.To address these issues, we propose Parts4Feature, a deep

neural network to learn 3D global features from semanticparts in multiple views. With a novel definition of generallysemantic parts (GSPs), Parts4Feature learns to detect GSPs inmultiple views from different 3D shape segmentation bench-marks. Moreover, it learns a 3D global feature from shapeclassification data sets, by transferring the learned knowledgeof part detection, and leveraging the detected GSPs in mul-tiple views. Specifically, Parts4Feature is mainly composedof a local part detection branch and a global feature learningbranch. Both branches share a region proposal module, whichenables locally and globally discriminative information to getpromoted by each other.The local part detection branch employs a novel neural net-

work derived from Fast R-CNN [Girshick, 2015] to learn todetect and localize GSPs in multiple views. In addition, theglobal feature learning branch incrementally aggregates thedetected parts in terms of learned part patterns with multi-attention. We propose a novel multi-attention mechanism tofurther increase the discriminability of learned features by notonly highlighting the distinctive parts and part patterns but al-so depressing the ambiguous ones. Our novel view aggrega-tion based on semantic parts prevents information loss causedby the widely used pooling, and it can understand each detect-ed part in a more detailed manner. In summary, our contribu-

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

766

tions are as follows:

i) We propose Parts4Feature, a novel deep neural networkto learn 3D global features from semantic parts in mul-tiple views, by combining part detection and global fea-ture learning together.

ii) We show that the novel structure of Parts4Feature is ca-pable of learning and transferring universal knowledgeof part detection, which allows Parts4Feature to lever-age discriminative information from another source (3Dshape segmentation) for 3D global feature learning.

iii) Our global feature learning branch introduces a novelview aggregation based on semantic parts, where theproposed multi-attention further improves the discrim-inability of learned features.

2 Related Work2.1 Mesh-based Deep Learning ModelsTo directly learn 3D features from 3D meshes, different novelconcepts, such as circle convolution [Han and others, 2016],mesh convolution [Han and others, 2017] were proposed toperform in deep learning models. These methods aim to learnglobal or local features from the geometry and spatial infor-mation on meshes to understand 3D shapes.

2.2 Voxel-based Deep Learning ModelsSimilar to images, voxels have regular structure tobe learned by deep learning models, such as CRB-M [Wu and others, 2015], fully convolutional denoisingautoencoders [Sharma et al., 2016], CNNs [Qi et al., 2016],GAN [Wu and others, 2016]. These methods usu-ally employ 3D convolution to better capture thecontextual information in local regions. Moreover,Tags2Parts [Muralikrishnan et al., 2018] discovered se-mantic regions that strongly correlate with user-prescribedtags by learning from voxels using a novel U-Net.

2.3 Deep Learning Models for Point CloudsAs a series of pioneering work, Point-Net++ [Qi and others, 2017] inspired various supervisedmethods to understand point clouds. Through self-reconstruction, FoldingNet [Yang et al., 2018] and Latent-GAN [Achlioptas and others, 2018] learned global featureswith different unsupervised strategies.

2.4 View-based Deep Learning ModelsSimilar to the light field descriptor (LFD),GIFT [Bai and others, 2017] measured the differ-ence between two 3D shapes using their correspond-ing view feature sets. Moreover, pooling panoramaviews [Shi and others, 2015; Sfikas and others, 2017] orrendered views [Su and others, 2015; Han et al., 2019] ismore widely used to learn global features. Different improve-ments from camera trajectories [Johns et al., 2016], viewaggregation [Wang et al., 2017; Han and others, 2019a],pose estimation [Kanezaki et al., 2018] are also presented.However, these methods can not leverage part-level informa-tion. In contrast, Parts4Feature learns and transfers universal

knowledge of part detection to facilitate 3D global featurelearning.

3 Parts4Feature3.1 OverviewParts4Feature consists of three main components as shown inFig. 1: a local part detection branch L, a global feature learn-ing branch G, and a region proposal module R, where R isshared by L and G and receives multiple views of a 3D shapeas input. We train Parts4Feature simultaneously under a lo-cal part detection benchmark Φ and a global feature learningbenchmark Ψ. The local part detection branch L learns toidentify GSPs in multiple views under Φ, while G learns aglobal feature from the detected GSPs in multiple views un-derΨ.For a 3D shape m in either Φ or Ψ, we capture V views

vi around it, forming a view sequence v = {vi|i ∈ [1, V ]}.First, the region proposal module R provides the features f i

j

of regions rij proposed in each view vi, where j ∈ [1, J ].Then, by analyzing the region features f i

j in v, branch Llearns to predict what and where GSPs are in multiple views.Finally, by aggregating the features f i

k of the top K regionproposals rij in each vi in v, the global feature learningbranch G produces the global feature f of shape m. Our ap-proach to aggregating region proposal features is based on Nsemantic part patterns θn with multi-attention for 3D shapeclassification, where θn are learned across all views in theglobal feature learning benchmarkΨ.

3.2 Generally Semantic PartsWe define a GSP as a local part in any semantic part categoryof any shape class, such as engines of airplanes or wheel-s of cars. Although our concept of GSPs simplifies all se-mantic part categories into a binary label by only determiningwhether a part is semantic or not, this allows us to exploit dis-criminative, part-level information from several different 3Dshape segmentation benchmarks for global feature learning.We use three 3D shape segmentation benchmarks involved

in [Kalogerakis and others, 2017], including ShapeNetCore,Labeled-PSB, and COSEG to construct the local part detec-tion benchmarkΦ and provide ground truth GSPs. We also s-plit the 3D shapes in each segmentation benchmark into train-ing and test sets according to [Kalogerakis and others, 2017].Fig. 2 shows the construction of ground truth GSPs. Foreach view vi of a 3D shape m shown in Fig. 2(a), we ob-tain its ground truth segmentation visualized in Fig. 2(b) fromthe shape segmentation benchmark. Then, we can isolateeach part category to precisely locate GSPs, as shown fromFig. 2(c) to Fig. 2(f). We emphasize each isolated part cat-egory in blue, where we locate the corresponding GSPs bycomputing the bounding box (red) of the colored regions. Fi-nally, we show all GSPs in view vi in Fig. 2(g). We collectall GSPs of shape m by repeating these procedures in all it-s V views. Note that we omit small GSPs (for example thelanding gear in Fig. 2(f)) whose bounding boxes are smallerthan 0.45 of the max bounding box in the same part category.


767

......

Deep

con

volu

tion

al n

etwork

......

Reg

ion

pro

posa

l netw

ork

an

d R

oI p

oolin

g

Reg

ion

featu

re ma

ps

......

......

......

All

Top

K

Top K regions

Top K regions

FC layers

FC layers

......

Region proposal module shared by branch L and G

Sh

ared

by m

ultip

le view

s

Global feature learning branch

Parts

aggreg

atio

n

Patternsattention

Pattern

saggreg

atio

nGlobalfeature

FC

layer

FC

layer

Sigmoid(semantic?)

Bboxregressor

Partsattention

Part patternlearning

Local part detection branch

View

featu

re ma

ps

R: G:

L:

V views

v1

vV

Figure 1: The demonstration of Parts4Feature framework.

(f)

(g)(a)

Segment truth

Mer

ge

loca

l p

art

s

(b)

(c) (d) (e)

Isola

te categ

ories

Figure 2: The illustration of ground truth GSPs construction.

3.3 Region Proposal Module RR provides region proposals rij in all views vi and their fea-tures f i

j , which are then forwarded to the local part detectionand global feature learning branches. Shared by all views viin v, R is composed of a Deep Convolutional Network (D-CN), and a Region Proposal Network (RPN) with Region ofInterest (RoI) pooling [Girshick, 2015].DCN is modified from a VGG CNN M 1024 net-

work [Chatfield et al., 2014], and it produces feature f i foreach view vi as 512 feature maps of size 12 × 12. Based onf i, RPN then proposes regions rij in a sliding-window man-ner. At each sliding-window location centered at each pixel off i, a region rij is proposed by determining its location tR andpredicting GSP probabilities pR with an anchor. The locationtR is a four dimensional vector representing center, width andheight of the bounding box. We use 6 scales and 3 aspect ra-tios to yield 6× 3 = 18 anchors, which ensures a wide rangeof sizes to accommodate region proposals for GSPs that maybe partially occluded in some views. The 6 scales relative tothe size of the views are [1, 2, 4, 8, 16, 32], and the 3 aspectratios are 1 : 1, 1 : 2, and 2 : 1. Altogether, this leads toJ = 2592 = 12× 12× 18 regions rij in each view vi.To train RPN to predict GSP probabilities pR, we assign

a binary label to each region rij indicating whether rij is a

GSP. We assign a positive label if the Intersection-over-Union(IoU) overlap between rij and any ground-truth GSP in vi

is higher than a threshold SR, and we use a negative labelotherwise. In each view vi we apply RoI pooling over regionsgiven by {tR} on feature maps f i. Hence, the features f i

j ofall J region proposals rij are 512×7×7 dimensional vectors,which we forward to the local part detection branch L. Inaddition, we provide the features f i

k of the top K regions rijaccording to their GSP probability pR to the global featurelearning branch G.

3.4 Local Part Detection Branch L

The purpose of this branch is to detect GSPs from the J re-gion proposals rij in each view vi. We employ L as an en-hancer of RPN, where L aims to learn what and where GSPsare in vi without anchors in a more precise manner. The in-tuition is that this in turn pushes RPN to propose more GSP-like regions, which we provide to the global feature learningbranch G.We feed the region features f i

j of rij into a sequence offully connected layers followed by two output layers. Thefirst one estimates the GSP probability pL that rij is a GSPusing a sigmoid function as an indicator. The second onepredicts the corresponding part location tL using a boundingbox regressor, where tL represents the same bounding boxparameters as tR in RPN. Similar to the threshold SR in R, Lemploys another threshold SL to assign positive and negativelabels for training. Denoting the ground truth probabilitiesand locations of positive and negative samples in RPNR as p′and t′, and similarly for L as p′′ and t′′, the objective functionof Parts4Feature for GSP detection is formed by the loss in Rand L, which is defined for each region proposal as follows,

OR+L(pR, pL, p′, p′′, tR, tL, t

′, t′′) =

Op(pR, p′) + λOt(tR, t

′) +Op(pL, p′′) + λOt(tL, t

′′),(1)

where Op measures the accuracy in terms of GSP probabili-ty by the cross-entropy function of positive labels, while Ot

measures the accuracy in terms of location by the robust L1

function as in [Ren and others, 2015]. The parameter λ bal-ances Op and Ot in both R and L. It works well in our exper-iments with a value of 1. In summary, Parts4Feature has the


768

powerful ability to detect GSPs by simultaneously leveragingthe view-level features f i in R and the part-level features f i

jin L, which addresses the difficulty of GSP detection frommultiple views caused by rotation and occlusion effects.

3.5 Global Feature Learning Branch G

This branch learns to map the features f ik of the topK region

proposals rik in each view vi in v to the 3D global feature f .To avoid information loss caused by widely used pooling foraggregation, G incrementally aggregates all V × K regionfeatures f i

k in terms of semantic part patterns θn with multi-attention, where we learn the patterns θn across all trainingdata in the global feature learning benchmarkΨ. The motiva-tion for learning part patterns to aggregate regions is that theappearance of GSPs is so various that it would limit the dis-criminability of global features f . Our multi-attention mech-anism includes attention weights for view aggregation on thepart-level and the part-pattern-level, denoted by α and β, re-spectively. Here, α models how each of the N patterns θnweights each of the V ×K regions rik, while β measures howthe final, global feature f weights each of the N patterns θn.Specifically, we employ a single-layer perceptron to learn

θn, where θn has the same dimension as f ik. α is a (V ×

K) × N matrix, where each entry α((i, k), n) is the atten-tion paid to each of the (V ×K) regions rik by the n-th pat-tern θn. α((i, k), n) is measured by a softmax function asexp(wT

nfik+ bn)/

∑n′∈[1,N ] exp(w

Tn′f i

k+ bn′). Withα, wefirst aggregate all (V × K) region features f i

k into a patternspecific aggregation φn in terms of each pattern θn by com-puting

∑i∈[1,V ],k∈[1,K] α((i, k), n)(θn − f i

k). Then, we fur-ther aggregate allN pattern specific aggregations φn into thefinal, global feature f of 3D shape m. This is performed bylinear weighting with the N dimensional vector β, such thatf =

∑n∈[1,N ] β(n)φn. For clarity of exposition, we explain

the details of how we obtain β further below.Finally, we use f to classify m into one of C shape

classes by a softmax function after a fully connected layer,where the softmax function outputs the classification prob-abilities p, such that each probability p(c) is defined asexp(uT

c f + ac)/∑

c′∈[1,C] exp(uTc′f + ac′). The objective

function of G is the cross entropy between p and the groundtruth probability p′,

OG(p,p′) = −

∑c∈[1,C]

p′(c)logp(c). (2)

The intuition behind modelling part-pattern-level attentionis to enable Parts4Feature to weight the pattern specific ag-gregations φn according to the 3D shape characteristics thatit has learned. This leads Parts4Feature to differentiate shapesin detail. To implement this, β is designed to capture the sim-ilarities between each of the N pattern specific aggregationsφn and the C shape classes. To represent the characteristicsof C shape classes, we propose to employ the weights uc inthe fully connected layer before the last softmax function, asillustrated in Fig. 1. We first project φn and uc into a com-mon space using matrices W1 and W2. Then we computenormalized similarities using a linear mapping with w and gas follows, β = softmax((W1[φ

Tn ]N +W2[u

Tc ]C)w+ g),

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3: The detected GSPs with pL > 0.6 in red boxes. (a) Chair.(b) Toilet. (c) Dresser. (d) Table. (e) Sofa. (f) Bed.

where learnable parameters W1 and W2 are N × N andN × C dimensional matrices, w and g are d and N dimen-sional vectors, [•]◦ means stacking all ◦ vectors • into a ma-trix row by row.

3.6 TrainingWe train R and L together under a local part detection bench-markΦ, andG under a global feature learning benchmarkΨ.The Parts4Feature objective is to simultaneously minimize E-q. 1 and Eq. 2, which leads to the loss

O =1

∥Φ∥∑

(p,t)∈ΦOR+L(pR, pL, p

′, p′′, tR, tL, t′, t′′)

+η

∥Ψ∥∑

p∈ΨOG(p,p

′),

(3)where the number of samples ∥·∥ is a normalization factor andη is a balance parameter. Since R and L are based on the ob-ject detection architecture of Fast R-CNN [Girshick, 2015],we adopt the approximate approach in [Ren and others, 2015]to jointly train R and L fast. In addition, we simultaneouslyupdate uc in the softmax classifier in G by ∂OG/∂uc and∂β/∂uc. This enables uc to be learned more flexibly for op-timization convergence, which is a connection across G. FortheΦ = Ψ case, parameters in R, L andG can be simultane-ously updated, otherwise, they are updated alternatively. Forexample, parameters in R and L are first updated under Φ,then, parameters in R (except RPN) and G are updated un-der Ψ, and this process is iterated until convergence. In ourfollowing experiments we use η = 1 .

4 Experiments and Analysis4.1 ParametersWe investigate how some important parameters affec-t Parts4Feature in shape classification under Model-Net [Wu and others, 2015].We first explore the IoU thresholds SR in R and SL in L

that are used to establish positive GSP samples using Mod-elNet40 [Wu and others, 2015] as Ψ, as shown in Table 1,where we initially use V = 12 views, K = 20 regions, andN = 256 patterns. With SR = 0.7 and increasing SL from0.5 to 0.8, the mean Average Presicion (mAP) under the test


769

Metrics (0.7, 0.5) (0.7, 0.6) (0.7, 0.7) (0.7, 0.8) (0.5, 0.5) (0.8, 0.5) (0.6, 0.6)mAP 77.28 69.35 66.12 56.97 75.39 72.32 69.51Acc 93.40 93.15 92.38 92.50 92.67 92.71 92.95

Table 1: The effects of SR and SL on the performance of Parts4Feature under ModelNet40.

Metric SL = 0.5 0.7 0.8 K = 10 30 N = 128 512 V = 3 6Acc 95.26 96.15 94.38 94.38 94.93 94.49 95.04 94.27 94.93

Table 2: The effects of K, N and d on the performance of Parts4Feature under ModelNet10.

Pooling AttentionMethods Acc Methods AccMaxPool 90.97 NoAtt 92.84MeanPool 92.18 PtAtt 93.17NetVLAD 93.50 PnAtt 93.72

No L 93.61 MultiAtt 96.15

Table 3: The view aggregation and attention comparison.

set of Φ decreases, and accordingly, the average instance ac-curacy under the test set of Ψ decreases, compared to thehighest classification accuracy 93.40%. With SL = 0.5, wealso decrease SR to 0.5 and increase it to 0.8 respectively. ThemAP only slightly drops from 77.28 to 75.39 and 72.32, al-though the corresponding accuracy decreases too. However,the mAP and the accuracy are not strictly positive correlated,as shown by “(0.6,0.6)”, which has lower mAP but higher ac-curacy than “(0.8,0.5)” and “(0.5,0.5)”. This comparison alsoimplies that SL affects part detection more than SR.Next, we apply the parameters setting “(0.7,0.5)” under

ModelNet10 [Wu and others, 2015], as shown by the first ac-curacy of 95.26% in Table 2. Increasing SL to 0.7 leads to aneven better result of 96.15%. We also find the slight effect ofK, N , and V on the performance.We visualize part detection and multi-attention involved in

our best result under ModelNet10 in Fig. 3 and Fig. 4, respec-tively. Although there are no ground truth GSPs under Mod-elNet10, Parts4Feature still successfully transfers the part de-tection knowledge learned fromΦ to detect GSPs in multipleviews. Moreover, β is learned to focus on the patterns withhigh part attentions in α, where the top-6 patterns with highpart attentions in α are shown below for clarity.

4.2 Ablation StudyFinally, in Table 3 we highlight our semantic part basedview aggregation and multi-attention method in branch Gunder ModelNet10. We replace our view aggregation withmax pooling, mean pooling, and NetVLAD, where we ag-gregate V × K region features f i

k for classification. Al-though these results are good, our novel aggregation withmulti-attention can further improve the results. For evaluat-ing multi-attention, we keep G unchanged and set all entriesin α and β to 1 (“NoAtt”). This leads to significantly worseperformance compared to our “MultiAtt”. Next, we employα and β separately. We find that both of part attention andpart pattern attention improve “NoAtt”, but α (“PtAtt”) con-tributes less than β (“PnAtt”). Moreover, we highlight the

50

100

150

200

50 100 150 200 250

0.0038 0.0031 0.0038 0.0033 0.0041 0.0038

50

100

150

200

50 100 150 200 250

0.0035 0.0037 0.0038 0.0041 0.0044 0.0037

50

100

150

200

50 100 150 200 250

0.0038 0.0037 0.0044 0.0030 0.0047 0.0040

50

100

150

200

25050 100 150 200

0.0035 0.0047 0.0029 0.0044 0.0041 0.0041

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Pattern

(a)Bathtub (b)Bed

(c)Bathtub(d)Chair

α

β attention

Part attention

α Part attention

Pattern β attention0

Figure 4: Multi-attention is visualized under the test set of Model-Net10. α and β are shown as matrices and numbers.

effect of branch L as an enhancer of module R by removingL (“No L”) from Parts4Feature, which is also justified by thedegenerated results.

4.3 ClassificationTable 4 compares Parts4Feature with the state-of-the-art in3D shape classification under ModelNet. The comparison areconducted under the same condition1.Under both benchmarks, Parts4Feature outperforms all it-

s competitors at the same condition, where “Our” are ob-tained with the parameters of our best accuracy under Mod-elNet40 in Table 1 and the ones under ModelNet10 in Ta-ble 2. This comparison shows that Parts4Feature effectivelyemploys part-level information to significantly improve thediscriminability of learned features. Parts4Feature is also out-performing under ShapeNet55 with the same parameters ofour best results under ModelNet10, as shown by the compar-ison in the last three rows in Table 7.To better demonstrate our classification results, we visu-

alize the confusion matrix of our classification result underModelNet10 and ShapeNet55 in Fig. 5 and Fig. 6, respective-ly. In each confusion matrix, an element in the diagonal linemeans the classification accuracy in a class, while other ele-ments in the same row means the misclassification accuracy.

1We use the same modality of views from the same camera sys-tem for the comparison, where the results of RotationNet are fromFig.4 (a) and (b) in https://arxiv.org/pdf/1603.06208.pdf. Moreover,the benchmarks are with the standard training and test split.


770

0.64

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.71

0

0

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0.02

0.02

0

0.01

0

0

0

0

0.01

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.79

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.03

0

0

0

0

0

0

0

0

0

0

0

0

0

0.03

0

0

0.01

0

0

0

0

0

0

0.89

0.22

0.19

0

0.12

0

0

0

0

0

0.05

0.07

0.05

0

0.03

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0.04

0

0

0

0

0

0

0.02

0

0

0

0

0.01

0

0

0

0

0

0

0

0.13

0.11

0

0

0

0

0

0

0

0

0

0

0.03

0.05

0

0.05

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.02

0.09

0.36

0

0.05

0

0

0

0

0

0.01

0.20

0

0.09

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0.04

0

0

0

0

0

0

0.02

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0.97

0

0

0

0

0

0.02

0

0.03

0

0

0

0

0

0

0.04

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.07

0

0

0

0

0.08

0

0

0

0.07

0

0

0

0

0

0.01

0

0.81

0

0

0

0

0

0.02

0.03

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.04

0

0

0

0.71

0

0

0

0

0.01

0

0

0

0.03

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1.00

0

0

0

0

0

0

0

0

0

0.05

0

0.04

0

0

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0.08

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0

0

0.79

0

0

0

0

0

0

0

0

0

0

0.09

0

0

0

0

0.01

0.02

0

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.07

0

0

0

0

0

0

0

0

0

0

0

0.96

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.06

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.56

0

0

0

0

0.02

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.58

0.04

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.03

0

0.03

0

0.02

0

0

0

0

0

0.24

0.47

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.86

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.04

0.03

0

0

0

0

0

0

0

0

0.01

0

0.50

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0.22

0

0

0.04

0.05

0

0

0

0

0.03

0

0

0

0.02

0

0

0.59

0.20

0.05

0.03

0

0

0

0

0

0.04

0.03

0

0.04

0

0

0

0

0.10

0

0

0.04

0

0.04

0

0

0

0

0

0

0.02

0

0.03

0

0

0.14

0

0.01

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.03

0.73

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.74

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.13

0

0

0.05

0

0.04

0.02

0

0

0

0

0.03

0

0.14

0

0.03

0

0.03

0.02

0.05

0

0.92

0.04

0

0.01

0.01

0.12

0.02

0

0.08

0

0

0

0.01

0.03

0

0

0.01

0.10

0

0.04

0

0

0.10

0

0

0.07

0.02

0

0

0.01

0.08

0.02

0

0

0.13

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.17

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.99

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.03

0

0

0

0

0

0.13

0.01

0

0

0.06

0

0.03

0

0.12

0

0

0

0

0.01

0

0

0

0.13

0

0.95

0.04

0.03

0

0

0

0

0

0

0.01

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0.02

0.01

0.03

0

0

0.01

0

0

0

0

0

0

0

0

0.02

0

0

0.06

0

0.06

0

0.05

0

0

0

0

0

0

0

0

0

0

0.03

0.90

0.09

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.03

0

0

0

0

0.01

0

0

0

0.01

0

0

0

0

0

0

0

0.13

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0.01

0.02

0.74

0

0

0.08

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.14

0

0.05

0

0

0.02

0

0

0.06

0

0.03

0

0

0

0.02

0

0

0.01

0

0.05

0

0.13

0

0

0

0

0.81

0.08

0

0.01

0.02

0

0.02

0

0.03

0

0

0

0

0.08

0

0

0

0

0

0

0.05

0

0.06

0.01

0

0

0

0

0.13

0

0.02

0

0

0

0.01

0

0

0.12

0

0

0

0.02

0

0

0.09

0

0.02

0

0

0

0.09

0

0

0

0

0.01

0.54

0.08

0.02

0

0

0.02

0

0

0

0.01

0

0

0.08

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.07

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.77

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0.03

0.01

0

0

0

0

0

0

0

0.01

0.03

0.08

0

0.84

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.59

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0.39

0.97

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0.05

0

0.03

0.03

0

0

0

0

0

0

0.04

0.08

0

0.12

0

0

0

0

0.17

0

0

0

0

0.02

0.06

0

0

0

0

0.86

0

0

0

0

0

0

0.23

0

0

0

0

0

0.64

0

0.01

0

0

0

0

0.10

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0.90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.05

0

0

0

0

0

0

0

0.03

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0.83

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.13

0

0

0

0.07

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.15

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.14

0.01

0

0

0.04

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0.03

0

0.04

0

0

0

0

0

1.00

0.78

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.77

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0.02

0.07

0

0

0

0.03

0

0

0.05

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.99

0.04

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0.01

0

0

0

0

0

0

0.27

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.94

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

1.00

0

0

0.07

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.87

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1.00

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.93

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.14

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.02

0

0

0

0.04

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.03

0

0

0

0.02

0

0

0

0

0

0

0

0

0.70

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0.04

0

0

0

0

0

0.01

0

0

0

0

0.02

0

0

0

0

0

0

0

0

0

0

0

0

0.07

0

0.95

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.09

0

0.85

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.04

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.88

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.83

0

0

0

0

0

0

0.05

0

0.09

0.08

0

0

0

0

0

0

0

0

0

0

0.18

0.12

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.65

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.06

0

0

0

0.97

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0

0

0

0

0.08

0

0

0

0

0

0

0.02

0

0

0

0

0

0

0

0.40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

Figure 6: The classification confusion matrix under ShapeNet55.

Methods Raw MN40 MN10MVCN[Su and others, 2015] View 90.10 -MVVC[Qi et al., 2016] Voxel 91.40 -3DDt[Xie and others, 2018] Voxel - 92.40PaiV[Johns et al., 2016] View 90.70 92.80Sphe[Cao et al., 2017] View 93.31 -GIFT[Bai and others, 2017] View 89.50 91.50RAMA[Sfikas and others, 2017] View 90.70 91.12VRN[Brock et al., 2016] Voxel 91.33 93.80RNet[Kanezaki et al., 2018] View 90.65 93.84PNetP[Qi and others, 2017] Point 91.90 -DSet[Wang et al., 2017] View 92.20 -VGAN[Wu and others, 2016] Voxel 83.30 91.00LAN[Achlioptas and others, 2018] Point 85.70 95.30FNet[Yang et al., 2018] Point 88.40 94.40SVSL[Han and others, 2019a] View 93.31 94.82VIPG[Han and others, 2019b] View 91.98 94.05Our View 93.40 96.15

Table 4: The classification comparison ModelNet.

1.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

1.00

0.00

0.00

0.00

0.01

0.01

0.00

0.01

0.00

0.00

0.00

0.92

0.01

0.00

0.00

0.00

0.09

0.00

0.00

0.00

0.00

0.00

0.95

0.00

0.08

0.00

0.00

0.00

0.00

0.00

0.00

0.01

0.01

1.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.02

0.00

0.86

0.01

0.00

0.00

0.00

0.00

0.00

0.01

0.00

0.00

0.00

0.98

0.00

0.00

0.00

0.00

0.00

0.06

0.00

0.00

0.05

0.00

0.91

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.99

Bathtub

Bed

Chair

Desk

Dresser

Monitor

Night stand

Sofa

Table

Toilet

Bathtub

Bed

Chair

Desk

Dresser

Monitor

Night stand

SofaTable

Toilet

Figure 5: The classification confusion matrix under ModelNet10.

The large diagonal elements shows that Parts4Feature is goodat classifying large-scale 3D shapes.We also conduct experiments with reduced number

of segmented shapes for training under ModelNet10.As shown in Table 5, trained by randomly sampled{0%, 1%, 5%, 10%, 25%, 50%} of 6,386 shapes, our resultsincrease accordingly. The good results with 0% segmentedshapes show that we not only learn from pixel-level infor-mation in 3D classification benchmarks, similar to existingmethods, but also improve performance further by absorbingpart-level information from 3D segmentation benchmark.


771

Acc 0% 1% 5% 10% 25% 50% 100%Instance 93.0 93.5 93.8 93.8 94.1 94.3 96.15Class 92.7 93.1 93.4 93.6 94.0 94.1 96.14

Table 5: The effect of less segmented shapes for training.

0.9 1

SHDLFDFisher vector3D ShapeNetsPanoRAMAMVCNGIFTParts4Feature

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

Pre

cisi

on

Recall

(a) ModelNet40

SHDLFD3D ShapeNetsPanoRAMAGIFTParts4Feature

0.9 1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

Pre

cisi

on

Recall

(b) ModelNet10

Figure 7: The PR curve comparison under ModelNet.

4.4 RetrievalWe further evaluate Parts4Feature in shape retrieval underModelNet and ShapeNetCore55 by comparing with the state-of-the-art methods in Table 6 and Table 7. These experi-ments are conducted under the test set, where each 3D shapeis used as a query to retrieve from the rest of the shapes,and the retrieval performance is evaluated by mAP. The com-pared results include LFD, SHD, Fisher vector, 3D ShapeNet-s [Wu and others, 2015], Pano [Shi and others, 2015], MVC-N [Su and others, 2015], GIFT [Bai and others, 2017], RA-MA [Sfikas and others, 2017] and Trip [He et al., 2018] un-der ModelNet.As shown in Table 6, Table 7, our results outperfor-

m all the compared results in each benchmark. BesidesTaco [Cohen et al., 2018] in Table 7, the compared micro-averaged results in Table 7 are from SHREC2017 shape re-trieval contest [Savva and others, 2017] with the same names.In addition, the available PR curves under ModelNet40 andModelNet10 are also compared in Fig. 7, which also demon-strates our outperforming results in shape retrieval.

5 ConclusionsParts4Feature is proposed to learn 3D global features frompart-level information in a semantic way. It successfullylearns universal knowledge of generally semantic part de-tection from 3D segmentation benchmarks, and effectivelytransfers the knowledge to other shape analysis benchmarksby learning 3D global features from detected parts in mul-tiple views. Parts4Feature makes it feasible to improve 3Dglobal feature learning by leveraging discriminative informa-tion from another source. Moreover, our novel view aggre-gation with multi-attention can also benefit Parts4Feature tolearn more discriminative features than widely used aggrega-

Data Pano MVCN GIFT RAMA Trip OursMN40 76.8 79.5 81.9 83.5 88.0 91.5MN10 84.2 - 91.1 87.4 - 93.8

Table 6: The retrieval (mAP) comparison under ModelNet.

MicroMethods P R F1 mAP NDCGKanezaki 81.0 80.1 79.8 77.2 86.5Zhou 78.6 77.3 76.7 72.2 82.7

Tatsuma 76.5 80.3 77.2 74.9 82.8Furuya 81.8 68.9 71.2 66.3 76.2Thermos 74.3 67.7 69.2 62.2 73.2Deng 41.8 71.7 47.9 54.0 65.4Li 53.5 25.6 28.2 19.9 33.0Mk 79.3 21.1 25.3 19.2 27.7Su 77.0 77.0 76.4 73.5 81.5Bai 70.6 69.5 68.9 64.0 76.5Taco 70.1 71.1 69.9 67.6 75.6Our 62.0 80.4 62.2 85.9 90.2

SVSL[Han and others, 2019a] 85.5VIPG[Han and others, 2019b] 83.0

Our classification 86.9

Table 7: Retrieval and classification comparison in terms of Micro-averaged metrics under ShapeNetCore55.

tion procedures. Our outperforming results show that Part-s4Feature is superior to other state-of-the-art methods.

AcknowledgmentsThis work was supported by National Key R&D Programof China (2018YFB0505400) and NSF under award number1813583. We thank all anonymous reviewers for their con-structive comments.

References[Achlioptas and others, 2018] Panos Achlioptas et al. Learn-

ing representations and generative models for 3D pointclouds. In The International Conference on MachineLearning, pages 40–49, 2018.

[Bai and others, 2017] Song Bai et al. GIFT: Towards scal-able 3D shape retrieval. IEEE Transaction on Multimedia,19(6):1257–1271, 2017.

[Brock et al., 2016] Andrew Brock, Theodore Lim, J.M.Ritchie, and Nick Weston. Generative and discriminativevoxel modeling with convolutional neural networks. In 3Ddeep learning workshop (NIPS), 2016.

[Cao et al., 2017] Zhangjie Cao, Qixing Huang, and KarthikRamani. 3D object classification via spherical projections.In International Conference on 3D Vision. 2017.

[Chatfield et al., 2014] Ken Chatfield, Karen Simonyan, An-drea Vedaldi, and Andrew Zisserman. Return of the devilin the details: Delving deep into convolutional nets. InBritish Machine Vision Conference, 2014.

[Cohen et al., 2018] Taco S. Cohen, Mario Geiger, JonasKohler, and Max Welling. Spherical CNNs. In Interna-tional Conference on Learning Representations, 2018.

[Girshick, 2015] Ross Girshick. Fast R-CNN. In IEEE In-ternational Conference on Computer Vision, pages 1440–1448, 2015.


772

[Han and others, 2016] Zhizhong Han et al. Unsupervised3D local feature learning by circle convolutional restrictedboltzmann machine. IEEE Transactions on Image Pro-cessing, 25(11):5331–5344, 2016.

[Han and others, 2017] Zhizhong Han et al. Mesh convo-lutional restricted boltzmann machines for unsupervisedlearning of features with structure preservation on 3Dmeshes. IEEE Transactions on Neural Network andLearning Systems, 28(10):2268 – 2281, 2017.

[Han and others, 2019a] Zhizhong Han et al. Seqviews 2se-qlabels: Learning 3D global features via aggregating se-quential views by rnn with attention. IEEE Transactionson Image Processing, 28(2):1941–0042, 2019.

[Han and others, 2019b] Zhizhong Han et al. View inter-prediction gan: Unsupervised representation learning for3D shapes by learning global shape memories to supportlocal view predictions. In AAAI, 2019.

[Han et al., 2019] Zhizhong Han, Mingyang Shang, XiyangWang, Yu-Shen Liu, and Matthias Zwicker. Y2seq2seq:Cross-modal representation learning for 3D shape and textby joint reconstruction and prediction of view and wordsequences. In AAAI, 2019.

[He et al., 2018] Xinwei He, Yang Zhou, Zhichao Zhou,Song Bai, and Xiang Bai. Triplet-center loss for multi-view 3D object retrieval. In The IEEE Conference on Com-puter Vision and Pattern Recognition, 2018.

[Johns et al., 2016] Edward Johns, Stefan Leutenegger, andAndrew J. Davison. Pairwise decomposition of imagesequences for active multi-view recognition. In IEEEConference on Computer Vision and Pattern Recognition,pages 3813–3822, 2016.

[Kalogerakis and others, 2017] Evangelos Kalogerakis et al.3D shape segmentation with projective convolutional net-works. In IEEE Conference on Computer Vision and Pat-tern Recognition, pages 6630–6639, 2017.

[Kanezaki et al., 2018] Asako Kanezaki, Yasuyuki Mat-sushita, and Yoshifumi Nishida. Rotationnet: Joint objectcategorization and pose estimation using multiviews fromunsupervised viewpoints. In IEEE Conference on Com-puter Vision and Pattern Recognition, 2018.

[Muralikrishnan et al., 2018] Sanjeev Muralikrishnan,Vladimir G. Kim, and Siddhartha Chaudhuri. Tags2parts:Discovering semantic regions from shape tags. In IEEEConference on Computer Vision and Pattern Recognition,pages 2926–2935, 2018.

[Qi and others, 2017] Charles Qi et al. Pointnet++: Deep hi-erarchical feature learning on point sets in a metric space.In Advances in Neural Information Processing Systems,pages 5105–5114, 2017.

[Qi et al., 2016] C R Qi, H Su, and M Niebner. Volumetricand multi-view cnns for object classification on 3D data. InIEEE Conference on Computer Vision and Pattern Recog-nition, pages 5648–5656, 2016.

[Ren and others, 2015] Shaoqing Ren et al. Faster R-CNN:Towards real-time object detection with region proposal

networks. In Advances in Neural Information ProcessingSystems, pages 91–99, 2015.

[Savva and others, 2017] Manolis Savva et al. SHREC’17Large-Scale 3D Shape Retrieval from ShapeNet Core55.In Eurographics Workshop on 3D Object Retrieval, 2017.

[Sfikas and others, 2017] Konstantinos Sfikas et al. Exploit-ing the PANORAMA Representation for ConvolutionalNeural Network Classification and Retrieval. In EG Work-shop on 3D Object Retrieval, pages 1–7, 2017.

[Sharma et al., 2016] Abhishek Sharma, Oliver Grau, andMario Fritz. VConv-DAE: Deep volumetric shape learningwithout object labels. In Proceedings of European Confer-ence on Computer Vision, pages 236–250, 2016.

[Shi and others, 2015] B. Shi et al. Deeppano: Deeppanoramic representation for 3D shape recognition. IEEESignal Processing Letters, 22(12):2339–2343, 2015.

[Su and others, 2015] Hang Su et al. Multi-view convolu-tional neural networks for 3D shape recognition. In Inter-national Conference on Computer Vision, pages 945–953,2015.

[Wang et al., 2017] Chu Wang, Marcello Pelillo, andKaleem Siddiqi. Dominant set clustering and poolingfor multi-view 3D object recognition. In Proceedings ofBritish Machine Vision Conference, 2017.

[Wu and others, 2015] Zhirong Wu et al. 3D ShapeNets: Adeep representation for volumetric shapes. In Proceed-ings of IEEE Conference on Computer Vision and PatternRecognition, pages 1912–1920, 2015.

[Wu and others, 2016] Jiajun Wu et al. Learning a proba-bilistic latent space of object shapes via 3D generative-adversarial modeling. In Advances in Neural InformationProcessing Systems, pages 82–90, 2016.

[Xie and others, 2018] Jianwen Xie et al. Learning descrip-tor networks for 3D shape synthesis and analysis. In IEEEConference on Computer Vision and Pattern Recognition,2018.

[Yang et al., 2018] Yaoqing Yang, Chen Feng, Yiru Shen,and Dong Tian. Foldingnet: Point cloud auto-encoder viadeep grid deformation. In IEEE Conference on ComputerVision and Pattern Recognition, 2018.


773

Parts4Feature: Learning 3D Global Features from Generally ... · Parts4Feature: Learning 3D Global...

Documents

Transcript of Parts4Feature: Learning 3D Global Features from Generally ... · Parts4Feature: Learning 3D Global...