PowerPoint Presentation

Deformation-Driven Topology-Varying 3D Shape Correspondence

Ibraheem Alhashim Kai Xu Yixin Zhuang Junjie Cao Patricio Simari Hao Zhang

Presenter:Ibraheem AlhashimSimon Fraser University

Sponsored by Good morning everyone, and thanks for the introduction1Shape CorrespondenceFundamental task in:

Shape morphing

Statistical shape modeling

Object recognition

ClassificationSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceShape correspondence is a fundamental task in many applications including object recognition, shape classification, statistical shape modeling, and shape morphing2

Shape CorrespondenceCorresponding man-made 3D shapes is challengingLarge variability in geometry & structureReal-world data is inconsistent & unlabeledSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceCorresponding man-made shapes is quite challenging since we have -among other issues- large variability in geometry and structure, plus currently available datasets are usually inconsistent and come with unlabeled parts3Shape CorrespondencePart-level correspondences

[Jain 12]

[Xu 12]

[Zheng 13]

[Averkiou 14]

[Kalogerakis 12]SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceIn the context of shape-creation, having a meaningful part-level correspondence is valuable for most automatic shape synthesis methodsThese methods typically work by replacing parts with compatible ones which are decided based on different part matching schemes4Shape CorrespondenceContinuous fine-grained correspondence is critical for continuous shape blending[Alhashim et al. 14]

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceMore specifically, a continuous and fine-grained part correspondence is critical for the topology-altering shape blending method we presented last year5

Previous WorksRigid alignment not sufficient for diverse shapes[Golovinskiy & Funkhouser 08]

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceA straightforward matching for man-made shapes can relay on a reasonable rigid alignment and a greedy assignmentThis however, is not sufficient for the diverse shapes found in the real-worldWe can see that for these beds, some parts are correctly matched while others are either incorrectly matched or not matched at all

6Previous Works

[Zheng et al. 14]

[Kim et al. 13]

[Laga et al. 14]

[Huang et al. 14]Co-analysis methodsCoarse resultsForced correspondence Set of shapes

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceMore recent efforts on corresponding man-made shapes are the co-analysis methods where we try to extract consistent matchings by looking for geometric or structural commonalitiesThese methods, however, typically result in coarse matches where multiple parts end up matching a single regionFurthermore, all parts are forced to belong to a specific grouping or label which is not correct, or might incorrectly label unique partsLastly, their performance is proportional to the size of set, making them less ideal for comparing pairs7Deformation-Driven Shape MatchingBest matching = minimal self-distortion as we deform one shape to match the other

[Zhang et al. 08]

[Sederberg & Greenwood 92]

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceAnother class of methods, that is more suitable for pairs, are the deformation-driven correspondence methodsThese methods build on the assumption that: the best matching is the one that guides a deformation of a shape to match a target with the least overall distortion{CLICK!} The intuition here is that significant twisting and bending typically indicates that some regions are incorrectly matched8ChallengeHow to apply a deformation-driven search to complex man-made shapes?Many disconnected componentsSemantically similar yet very differentDiscrepancy in part count & structural relations

BackSeatLegs

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceIn this work, we are looking at adapting this strategy to matching geometrically and topologically different man-made shapesThere are several challenges to overcomeFirst, man-made objects are typically made of many disconnected components, we need a deformation model that supports thatSecond, we typically find semantically similar parts that are geometrically very different, so we need to be less sensitive to these differencesThird, we need to be able to handle the abundant discrepancy in part count and topology in man-made shapes

9Our ProposalThe GeoTopo transform

Piece-wise continuous part correspondenceSupports topological changes No prior or fixed number of segmentsEfficient to computeWorks on pairs

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceIn this work, we propose The GeoTopo transform A deformation-driven, topology-varying, correspondence searchWe develop a deformation model and an associated energy that:Computes piece-wise fine-grained part correspondencesIt supports topological changes including part splitting and mergingIt does not need any prior information and is not limited to a fixed number of partsIt is also efficient to computeAnd it is designed to handle shape pairs 10Our Proposal

The GeoTopo transform

Deformation modelDistortion EnergySA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceOur two key contributions areA deformation model that is structure-aware, and allows for topological changes And second, an energy that captures structural distortions, which we use to guide our correspondence search11

Deformation ModelDeformation suitable for man-made shapesSupports disconnected componentsStructure-aware (preserving part relations)Allows for topological changes

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceOur proposed deformation model is suitable for man-made shapesIt works on disconnected componentsIt is also structure-aware, that is, it retains structural relations between the parts including symmetry and contactAnd it supports changes in topology, so we can also include one-to-many correspondences in our search

12Self-Distortion EnergyStructural distortion in three terms:Distortion on all pairs of connected partsConnectivity between partsSolidity measure for parts changing topology

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceOur proposed energy tries to encode structural distortions by considering three terms for a deforming shape:The first is the shape distortion term, were we penalize large deviations in the original part arrangements The second is a connectivity term, that favors assigning correspondences that keep connected parts close to each otherAnd the third is a solidity measure for parts changing topology, for example when a single part is expect to split into many

13Shape RepresentationA structure graph of part skeletons[Alhashim et al. 2014]Skeletons are fitted by parametric curves / sheets

Parametric curvesParametric sheetSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceFirst, in order to simplify the deformation process, facilitate topological transformations, and minimize the effect of geometric differences: we define our deformation model and energy on the shape abstraction we have used in our previous work on shape blendingIn this representation, we abstract the part by fitting their skeletons with a parametric curve or sheetThe set of abstractions, along with their contact and symmetry relations, make up the structure graph we will use in this work14Structural Rods3D shapeCurve-sheet abstractions

Structural rodsSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceWe now look at how we define our distortion energy:First, for each pair of connected parts{CLICK!} We uniformly sample the abstracting curve and sheet and connect all samples from one part to the other as a set of vectors which we call structural rods, one such rod is shown here in blueWe will use these rods in computing our energy terms15EnergyDistortion term EdOverall change of part arrangementsChange in angle

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceThe first term in our energy is the distortion term Ed, which is defined over the set of all structural rodsThis term tries to capture the overall change of part arrangementsWhich we compute by looking at the change in angle of the structural rods before and after a deformation step16EnergyDistortion term Ed

Best correspondence

Before deformation

After deformationSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceHere we see a figure visualizing all rods defined on a chair, where hot colors represent large changesWe can see here how one deformation significantly changed the rods directions especially at the legs, So this term in effect represents the amount of bending or twisting incurred by a deformation on a pair of connected parts17EnergyConnectivity term EcRelative length of shortest rods before and after deformation

Source shapeDeformed shapeTargetSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceThe second term in our energy is the connectivity term where weWe do so by looking at the relative length of the shortest rods before and after the deformationIn this example, the leg on the source shape was assigned to the middle part on the target. Note how the shortest rod d is affected by such an assignment18EnergySolidity term EsRatio between the volume of a part to the volume of its convex hullMeasures the effect of a split / merge

HighLowHighSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceThe final term in our energy is the solidity term where we try to rank one-to-many correspondences based on the heuristic that solid parts need to remain soFor this we use the ratio between the volume of a part, or group of parts, to the volume of the convex hullHere we see an illustrative example showing different parts and their solidity19EnergySolidity term Es

?

?SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceFor a more concreate example, consider these two stools with largely dissimilar topsThere are multiple choices to correspond the top part but they all have different solidity valuesThe parts in green and red have high solidity which matches with the solidity of the source shapes top20Deformation ProcessDeform-to-fit matched parts, then propagate

CurvesSheets1. Align centers2. Match extremities 3. Deform towards target4. Propagate edit to others

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceWe looked at the three terms defining our energy, lets briefly look at our proposed deformation mechanismWe essentially apply a deform-to-fit approach in which we:First, align the centers of the part abstractions, here we show a blue source part being matched to the grey target part,Second we match the extremities of the parametric representations using spatial proximityThird we deform the curve or sheet by linear interpolationAnd lastly, we apply an edit propagation step in which we try to enforce symmetry and contact constraintsMore details on the deformation can be found in the paper21SearchSearch tree path: set of matched parts on the sourceBeam search + pruning

3D shapesCurve-sheet abstractionsseat-seatback bar-back barback-backleg back-leg backleg front-leg frontSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceNow with our deformation model and energy that measures structural distortion we can start a search for the best correspondence, that is, the one with the least amount of energyIn our search, each path to a leaf node on our search tree represents a set of matches for all parts on the sourceThe search space of possible assignments can grow exponentiallyIn order to make the search manageable we apply a variant of beam search with some pruning rules, the details of which can be found in the paper22Results

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceNow lets look at results obtained using our correspondence algorithmNote the fine-grained part correspondences that have not been searched for in previous worksOur method can work on different shape classes with different numbers of parts23Results

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceHere we see some more results with different topological complexityWe use the same settings when running our algorithm on all of these resultsAnd the typical time for the more complicated shape pairs is around 2 seconds24

ApplicationsShape blending

fully automatically!SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceContinuous shape blending is a key application that we target for our methodIn these examples we are able to achieve good matching, in turn allowing us to generate plausible blending results fully automatically!25Applications

Topological medoidShape Classification

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceAnother application we have tried our method with is computing a topological mediod, by using our energy as a distanceThe medoid can be regarded as a representative shape for the given setWe have also looked at how our energy could be used to better classify and cluster shapes based on their structureFor example, in the set of chairs, our measure can cluster chairs based on weather they have vertical or horizontal back bars and better separate office chairs that have or lack armrests26EvaluationGround truth75 shapes, 5 categories (chair, airplane, table, bed, velocipede)Fine and coarse labels

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceTo quantitatively evaluate the performance of our method we have created a ground truth set of 75 topologically complex shapes covering 5 categories in which we manually label all their partsEach part in a shape end up having a fine and coarse labeling, we do so to support comparisons with co-analysis methods27Evaluation[Xu 12] Fuzzy part correspondence (baseline)Works on pairsMatch based on part OBB similarity[Zheng 14] Recurring part arrangementsFind semantic consistency between part arrangements Performs better than co-segmentation in the presence of large shape variability[Kim 13] Deformable part-based templatesBetter suited for large shape setsSupports poorly segmented inputs + can be fully auto.SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceWe evaluated against three past methodsThe first is a baseline method, that supports shape pairs, and matches parts by comparing the similarity of their bounding boxesThe other two methods are recent co-analysis methods One method uses the concept of recurring part arrangements, and the other is the deformable part-based templates that have been tested on large datasets and supports working on poorly segmented input models28EvaluationFine-grained correspondence benchmark

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceThis table shows the evaluation with the baseline method on the fine-grained part correspondencesAs we can see, our method outperforms the baseline in all categories in both the precision of the correspondences and its recall

29EvaluationCoarse correspondence benchmark (co-analysis)

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceAs for comparing with co-analysis methods, we use a coarse labeling of the ground truth, In this table we also see that we outperform past methodsMore notably we achieve high relative accuracy for the table set, which exhibit large variability in their legs, and the bike set which exhibit complexity in terms of number of parts and how they connect to each other30SummaryGeoTopo: topology-varying deformation model for a fine-grained correspondence search

Key contribution: a deformation model and a self-distortion energy, defined on structural rods, assess shape matching quality based on preservation of structure

Our framework shows promising results on challenging datasets with much room for improvementSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceAnd now to summarize, in this paper:We have proposed the GeoTopo transform, a topology-varying deformation model for fine-grained correspondenceThe key contribution is a topology-changing deformation model and a self-distortion energy, defined on structural rods, that assess shape matching quality based on the preservation of structureWe think our framework shows promising results on challenging datasets, but there is room for improvement!

31Limitations

Initial segmentationLarge geo. and topo. differencesSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceTwo main limitations that affect the quality of our results are:First, the initial segmentation of the input shapes. We can see that the arms on one robot is only partially matched with the other, and for the chairs, we can see that the legs on the right are not fully matched since they are segmented into one segment on one but not the otherSecond, our measure can still be greatly affected by large geometric and topological differences, for example, here we see that the wheels on the bike and tricycle models are not matched32Future WorkSegmentationOnline shape repositories are not well segmented Incorporate a segmentation search along with the correspondence search

High energyLow energy segmentationSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceTo address these issues, and increase both the performance and applicability of our method, we can look at incorporating the following:First, in order to apply our method on large scale data found online we need to address the segmentation challengeOne way to do so is to incorporate segmentation in our search where we compute our correspondence on multiple candidate segmentations and return the one with the least energyHere we show an example that we have tried, we can see that the segmentations on the right result in lower energy and better matching than for the original supplied segmentation on the left

33Future WorkCo-analysisFine-grained correspondence on a setLooking for consistent assignments

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceThe other direction we are considering is bringing the level of fine-grained correspondence we are now able to compute to the co-analysis domainThat is, we compute correspondences on a set by looking for consistent fine-grained assignmentsIn this example, our solution initially misses matching the legsBut when we match the two shapes via a third one, we obtain better results across all three shapes34

THANK YOU!gruvi.cs.sfu.ca/project/geotopoACKNOWLEDGMENTSAnonymous reviewers, authors who provided code, funding from:

NSFCSA2015.SIGGRAPH.ORGSponsored by And with that I conclude my talk, Id like to thank the anonymous reviewers for their feedback and suggestionsAnd Id like to thank you all for you attention, and please check out our project page for source code and dataThank you!35Partial Matching

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceEnergy Terms

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceAutomatic Segmentation

SDFCon. AwareConvex. AnalysisApproximate Convexity AnalysisSA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceCost & Quality Trade-off

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceTiming & Shapes Complexity

SA2015.SIGGRAPH.ORG#Deformation-Driven Topology-Varying 3D Shape CorrespondenceStructure PreservationEdit propagationReinforce symmetry and contact relations

41Topology-Varying Shape Matching and Modeling41