IIIT

Procedural 3D Building Reconstruction

using Shape Grammars and Detectors

Markus Mathias

Anđelo Martinović

Julien Weissenberg

Luc Van Gool

May, 19th 2011

IIIT2/31

System Overview

Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }

IIIT3/31

System Overview

Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }

IIIT4/31

System Overview

Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }

IIIT5/31

System Overview

Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }

IIIT6/31

System Overview

Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }

Inverse ProceduralModeling System

IIIT7/31

System Overview

Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }

Inverse ProceduralModeling System

IIIT8/31

Modelling: Procedural Modelling

• Create models from set of rules separate from evaluation engine

• Generating large scenes with only a few amount of rules

• Models are compact and semantically meaningful

• Used for plant modelling, façades and even entire cities including street network, building, façades, trees

• BUT: Focus so far mostly on creation of new virtual buildings, not existing ones

Images courtesy of www.procedural.com

IIIT9/31

Pictures

Grammar

?

How can we use prior knowledge encoded by the grammar?

Inverse Procedural Modeling

IIIT10/31

Pictures

Grammar

How can we use prior knowledge encoded by the grammar?

Asset detectors!

Asset detectors relate image areas with grammar symbols

Inverse Procedural Modeling

IIIT11/31

GrammarInterpreter

VisionModule

AssetDetectors

3D ReconstructionModule

Inverse Procedural Modeling System

IIIT12/31

GrammarInterpreter

VisionModule

AssetDetectors

3D ReconstructionModule

Inverse Procedural Modeling System

IIIT13/31

Grammar Interpreter

Analyzes input grammar Extracts semantic information CGA shape:

Standardized description Powerful shape operations Readable by humans Tool for rendering available (Cityengine)

GrammarInterpreter

VisionModule

Symbol listMatching symbols

Structural InformationEstimated attributes

IIIT14/31

Temple Grammar Example

Colonnade --> split(x){ ~ columnSpacing : Column | { ~ columnSpacing : Column }*

| ~ columnSpacing : Column }

Column --> split(y){ ~1 : Shaft | capitalHeight : Capital }

Capital --> split(y){ ~echinusPartsHeight: i(echinusAsset) | ~abacusPartsHeight : i(abacusAsset) }

Shaft --> i(shaftAsset)

Colonade

IIIT15/31

GrammarInterpreter

VisionModule

AssetDetectors

3D ReconstructionModule

Inverse Procedural Modeling System

IIIT16/31

Asset Detector

Felzenszwalb's part based Detector is trained on a diverse set of examples, e.g. Doric, Corinthian and Ionic capitals

High confidence detections through the reconstruction process Retraining of the detector including these detections leads to

specialized detector

IIIT17/31

GrammarInterpreter

VisionModule

AssetDetectors

3D ReconstructionModule

Inverse Procedural Modeling System

IIIT18/31

3D reconstruction module

Using SfM web service ARC3D:● Sparse point-cloud● Camera calibrations ● Only used to support our system and not providing parts

for output model

ARC3D

IIIT19/31

GrammarInterpreter

VisionModule

AssetDetectors

3D ReconstructionModule

Inverse Procedural Modeling System

IIIT20/31

Vision Module

Estimation of the dominant facade planes

Re-weighting of detections:

IIIT21/31

Vision Module

Accumulation of detections in 3D

Parameter estimation for Grammar Interpreter● Lot size, Asset size, Repeat distances, asset color

IIIT22/31

Vision Module

Substantiation of the semantic information coming from the grammar interpreter:

● Spatial relation of detected assets

● Similarity detection if asset appears in “repeat”

IIIT23/31

Vision module - Results

Determined parameters are translated by the grammar interpreter to the appropriate grammar attributes:

Detections with high confidence are included into the detector training set:

189+188 (new detections) for capitals 204+124 (new detections) for shafts Specialized column and shaft detector for Greek Doric temples: Detection rate for capitals increased by 7.31% (FP at 2.2) Detection rate for shafts increased by 14.89% (FP at 5.4)

IIIT24/31

Results – Temple of Poseidon

IIIT25/31

Results – Temple of Poseidon

IIIT26/31

Results – Temple of Athena

IIIT27/31

Results – Temple of Athena

IIIT28/31

Results – Parthenon Replica in Nashville

IIIT29/31

Results – Parthenon Replica in Nashville

IIIT30/31

Future Work

Registration of model to the point-cloud: Measurement of the accuracy of the estimated model Adjustment of remaining parameters

Extension to use the system on houses of different styles: Haussman Neo-classical Renaissance

IIIT31/31

Questions?