Real-Time 3D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University...

Real-Time 3D Model Real-Time 3D Model AcquisitionAcquisition

Szymon RusinkiewiczSzymon Rusinkiewicz

Olaf Hall-HoltOlaf Hall-Holt

Marc LevoyMarc Levoy

Princeton UniversityPrinceton University

Stanford UniversityStanford University

3D Scanning3D Scanning

Possible Research GoalsPossible Research Goals

• Low noiseLow noise

• Guaranteed high accuracyGuaranteed high accuracy

• High speedHigh speed

• Low costLow cost

• Automatic operationAutomatic operation

• No holesNo holes

3D Model Acquisition Pipeline3D Model Acquisition Pipeline

3D Scanner3D Scanner3D Scanner3D Scanner



View PlanningView PlanningView PlanningView Planning



AlignmentAlignmentAlignmentAlignmentView PlanningView PlanningView PlanningView Planning



AlignmentAlignmentAlignmentAlignment

MergingMergingMergingMerging





MergingMergingMergingMergingDone?Done?Done?Done?







DisplayDisplayDisplayDisplay

3D Model Acquisition Difficulties3D Model Acquisition Difficulties

• Much (often most) time spent on “last Much (often most) time spent on “last 20%”20%”

• Pipeline not optimized for hole-fillingPipeline not optimized for hole-filling

• Not sufficient just to speed up scanner –Not sufficient just to speed up scanner –must design pipeline for fast feedbackmust design pipeline for fast feedback

Real-Time 3D Model AcquisitionReal-Time 3D Model Acquisition

Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline






HumanHuman







Challenge:Challenge:Real TimeReal Time







Part I:Part I:Structured-LightStructured-Light

TriangulationTriangulation







Part II:Part II:Fast ICPFast ICP







Part III:Part III:Voxel GridVoxel Grid


• Project laser stripe onto objectProject laser stripe onto object

ObjectObject

LaserLaser

CameraCameraCameraCamera

CameraCameraCameraCamera


• Depth from ray-plane triangulationDepth from ray-plane triangulation

LaserLaser

(x,y(x,y))

ObjectObject


• Faster acquisition: project multiple Faster acquisition: project multiple stripesstripes

• Correspondence problem: which stripeCorrespondence problem: which stripeis which?is which?

Continuum of Triangulation Continuum of Triangulation MethodsMethods

Slow, robustSlow, robust Fast, fragileFast, fragile

Multi-stripeMulti-stripeMulti-frameMulti-frame

Single-frameSingle-frameSingle-stripeSingle-stripe

Time-Coded Light PatternsTime-Coded Light Patterns

• Assign each stripe a unique illumination Assign each stripe a unique illumination codecodeover time [Posdamer 82]over time [Posdamer 82]

SpaceSpace

TimeTime

Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)

CodeCodeCodeCode

Codes for Moving ScenesCodes for Moving Scenes

• Assign time codesAssign time codesto stripe to stripe boundariesboundaries

• Perform frame-to-framePerform frame-to-frametracking of correspondingtracking of correspondingboundariesboundaries– Propagate illumination historyPropagate illumination history

[Hall-Holt & Rusinkiewicz, ICCV 2001][Hall-Holt & Rusinkiewicz, ICCV 2001]

Designing a CodeDesigning a Code

• Want many “features” to track:Want many “features” to track:lots of black/white edges at each framelots of black/white edges at each frame

• Try to minimize ghosts – WW or BB Try to minimize ghosts – WW or BB “boundaries” that can’t be seen directly“boundaries” that can’t be seen directly

Designing a CodeDesigning a Code

0011001111101110

10111011 01100110

01000100 10011001

0001000100010001 1100110011001100

00000000 11011101

10101010 01110111

10001000010101011111111111111111 0010001000100010

[Hall-Holt & Rusinkiewicz, ICCV 2001][Hall-Holt & Rusinkiewicz, ICCV 2001]

ImplementationImplementation

• Pipeline:Pipeline:

• DLP projector illuminates scene @ 60 DLP projector illuminates scene @ 60 Hz.Hz.

• Synchronized NTSC camera captures Synchronized NTSC camera captures videovideo

• Pipeline returns range images @ 60 Hz.Pipeline returns range images @ 60 Hz.

ProjectProjectCodeCode

ProjectProjectCodeCode

CaptureCaptureImagesImagesCaptureCaptureImagesImages

FindFindBoundariesBoundaries

FindFindBoundariesBoundaries

MatchMatchBoundariesBoundaries

MatchMatchBoundariesBoundaries DecodeDecodeDecodeDecode ComputeCompute

RangeRangeComputeCompute

RangeRange







Part II:Part II:Fast ICPFast ICP

Aligning 3D DataAligning 3D Data

• ICP (Iterative Closest Points): for each ICP (Iterative Closest Points): for each point on one scan, minimize distance to point on one scan, minimize distance to closest point on other scan…closest point on other scan…

Aligning 3D DataAligning 3D Data

• … … and iterate to find alignmentand iterate to find alignment– Iterated Closest Points (ICP) [Besl & McKay Iterated Closest Points (ICP) [Besl & McKay

92]92]

ICP in the Real-Time PipelineICP in the Real-Time Pipeline

• Potential problem with ICP: local minimaPotential problem with ICP: local minima– In this pipeline, scans close togetherIn this pipeline, scans close together

– Very likely to converge to correct (global) Very likely to converge to correct (global) minimumminimum

• Basic ICP algorithm too slow (~ seconds)Basic ICP algorithm too slow (~ seconds)– Point-to-plane minimizationPoint-to-plane minimization

– Projection-based matchingProjection-based matching

– With these tweaks, running time ~ millisecondsWith these tweaks, running time ~ milliseconds

[Rusinkiewicz & Levoy, 3DIM 2001][Rusinkiewicz & Levoy, 3DIM 2001]







Part III:Part III:Voxel GridVoxel Grid

Merging and RenderingMerging and Rendering

• Goal: visualize the model well enoughGoal: visualize the model well enoughto be able to see holesto be able to see holes

• Cannot display all the scanned data – Cannot display all the scanned data – accumulates linearly with timeaccumulates linearly with time

• Standard high-quality merging methods:Standard high-quality merging methods:processing time ~ 1 minute per scanprocessing time ~ 1 minute per scan


++


• Point rendering, using accumulated normals for lightingPoint rendering, using accumulated normals for lighting

Example: PhotographExample: Photograph

18 cm.18 cm.

ResultResult

PostprocessingPostprocessing

• Real-time displayReal-time display– Quality/speed tradeoffQuality/speed tradeoff

– Goal: let user evaluate coverage, fill holesGoal: let user evaluate coverage, fill holes

• Offline postprocessing for high-quality Offline postprocessing for high-quality modelsmodels– Global registrationGlobal registration

– High-quality merging (e.g., using VRIP High-quality merging (e.g., using VRIP [Curless 96])[Curless 96])

Postprocessed ModelPostprocessed Model

Recapturing AlignmentRecapturing Alignment

SummarySummary

• 3D model acquisition pipeline optimized 3D model acquisition pipeline optimized for obtaining complete, hole-free modelsfor obtaining complete, hole-free models

• Use human’s time most efficientlyUse human’s time most efficiently

• Pieces of pipeline selected for real-time Pieces of pipeline selected for real-time use:use:– Structured-light scanner for moving objectsStructured-light scanner for moving objects

– Fast ICP variantFast ICP variant

– Simple grid-based merging, point renderingSimple grid-based merging, point rendering

LimitationsLimitations

• Prototype noisier than commercial systemsPrototype noisier than commercial systems– Could be made equivalent with careful engineeringCould be made equivalent with careful engineering

– Ultimate limitations on quality: focus, textureUltimate limitations on quality: focus, texture

• Scan-to-scan ICP not perfect Scan-to-scan ICP not perfect alignment drift alignment drift– Due to noise, miscalibration, degenerate geometryDue to noise, miscalibration, degenerate geometry

– Reduced, but not eliminated, by “anchor scans”Reduced, but not eliminated, by “anchor scans”

– Possibly combine ICP with separate trackersPossibly combine ICP with separate trackers

Future WorkFuture Work

• Faster scanningFaster scanning– Better stripe boundary trackingBetter stripe boundary tracking

– Multiple cameras, projectorsMultiple cameras, projectors

– High-speed cameras, projectorsHigh-speed cameras, projectors

• Application in different contextsApplication in different contexts– Cart- or shoulder-mounted for digitizing Cart- or shoulder-mounted for digitizing

roomsrooms

– Infrared for imperceptibilityInfrared for imperceptibility

AcknowledgmentsAcknowledgments

• Collaborators:Collaborators:– Li-Wei HeLi-Wei He

– James DavisJames Davis

– Lucas PereiraLucas Pereira

– Sean AndersonSean Anderson

• Sponsors:Sponsors:– SonySony

– IntelIntel

– IntervalInterval

Real-Time 3D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University...

Documents

Transcript of Real-Time 3D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University...