Real-Time 3D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University...

Real-Time 3D Model Real-Time 3D Model AcquisitionAcquisition

Szymon RusinkiewiczSzymon Rusinkiewicz

Olaf Hall-HoltOlaf Hall-Holt

Marc LevoyMarc Levoy

Princeton UniversityPrinceton University

Stanford UniversityStanford University

3D Scanning3D Scanning

Possible Research GoalsPossible Research Goals

• Low noiseLow noise

• Guaranteed high accuracyGuaranteed high accuracy

• High speedHigh speed

• Low costLow cost

• Automatic operationAutomatic operation

• No holesNo holes

3D Model Acquisition Pipeline3D Model Acquisition Pipeline

3D Scanner3D Scanner3D Scanner3D Scanner

View PlanningView PlanningView PlanningView Planning

AlignmentAlignmentAlignmentAlignmentView PlanningView PlanningView PlanningView Planning

AlignmentAlignmentAlignmentAlignment

MergingMergingMergingMerging

MergingMergingMergingMergingDone?Done?Done?Done?

DisplayDisplayDisplayDisplay

3D Model Acquisition Difficulties3D Model Acquisition Difficulties

• Much (often most) time spent on “last Much (often most) time spent on “last 20%”20%”

• Pipeline not optimized for hole-fillingPipeline not optimized for hole-filling

• Not sufficient just to speed up scanner –Not sufficient just to speed up scanner –must design pipeline for fast feedbackmust design pipeline for fast feedback

Real-Time 3D Model AcquisitionReal-Time 3D Model Acquisition

Real-Time 3D Model Acquisition Real-Time 3D Model Acquisition PipelinePipeline

HumanHuman

Challenge:Challenge:Real TimeReal Time

Part I:Part I:Structured-LightStructured-Light

TriangulationTriangulation

Part II:Part II:Fast ICPFast ICP

Part III:Part III:Voxel GridVoxel Grid

• Project laser stripe onto objectProject laser stripe onto object

ObjectObject

LaserLaser

CameraCameraCameraCamera

• Depth from ray-plane triangulationDepth from ray-plane triangulation

LaserLaser

(x,y(x,y))

ObjectObject

• Faster acquisition: project multiple Faster acquisition: project multiple stripesstripes

• Correspondence problem: which stripeCorrespondence problem: which stripeis which?is which?

Continuum of Triangulation Continuum of Triangulation MethodsMethods

Slow, robustSlow, robust Fast, fragileFast, fragile

Multi-stripeMulti-stripeMulti-frameMulti-frame

Single-frameSingle-frameSingle-stripeSingle-stripe

Time-Coded Light PatternsTime-Coded Light Patterns

• Assign each stripe a unique illumination Assign each stripe a unique illumination codecodeover time [Posdamer 82]over time [Posdamer 82]

SpaceSpace

TimeTime

Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)Illumination history = (WB),(BW),(WB)

CodeCodeCodeCode

Codes for Moving ScenesCodes for Moving Scenes

• Assign time codesAssign time codesto stripe to stripe boundariesboundaries

• Perform frame-to-framePerform frame-to-frametracking of correspondingtracking of correspondingboundariesboundaries– Propagate illumination historyPropagate illumination history

[Hall-Holt & Rusinkiewicz, ICCV 2001][Hall-Holt & Rusinkiewicz, ICCV 2001]

Designing a CodeDesigning a Code

• Want many “features” to track:Want many “features” to track:lots of black/white edges at each framelots of black/white edges at each frame

• Try to minimize ghosts – WW or BB Try to minimize ghosts – WW or BB “boundaries” that can’t be seen directly“boundaries” that can’t be seen directly

Designing a CodeDesigning a Code

0011001111101110

10111011 01100110

01000100 10011001

0001000100010001 1100110011001100

00000000 11011101

10101010 01110111

10001000010101011111111111111111 0010001000100010

[Hall-Holt & Rusinkiewicz, ICCV 2001][Hall-Holt & Rusinkiewicz, ICCV 2001]

ImplementationImplementation

• Pipeline:Pipeline:

• DLP projector illuminates scene @ 60 DLP projector illuminates scene @ 60 Hz.Hz.

• Synchronized NTSC camera captures Synchronized NTSC camera captures videovideo

• Pipeline returns range images @ 60 Hz.Pipeline returns range images @ 60 Hz.

ProjectProjectCodeCode

CaptureCaptureImagesImagesCaptureCaptureImagesImages

FindFindBoundariesBoundaries

MatchMatchBoundariesBoundaries

MatchMatchBoundariesBoundaries DecodeDecodeDecodeDecode ComputeCompute

RangeRangeComputeCompute

RangeRange

Part II:Part II:Fast ICPFast ICP

Aligning 3D DataAligning 3D Data

• ICP (Iterative Closest Points): for each ICP (Iterative Closest Points): for each point on one scan, minimize distance to point on one scan, minimize distance to closest point on other scan…closest point on other scan…

Aligning 3D DataAligning 3D Data

• … … and iterate to find alignmentand iterate to find alignment– Iterated Closest Points (ICP) [Besl & McKay Iterated Closest Points (ICP) [Besl & McKay

92]92]

ICP in the Real-Time PipelineICP in the Real-Time Pipeline

• Potential problem with ICP: local minimaPotential problem with ICP: local minima– In this pipeline, scans close togetherIn this pipeline, scans close together

– Very likely to converge to correct (global) Very likely to converge to correct (global) minimumminimum

• Basic ICP algorithm too slow (~ seconds)Basic ICP algorithm too slow (~ seconds)– Point-to-plane minimizationPoint-to-plane minimization

– Projection-based matchingProjection-based matching

– With these tweaks, running time ~ millisecondsWith these tweaks, running time ~ milliseconds

[Rusinkiewicz & Levoy, 3DIM 2001][Rusinkiewicz & Levoy, 3DIM 2001]

Part III:Part III:Voxel GridVoxel Grid

Merging and RenderingMerging and Rendering

• Goal: visualize the model well enoughGoal: visualize the model well enoughto be able to see holesto be able to see holes

• Cannot display all the scanned data – Cannot display all the scanned data – accumulates linearly with timeaccumulates linearly with time

• Standard high-quality merging methods:Standard high-quality merging methods:processing time ~ 1 minute per scanprocessing time ~ 1 minute per scan

Merging and RenderingMerging and Rendering

• Point rendering, using accumulated normals for lightingPoint rendering, using accumulated normals for lighting

Example: PhotographExample: Photograph

18 cm.18 cm.

ResultResult

PostprocessingPostprocessing

• Real-time displayReal-time display– Quality/speed tradeoffQuality/speed tradeoff

– Goal: let user evaluate coverage, fill holesGoal: let user evaluate coverage, fill holes

• Offline postprocessing for high-quality Offline postprocessing for high-quality modelsmodels– Global registrationGlobal registration

– High-quality merging (e.g., using VRIP High-quality merging (e.g., using VRIP [Curless 96])[Curless 96])

Postprocessed ModelPostprocessed Model

Recapturing AlignmentRecapturing Alignment

SummarySummary

• 3D model acquisition pipeline optimized 3D model acquisition pipeline optimized for obtaining complete, hole-free modelsfor obtaining complete, hole-free models

• Use human’s time most efficientlyUse human’s time most efficiently

• Pieces of pipeline selected for real-time Pieces of pipeline selected for real-time use:use:– Structured-light scanner for moving objectsStructured-light scanner for moving objects

– Fast ICP variantFast ICP variant

– Simple grid-based merging, point renderingSimple grid-based merging, point rendering

LimitationsLimitations

• Prototype noisier than commercial systemsPrototype noisier than commercial systems– Could be made equivalent with careful engineeringCould be made equivalent with careful engineering

– Ultimate limitations on quality: focus, textureUltimate limitations on quality: focus, texture

• Scan-to-scan ICP not perfect Scan-to-scan ICP not perfect alignment drift alignment drift– Due to noise, miscalibration, degenerate geometryDue to noise, miscalibration, degenerate geometry

– Reduced, but not eliminated, by “anchor scans”Reduced, but not eliminated, by “anchor scans”

– Possibly combine ICP with separate trackersPossibly combine ICP with separate trackers

Future WorkFuture Work

• Faster scanningFaster scanning– Better stripe boundary trackingBetter stripe boundary tracking

– Multiple cameras, projectorsMultiple cameras, projectors

– High-speed cameras, projectorsHigh-speed cameras, projectors

• Application in different contextsApplication in different contexts– Cart- or shoulder-mounted for digitizing Cart- or shoulder-mounted for digitizing

roomsrooms

– Infrared for imperceptibilityInfrared for imperceptibility

AcknowledgmentsAcknowledgments

• Collaborators:Collaborators:– Li-Wei HeLi-Wei He

– James DavisJames Davis

– Lucas PereiraLucas Pereira

– Sean AndersonSean Anderson

• Sponsors:Sponsors:– SonySony

– IntelIntel

– IntervalInterval

Real-Time 3D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University...

Documents

Transcript of Real-Time 3D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University...

An RGBN Benchmark - Princeton Universitygfx.cs.princeton.edu/pubs/Berkiten_2016_ARB/rgbn_benchmark.pdf · An RGBN Benchmark Sema Berkiten Szymon Rusinkiewicz Princeton University

QSplat: A Multiresolution Point Rendering …graphics.stanford.edu/papers/qsplat/qsplat_paper.pdfQSplat: A Multiresolution Point Rendering System for Large Meshes Szymon Rusinkiewicz

How Well Do Line Drawings Depict Shape? Forrester Cole Kevin Sanik Doug DeCarlo Adam Finkelstein Thomas Funkhouser Szymon Rusinkiewicz Manish Singh RutgersPrinceton.

Atomic Volumes for Mesh Completion Joshua Podolak Szymon Rusinkiewicz Princeton University.

Efficient Variants of the ICP Algorithm Szymon Rusinkiewicz Marc Levoy Stanford University.

Finite State Machines (FSMs) and RAMs and inner workings of CPUs COS 116, Spring 2010 Guest: Szymon Rusinkiewicz.

Real-Time 3D Model Acquisition · Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University Stanford University† Abstract The digitization of the 3D shape of real objects

Computer Graphics...Administrative Matters • Instructors Szymon Rusinkiewicz TA #1: Vladimir ( Vova) Kim TA #2: TBA • Book Fundamentals of Computer Graphics Peter Shirley and Steve

3D Photography: Stereo - CVG @ ETHZ · 3D Photography: Stereo Marc Pollefeys, ... (some slides from Szymon Rusinkiewicz, ... Triangulation Scanner Issues

EFFICIENT VARIANTS OF THE ICP ALGORITHM Szymon Rusinkiewicz Marc Levoy.

This is a Mr. Levoy PowerPoint

Real-time Acquisition and Rendering of Large 3D Models Szymon Rusinkiewicz.

Temporally Coherent Completion of Dynamic Shapes Hao Li, Linjie Luo, Daniel Vlasic, Pieter Peers, Jovan Popović, Mark Pauly, Szymon Rusinkiewicz ACM Transactions.

Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Estimating Curvatures and Their Derivatives on …Estimating Curvatures and Their Derivatives on Triangle Meshes Szymon Rusinkiewicz Princeton University Abstract The computation of

COS 429: Computer Vision · 2009-09-17 · COS 429: Computer Vision •Instructor: Szymon Rusinkiewicz smr@cs.princeton.edu •TA: Linjie Luo linjiel@cs.princeton.edu ... properties

Computer Graphics - Princeton University Computer · PDF fileComputer Graphics Szymon Rusinkiewicz Princeton University . COS 426, Spring 2012 . Overview • Administrivia People,

“QSplat : A Multiresolution Point Rendering System for Large Data Meshes” Authors: Szymon Rusinklewicz Marc Levoy Presentation: Nathaniel Fout.

3D Scan Matching and Registration ICCV 2005 Short Course Szymon Rusinkiewicz Michael Kazhdan Benedict Brown Princeton University Johns Hopkins University.

Shape Matching and Anisotropy Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewicz Princeton University Michael Kazhdan, Thomas Funkhouser, and.