Raskar Npar04final

Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu

Image Fusion for Context Enhancementand Video Surrealism

Adrian IlieAdrian Ilie

UNC Chapel HillUNC Chapel Hill

Ramesh RaskarRamesh Raskar

Mitsubishi Electric Mitsubishi Electric Research Labs,Research Labs,

(MERL)(MERL)

Jingyi YuJingyi Yu

MITMIT


Dark Bldgs

Reflections on bldgs

Unknown shapes


‘Well-lit’ Bldgs

Reflections in bldgs windows

Tree, Street shapes

Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, YuBackground is captured from day-time scene using the same fixed camera

Night Image

Day Image

Context Enhanced Image


Mask is automatically computed from scene contrast


But, Simple Pixel Blending Creates Ugly Artifacts

Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, YuPixel Blending

Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, YuPixel Blending

Our Method:Integration of

blended Gradients


OutlineOutline

• Context Enhancement

• Gradient-based Fusion

• Video Enhancement

• Surrealism


Nighttime imageNighttime image

Daytime imageDaytime image

Gradient fieldGradient field

Importance Importance image Wimage W

Fina

l res

ult

Fina

l res

ult


Mixed gradient fieldMixed gradient field

GG11 GG11

GG22 GG22

xx YY

xx YY

II11

I2

GG GGxx YY


Reconstruction from Gradient Reconstruction from Gradient FieldField

• Problem: minimize errorI’ – G|• Estimate I’ so that

G = I’

• Poisson equation

I’ = div G

• Full multigridsolver

I’I’

GGXX

GGYY


Why Gradient-based ApproachWhy Gradient-based Approach

• Comparison of intensity values are important• Maintain gradients to capture local variations• Directly solve for desired gradients

• Maintain subtle details• Mix dissimilar images• No need for precise segmentation


ComparisonComparison

• Average– Subtle details are lost

• Pixel-wise blending– Sharp transitions


IssuesIssues

• Boundary conditions• Color shifts


Boundary ConditionsBoundary Conditions

• Assumed Neumann condition at borders, I’ · N = 0, • Enforced by haloing image with blacks


Color ShiftColor Shift• Mixing dissimilar images

• Goal: final image appearance matches input images at corresponding pixels

Ifinal(x,y) = c1Ipoisson(x,y) + c2

• SolveWi(x,y) Ioriginal(x,y) = c1

Ipoisson(x,y) + c2

• Each color channel reconstructed separately


OutlineOutline




• Surrealism


Enhanced video Note: exit ramp, lane dividers, buildings not visible in

original night video, but clearly seen here.

Overview of ProcessOverview of ProcessDay time image: By averaging

5 seconds of day videoOriginal night time traffic camera

320x240 video

Mask frame (for frame above): Encodes pixel with intensity change

Input

Output


AlgorithmAlgorithmFrame Frame NN

Daytime imageDaytime image

TimeAveraged TimeAveraged importance maskimportance mask


Final resultFinal result

Processed binary maskProcessed binary mask

Gra

dien

t fie

ldG

radi

ent f

ield

Frame Frame N-1N-1

Mixed gradient fieldMixed gradient field


OutlineOutline• Context Enhancement



• Related Work

• Surrealism


Related WorkRelated Work

• Spatio-temporal Composition– Duchamp (Nude descending a staircase)– Freeman 2002– Fels 1999, Klein 2002, Cohen 2003

• Gradient-based Techniques– Multi-spectral: Socolinsky 1999– Shadow removal: Weiss 2001– High dynamic range: Fattal 2002– Image editing: Perez 2003– Some at Siggraph’04


Rene Magritte, ‘Empire of the Light’

Surrealism


OutlineOutline




• Surrealism


Time-lapse MosaicsTime-lapse Mosaics

Maggrite StripesMaggrite Stripes

timetime


Time Lapse MosaicTime Lapse Mosaic


tt


Sunrise at NightSunrise at Night


BiSolar SystemBiSolar System


DiscussionDiscussion• User Experience

– More effective in conveying scene context– ‘Dreamy’ appearance– Nonrealistic : False conditions

• Applications– Tools for artists– Surveillance– Amusement park rides

• Performance– ~1 sec/frame for 320x240– ~ 3 min for 4Mpixel image


Image Fusion for Context EnhancementImage Fusion for Context Enhancement

• Nonrealistic but comprehensible context• Fusion using multiple images

– Enhancing night images with day bgrnd• Gradient-based fusion• Video surrealism tools

tt

Raskar Npar04final

Technology

Transcript of Raskar Npar04final