Post on 29-Mar-2015
Super-Resolution Texturing for Super-Resolution Texturing for Online Virtual GlobesOnline Virtual Globes
Diego Rother, Lance Williams and Guillermo SapiroDiego Rother, Lance Williams and Guillermo SapiroUniversity of Minnesota and Google, Inc.University of Minnesota and Google, Inc.
Internet Vision Workshop (CVPR 2008)
Online Virtual Globes
Online Virtual Globes
ClipMap pyramid.
Server Client
Requests
Tiles
• Problems:– Huge storage space required (one of the largest organized collections of imagery on the Internet).– Expensive acquisition and high transmission bandwidth.– Interpolation beyond available resolution: unnatural.
User
Earth surface
– Stereotypical. – Rapidly changing.– Identity important, details not.
Proposed Solution
• Proposition:– Synthesize, on the client, details for the
lower pyramid levels.– Using super-resolution techniques.– Harnessing labels and textures samples
from users (wiki model).
• Requirements:– Fast.– Seamless transition between layers.
Results
User input 1: Labels
Using interactive segmentation as in:Bai, X. and Sapiro, G., "A geodesic framework for fast interactive image and video segmentation and matting." ICCV, 2007.
Original frame User provided labels
Class1
(grass)
Class2
(path)
User input 2: Keyframed Texture
Keyframe1
Keyframe2
Keyscale1
Keyscale2
• User provides the texture pyramid:
Texture pyramid.
e.g., in meters/pixel
Synthesis of a New Layer
Input:from Server
Output:New Layer
Input:from Users
ClipMap pyramidLabels
Texture pyramid.
System Overview
Undo Color Matching
Texture Transfer
Color Matching
Interpolation
Interpolation
Pyramid of Training Textures
(from users)
Inputs
Last
Lay
er(fr
om s
erve
r)La
bels
(from
use
rs)
Outputs
New
Lay
erN
ew L
abel
s
Selection of the Training Image
Texture transfer: 1st passTraining Texture (from the texture pyramid)
Color matched image, without high frequencies
• Wei, L. and Levoy, M., "Fast Texture Synthesis using Tree-structured Vector Quantization." SIGGRAPH, 2000.• Efros, A. A. and Leung, T. K., "Texture Synthesis by Non-parametric Sampling." ICCV, 1999.
Small contexts → Fast
Y Channel(luminance)
I and Q Channels (chrominance)
Mean and Gradient Only Mean
Similarity between contexts considers
Source Locations
Produces 1-Pixel Patches:Contiguous areas copied verbatim from the training texture.
Texture transfer: 2nd, 3rd and 4th passes
• Ashikhmin, M., "Synthesizing Natural Textures." ACM Symposium on Interactive 3D Graphics. 2001.
Few candidates → Fast
Training TextureColor
ChannelPass
2nd
3rd and 4th
Y Channel(luminance)
I and Q Channels (chrominance)
Similarity between contexts considers
Produces Bigger Patches
Texture transfer from the same texture
Texture Pyramid
1st synthetic frame
2nd synthetic frame
• Patch interior (lilac and violet) directly copied.
• Patch boundaries (pink) synthesized in 4 passes as before.
• Doubles the patch size.
ClipMap Pyramid
Results: Maracanã
No texture transferred
No texture transferred
No texture transferred
No texture transferred
Texture pyramid (user input)
ClipMap pyramid (result)
Results: Maracana
No texture transferred
No texture transferred
No texture transferred
No texture transferred
ClipMap pyramid (result)
Source Locations (result)
Results: Field
No texture transferred
No texture transferred
No texture transferred
No texture transferred
No texture transferred
No texture transferred
No texture transferred
No texture transferred
No texture transferred
No texture transferred
ClipMap pyramid (result)
Texture pyramid (user input)
Source Locations (result)
Results: BeachClipMap pyramid (result)
Conclusions
• Proposed solution:– Reduces storage, bandwidth, and acquisition costs.– Improves appearance and information content.– Is fast (low dimensional contexts).