Mobile Visual Computing(Invited Paper)

Kari Pulli, Wei-Chao Chen, Natasha Gelfand, Radek Grzeszczuk,Marius Tico, Ramakrishna Vedantham, Xianglin Wang, Yingen Xiong

Nokia Research Center, Palo Alto, CA, USAfirstname.lastname@nokia.com

Abstract

Smart phones are becoming visual computing power-houses. Using sensors such as camera, GPS, and others,the device can provide a new user interface to the realworld, augmenting user’s view of the world with additionalinformation and controls. Combining computation with im-age capture allows new kind of photography that can bemore expressive than is possible to obtain with a traditionalcamera. New APIs allow harnessing more computationpower of a smart phone to visual processing than what onecan obtain from just a CPU.

1. Introduction

The modern smart phone is a compact visual computingpowerhouse. It has a capable CPU, and an increasing numberof mobile phones include a graphics processor (GPU). Theysport a high-quality color display, up to screen resolutionsof 640 × 480 and beyond. Most high-end phones have acamera that can take megapixel images and at least VGAresolution video. There are separate co-processors or DSPsfor image processing and encoding and decoding. Sensorssuch as GPS, electronic compass, and accelerometers helpin determining where the device is and what it points at, andfast data connection allows connection to large databases.

We cover two major application areas, followed by emerg-ing technologies that allow faster visual processing onmobile devices. We call the first area augmented reality.There various sensors, especially the camera, are used tomake sense of the world around the user, and the outputmechanisms, especially the display, are used to provide userwith information about objects and locations around her.

Camera phones are also beginning to replace dedicateddigital cameras. However, sometimes the nature of multi-purpose device and the requirement of compact size meancompromises in camera elements such as sensor size andlens optics. By taking several images and combining theimages using the general computational capacity of a smartphone, one can improve the quality of the images. Theimprovements could relate to overcoming some sensor lim-itations, or create new kinds of images that cannot be

captured with a traditional camera. This application area isoften known as computational photography.

Finally, we describe how the various processing elementsavailable in smart phones can be harnessed for visualcomputation. The CPU is readily available, and the secondgeneration of mobile GPUs provides programmable vertexand pixel shaders that can be used for image processing.Yet other processing elements such as DSPs are typicallynot readily available for application programmers; OpenCLallows easier programming and allocation of computation toCPUs, DSPs, and GPUs.

2. Augmented Reality

Augmented Reality (AR) means experiencing the realworld and augmenting the experience, often by adding im-ages of virtual objects or textual annotations over the scene.AR can provide a fundamentally better user experience on amobile system than is possible on desktop. The first task foran AR system is to recognize nearby objects, for examplevia visual recognition using camera images. If you want toaugment the images in real time, you have to also track theview and objects. Neither of these tasks is feasible unlessyou have a model of the world around you.

2.1. Object recognition

We have built an outdoors augmented reality system formobile phones that matches camera-phone images againsta large database of location-tagged images using a robustimage retrieval algorithm [11], see Fig. 1. Matching isperformed against a database of highly relevant features,which is continuously updated to reflect changes in theenvironment. We achieve fast updates and scalability bypruning irrelevant features based on proximity to the user.

Transmission and storage of robust local descriptors areof critical importance in the context of mobile distributedcamera networks and large indexing problems. We haveproposed a framework for computing a low bit-rate featurethat represents gradient histograms as tree structures whichcan be efficiently compressed [3]. Distances between de-scriptors can be efficiently computed in their compressedrepresentation, eliminating the need for decoding.

Figure 1. Our outdoors augmented reality system aug-ments the viewfinder with information about the objectsit recognizes in the image taken with a phone camera.

Object ID

Database

No

Yes

Yes

No

Capture

Image

Update Label

Locations & Render

Capture

Image

Detect Features

& Query Image

Render

Labels

Image

Database

Initialization

Object

Tracking

Track Known Features

with SURFTrac

Project Subgraph

Features to

Current Image

New Object

ID in View?

Update Feature

Locations

Update Tracked

Interest PointsSURFTrac and

Query ImageSwitch to New

Keynode

Found

Matches?Determine

Keynode

Figure 2. Real-time tracking and recognition pipeline.

2.2. Tracking

For continuous overlaying of text or graphics in ARapplications, one can simply repeat the same object recogni-tion step for every input image. However, tracking is muchcheaper than repeatedly trying to recognize objects in eachframe. Our viewfinder alignment algorithm [1] can track thecamera motion in real time on a mobile device, which makesit possible to interleave object recognition and tracking forreal-time augmentation. One can also accelerate trackingby using dedicated video encoding hardware and extractingmotion vectors [12].

In order to compensate for the longer latency in objectrecognition, it becomes necessary to pipeline tracking to-gether with object recognition. Load-balancing these twotasks becomes a tricky system design issue. Alternatively,we have developed an algorithm called SURFTrac [10] thattracks with the same features used in object recognition. Thismeans that tracking and object recognition can be performedcontinuously to achieve natural load balance between the

tasks. The SURFTrac pipeline is illustrated in Fig. 2.

Figure 3. A sample view of a generated instruction. Thedirection to take is displayed prominently, and a previewof the next step is shown on top corner.

2.3. Scene modeling

Having a 3D model of an environment and being able toregister the pose of images taken by a mobile phone useragainst that model creates new opportunities for richer andmore immersive augmented reality applications. In [5], weuse computer vision techniques to compute camera posesof image collections of landmarks and register them to theworld using GPS information extracted from the image tags.Computed camera poses allow us to augment the imageswith navigational arrows that fit the environment. We alsoutilize an image matching pipeline based on robust localdescriptors to give users of the system the ability to capturean image and receive navigational instructions overlaid ontheir current context.

3. Computational Photography

Camera phones are convenient computational photogra-phy platforms as they include an increasingly high qualitycamera together with a general purpose computing device.Here we introduce two computational photography applica-tions, high dynamic range imaging and panorama capture.

3.1. High Dynamic Range Imaging

Dynamic range of a scene is defined as the ratio betweenthe minimum and maximum brightness values present inthat scene. Many common scenes have dynamic range thatexceeds the maximum range of brightness values that canbe recorded by an image sensor, resulting in loss of detailin the shadows, clipped highlights, or both. HDR imagingis a computational technique that combines several regularimages taken at different exposures into a single image withexpanded dynamic range that better reflects the brightnessvariations in the scene.

Figure 4. First row: An exposure stack taken at a sculpture garden. Second row, from left to right: single bestexposure has areas that are either too bright or too dark, notice the clipped highlights on the building facade andloss of detail in trees and shadows; standard HDR restores those details, but creates ghosts of walking people;consistent HDR image generated by our algorithm.

The basic approach to HDR imaging assumes that nothingin the scene moves, so all images can be used to calculate aradiance value for each pixel. However, many scenes havemoving targets, such as people, and it is almost impossibleto take an image sequence of a tree in outdoors withoutany motion of its branches or leaves. Gallo et al. [4]developed a method that detects image locations that havechanged, selects a suitable anchor view based on whichthe final, consistent image is created, and uses as muchdata from other images as possible to calculate accurateradiance values at each pixel (see Fig. 4). In this way, nouser interaction is needed to generate an artifact-free highdynamic range image, making it possible for the camera toproduce images that more faithfully represent the capturedscene, even given the dynamic range limitations of currentcamera sensors and displays.

3.2. Capturing panoramas

We have developed a complete mobile panorama appli-cation that captures and stitches multiple high resolutionimages in order to create an image with a larger field of view.We track the camera motion and capture high-resolutionimages at appropriate times, register the images, map andresample them into spherical coordinates, and blend theminto a panorama.

Camera motion is roughly estimated by tracking consec-utive viewfinder frames captured at a high frame rate. Ouralignment algorithm [1] creates compact summaries of theframes that allow rapid tracking of the camera motion. Thehigh resolution images used to build the final panoramaare captured automatically as user pans the scene with

Figure 5. Panorama is automatically captured andconstructed as user pans the camera around the scene.

the camera. As the camera captures an image, the user isinstructed to stop moving to avoid motion blur (see Fig. 5).

Once the high resolution images are captured, they areregistered and blended into the final panorama. An unlimitedangle of view is enabled by mapping the captured imagesonto a sphere. We also register images in spherical coor-dinates. Image-based registration is adopted at the coarselevels of an image pyramid where the features are lessreliable, but feature-based matching is employed at the finerlevels of the pyramid. This registration approach is robust toboth moving objects in the scene and significant illuminationdifferences between images. A seamless image stitching isfinally achieved by labeling (selecting which input images

contribute to which areas of the output image [7]) andPoisson blending (taking the gradients of the input imagesand solving a consistent output image [8]).

4. Standards for Mobile Visual Computing

Images contain lot of pixels, and processing them requireslot of computation power. New APIs, and the hardwarethey allow application programmers to access, enable fasterexecution of image processing applications. Here we addressmobile 3D graphics standards, and a new standard, OpenCL,for high performance computing even on mobile devices.

4.1. Mobile Graphics APIs

Mobile graphics APIs such as OpenGL ES and M3G [9]enable new types of applications on mobile devices [2]. Theobvious ones include faster user interfaces with more eyecandy, interactive games, and browsing maps and web pages.However, they are useful also for image processing tasks. Forexample image warping can be accelerated easily 10-foldwith OpenGL ES 1.1 fixed functionality graphics pipeline,even when accounting for the data transfer overhead. Evenif the device does not have special graphics hardware, thehighly assembly-optimized software graphics engines cantypically process the pixels faster than regular image pro-cessing C-code. The second generation of graphics hardwareand the matching APIs (OpenGL ES 2.0 and M3G 2.0)allow more flexible pixel processing using programmableshaders, increasing the number of algorithms acceleratablevia graphics hardware.

4.2. OpenCL

High-end camera phones have much more processingpower than just the CPU, but usually the other processingunits are not easily accessible by the application program-mers. Their units may be dedicated to some particularprocess, such as the phone modem, voice processing, orvideo encoding, but those units can be idle when visualprocessing is needed. Even if the additional units wereavailable for the programmer, they may be difficult to pro-gram, each supporting a different instruction set. OpenCL,Open Computing Language, [6] is a new standard that hidesthe hardware differences under a unified abstraction layer.GPGPU (general processing on graphics processing units)has utilized the GPU instruction sets for other tasks such asimage processing. OpenCL generalizes the GPGPU conceptas it not only hides the details on which GPU the numbercrunching takes place, but also whether the processing isdistributed to GPUs, CPUs, or even DSPs, providing aunified programming model for all these execution units.OpenCL will make it easier to harness all the power ofmobile devices for visual computing.

5. Conclusion

For augmented reality, smart phones now provide compu-tation power, cameras and other sensors, data connectivity,and good displays, and most importantly, are ubiquitous andeasy to use when actively moving. The key tasks of near-real-time image recognition and real-time tracking have beendemonstrated on smart phones, and modeling the real worldto support them in large scale is becoming feasible. Cameraphones are also ideal computational photography platforms,providing easy programming access to researchers, and moreversatile I/O capabilities than traditional digital cameras.New standards such as OpenGL ES and OpenCL makethe varied computing hardware on mobile phones moreaccessible to application developers.

References

[1] A. Adams, N. Gelfand, and K. Pulli. Viewfinder alignment.Computer Graphics Forum, 27(2):597–606, 2008.

[2] T. Capin, K. Pulli, and T. Akenine-Moller. The state of theart in mobile graphics research. IEEE Computer Graphicsand Applications, Jul-Aug 2008.

[3] V. Chandrasekhar, G. Takacs, D. Chen, S. S. Tsai,R. Grzeszczuk, and B. Girod. CHoG: Compressed His-togram of Gradients: A Low Bit-Rate Feature Descriptor.In IEEE Conf. on Computer Vision and Pattern Recognition(CVPR09), 2009.

[4] O. Gallo, N. Gelfand, W.-C. Chen, M. Tico, and K. Pulli.Artifact-free high dynamic range imaging. In IEEE Int. Conf.on Computational Photography (ICCP09), 2009.

[5] H. Hile, R. Grzeszczuk, A. Liu, R. Vedantham, J. Kosecka,and G. Borriello. Landmark-Based Pedestrian Navigationwith Enhanced Spatial Reasoning. In 7th InternationalConference on Pervasive Computing. Springer, 2009.

[6] Khronos Group. The OpenCL Specifica-tion, Version 1.0, 2008. Available fromwww.khronos.org/registry/cl/specs/opencl-1.0.33.pdf.

[7] V. Kwatra, A. Schodl, I. Essa, G. Turk, and A. Bobick.Graphcut Textures: Image and Video Synthesis Using GraphCuts. SIGGRAPH, 2003.

[8] P. Perez, M. Gangnet, and A. Blake. Poisson image editing.ACM Transactions on Graphics (SIGGRAPH ’03), pages313–318, 2003.

[9] K. Pulli, T. Aarnio, V. Miettinen, K. Roimela, and J. Vaarala.”Mobile 3D Graphics with OpenGL ES and M3G”. MorganKauffman, 2007.

[10] D.-N. Ta, W.-C. Chen, N. Gelfand, and K. Pulli. SURFTrac:Efficient Tracking and Continuous Object Recognition usingLocal Feature Descriptors. In IEEE Conf. on Computer Visionand Pattern Recognition (CVPR09), 2009.

[11] G. Takacs, V. Chandrasekhar, N. Gelfand, Y. Xiong, W.-C. Chen, T. Bismpigiannis, R. Grzeszczuk, K. Pulli, andB. Girod. Outdoors Augmented Reality on Mobile Phoneusing Loxel-Based Visual Feature Organization. In MIR’08: Proceeding of the 1st ACM International Conference onMultimedia Information Retrieval, pages 427–434, 2008.

[12] G. Takacs, V. Chandrasekhar, B. Girod, and R. Grzeszczuk.Feature Tracking for Mobile Augmented Reality Using VideoCoder Motion Vectors. In ISMAR ’07: Proceedings of theSixth IEEE and ACM International Symposium on Mixed andAugmented Reality, 2007.