Realtime Video Stabilization for Moving Platforms

Paper : Real-time Video Stabilization For Moving Platforms

21st Bristol UAV Systems Conference – April 2007 of 14

Real-time Video Stabilization For Moving Platforms

Eddy Vermeulen

Barco nv

[email protected] Tel: +32 (0)56 233 202

ABSTRACT This paper describes a method and practical implementation to stabilize video sequences obtained from a camera mounted on a moving platform. Specific for these platforms is a twofold camera motion. The first one is from the motion path of the vehicle itself which has to be followed as closely as possible. The second one is due to mechanical vibrations of the mounted sensor and/or environmental influences. The second motion disturbance has to be filtered out as much as possible from the camera image. From a market perspective, the paper also discusses the need for electronic stabilization in defense & security markets, as well as some future impacts of technology and market trends.

BIOGRAPHY Eddy Vermeulen has worked as a product manager for the international market in the defense and security market unit of Barco since 2004. As product manager, he supports and helps define imaging and network-centric products. He has worked at Barco in various product support positions in the defense, security, and air traffic control markets since 1999. Prior to Barco, he worked in technical support positions in the commercial computing and networking markets.



Contents Real-time Video Stabilization For Moving Platforms ............................................................ 1

ABSTRACT.................................................................................................................. 1 BIOGRAPHY ............................................................................................................... 1

Contents................................................................................................................................ 2 Introduction .......................................................................................................................... 3 The Need for Real-time Stabilization .................................................................................... 3

UAV / UGV / USV Platforms ........................................................................................... 3 Ground Vehicle Platform................................................................................................... 3 Naval Platform.................................................................................................................. 4 Mechanical Versus Electronic Stabilization....................................................................... 4

The Stabilization Process ...................................................................................................... 6 Overview .......................................................................................................................... 6 Registration....................................................................................................................... 6 Filtering the Global Motion Parameters ............................................................................. 6 Warping ............................................................................................................................ 7 Video Output Formatting .................................................................................................. 7 Optional Generic Warp Pre-processing.............................................................................. 8 Controlling the process...................................................................................................... 9

Practical Implementation..................................................................................................... 10 Overview ........................................................................................................................ 10 Component Implementation : Computational Performance of FPGAs ............................ 10 Board Implementation : Acquisition and Visualization .................................................... 11 System Implementation : External Control & Tuning ...................................................... 12

Future Impact of Current Trends ......................................................................................... 14 Sensor Resolution ........................................................................................................... 14 Processing Components................................................................................................... 14



Introduction This paper describes a method and practical implementation to stabilize video sequences obtained from a camera mounted on a moving platform. Specific for these platforms is a twofold camera motion. The first one is from the motion path of the vehicle itself which has to be followed as closely as possible. The second one is due to mechanical vibrations of the mounted sensor and/or environmental influences (e.g. wind). The second motion disturbance has to be filtered out as much as possible from the camera image. The first part of this paper looks into the need and advantages of electronic video stabilization, specifically related to the defense and surveillance markets. The second part describes a stabilization process and associated tuning issues. The third part handles the physical implementation. Finally, some future impacts of technology and market trends are considered.

The Need for Real-time Stabilization

UAV / UGV / USV Platforms UAV platforms used to be mainly intelligence gathering platforms, obtaining aerial digital still images or making video recordings for analysis or mapping purposes once the vehicle was recovered. Today, an increasing number of UAV platforms are being equipped with streaming video solutions for immediate observation. This has become possible due to the increasing payload capabilities, increasing data communication bandwidth, and improvements in video compression technology. The result is a wider range of applications that require direct view capabilities. A soldier in the field benefits from fast environmental reconnaissance without going into the risk area himself. A UAV that is deployed for border patrol can now provide live tracking of suspicious movement to the operator in the ground control station. The value of motion video information from mobile platforms may, however, be affected by the uncontrollable motion of the sensors. All of these platforms have to deal with mechanical vibration caused by engine parts. UAV mounted sensors are unstable due to unexpected roll, yaw, and pitch caused by atmospheric conditions. UGV mounted sensors suffer from terrain roughness. USV mounted sensors deal with unpredictable wave motion. Detecting or tracking small targets of interest in this unstable motion video is very difficult and exhausting to an operator. Remote platforms often use radio links to transfer video. The limited bandwidth of these carrier signals requires video compression prior to the transfer of the signal. A common compression technique such as MPEG relies heavily on intra-frame compression. Instead of transmitting full picture contents, only differences between sequential frames is encoded and embedded in a compressed video stream. As stabilization will eliminate sudden jumps from a video sequence, it will allow a more efficient encoding when applied at the sensor platform. In all of these applications, stabilized video improves the image quality and results in improved detection capabilities of the operator.

Ground Vehicle Platform Army warfare today is shifting from a symmetrical to an asymmetric threat. Conflicts are no longer between two nation armies on a well-defined battlefield, but rather small dispersed units of individuals that move in unpredictable ways. The battle scene has moved to remotely deployed units, often in urban territory with an environment of low visibility and possible threats around every corner. With this shift in operational environment, it is of no surprise that many army programs worldwide are focusing on small combat vehicles, rather then large main battle tanks. Examples of such programs are Future Combat Systems (FCS) in the US, Future Rapid Effect System (FRES) in the UK, Splitterskyddad EnhetsPlattform (SEP) in Sweden, and Bulle Opérationnelle Aéroterrestre (BOA) in France.



These vehicles must be able to operate fully enclosed, securing the crew inside the armor protection. During such operations, driver and commanders have to rely on externally mounted electro-optical sensors to view the environment. Engine vibration and rough terrain cause a lot of vibration in these images making it hard to interpret the imagery or to drive the vehicle. Many of the sub-programs are looking into enhancing local situational awareness and driver viewing systems. Also, turret modules and weapon systems are often controlled through a long-range gimbal system. Firing weapons is probably the worst cause of high frequency vibration of those sensors.

Naval Platform Surveillance camera's on ships are typically used with low FOV to scan the horizon, the coast (in case of littoral operations), or the sky for threats. They are typically mounted on high masts that are subject to mechanical vibration, the movement of the sea, and atmospheric conditions. Also, gun control is often done remotely in the command center of the ship, based on gun-mounted cameras. These are subject to high vibration when in use. In fact, images from naval platforms are often the most difficult ones to stabilize because of the low contrast image content (think about dark weather conditions, looking at a grayish water horizon against a misty sky).

Mechanical Versus Electronic Stabilization With most cameras mounted inside gimbals offering mechanical stabilization, one may wonder whether there still is a need for electronic stabilization. There are several reasons why mechanical stabilization does not exclude the use of electronic stabilization. First, mechanical stabilization does not remove all image vibration. The capability of a gimbal to compensate for vibration is typically specified by the minimum angular movement that can be mechanically stabilized. It is measured in micro-radians. Based on this stability limit, the camera resolution and the field of view, one can calculate the remaining picture movement in pixels at the observing workstation as illustrated in the tables below.

Horizontal transition in pixels 768 x 576

Stability (uRad) 5 10 20 50 75 100 200 FOV (°)

0,25 1 2 4 9 13 18 35 0,3 1 1 3 7 11 15 29 0,5 0 1 2 4 7 9 18 1 0 0 1 2 3 4 9 5 0 0 0 0 1 1 2

10 0 0 0 0 0 0 1 20 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0

Horizontal transition in pixels 1920 x 1080

Stability (uRad) 5 10 20 50 75 100 200 FOV (°)

0,25 2 4 9 22 33 44 88 0,3 2 4 7 18 28 37 73 0,5 1 2 4 11 17 22 44 1 1 1 2 6 8 11 22 5 0 0 0 1 2 2 4

10 0 0 0 1 1 1 2 20 0 0 0 0 0 1 1 30 0 0 0 0 0 0 1

camera resolution NTSC : 640 x 480 PAL : 768 x 576 HD : 1280 x 720 full HD : 1920 x 1080

Field of View 0,2° - 30°

Stability = Minimal mechanically stabilized angular movement 10 - 100 microRadians



The rows in these tables represent different field of view (FOV) operations. The higher an operator zooms in with the camera, the lower this FOV is and the smaller the observed region is. For an airborne platform such as a UAV looking at specific targets on the ground, a very narrow FOV (0,25 to 5 degrees) is quite common. The columns in these tables represent different stability specifications, the capability to mechanically compensate for vibration. This is a gimbal specification. The calculated values represent the number of pixels a video image is shifted as a result of the remaining uncompensated movement. Values of 0 represent a sub-pixel displacement. This calculation is presented for two types of sensor resolution : a typical PAL camera and a high resolution HD sensor. For overview purposes, the resulting image shifts are divided in three regions : sub-pixel vibration, small displacements (less then 4 pixels) and larger displacements. Per these calculations, a narrow FOV of 1° on a PAL camera and a common stability of 100 uRad results in a picture shift of 4 pixels. In a market trend targeting higher resolution EO-sensors, the effect will increase. A full high definition sensor would have a remaining vibration of 11 pixels under the same circumstances. A second reason in favor of electronic image stabilization is cost. High performance gimbals can provide stabilization as good as 10 micro-radians, however, this kind of optical system is extremely expensive. A more affordable gimbal system typically has a mechanical stabilization specification of 100 micro-radians. Cost considerations drive the need to add electronic stabilization. Thirdly, the more advanced mechanical stabilization solutions come with a certain weight and volume impact. On an airborne platform such as a UAV, payload weight is a critical factor that affects range and endurance. Not only does electronic stabilization come at a lower weight when embedded in the platform, it can also be integrated at a remote viewing position. This post-processing option eliminates all weight and volume impacts on the moving platform itself. A further advantage of electronic stabilization is its flexibility in control. As opposed to mechanical stabilization, an operator at the observation console, or smart software, can dynamically adjust or even enable / disable the stabilization process and its parameters. Software can detect operator interactions such as panning and zooming and apply immediate changes in the movement compensation of the image. This avoids the typical transition latency caused by the algorithm trying to compensate an intentional motion. Speed and change in direction of the platform can be dynamically used to alter the motion compensation in selected directions or at expected shift speeds. In certain cases, operators may want to disable the stabilization processing because an unstabilized image may provide more information. This is best explained by the example application of looking through a grid, such as a gate with many vertical bars. When moving laterally in front of such an object, the human mind integrates the different picture frames and 'memorizes' them, which will give the impression of what one sees behind the bars. A stabilized picture will partially hide the details behind the intersections and may, as such, hide important visual information. One can conclude that mechanical stabilization does not exclude electronic stabilization; rather there are good reasons, depending on the platform environment, to choose a combined solution where both methods complement each other's weaknesses.



The Stabilization Process

Overview Electronic video image stabilization is basically a three-step process. The first step is called image registration, which is estimating the global motion parameters. The second step is applying a filter on these motion parameters to filter out the motion of the vehicle. Finally, a warping function is applied to the input image with the inverse of the previous motion vectors. After this process, a final video output formatting is performed to handle the borders of the warped image for which no current information is available. The described solution also includes an additional pre-processing warp function which can optionally be applied to deal with lens distortions. The whole process needs to be controllable and tunable, preferably in a dynamic way to adapt to changes in environment and to user interactions.

Registration Displacement of one frame to the next is defined by a horizontal translation, a vertical translation, and a rotation component. The task of the registration part of the algorithm is to find these 3 global motion parameters to which a new incoming frame must be subjected to fit as closely as possible to the previous frame. In fact, a perfect match will not be possible because in practice, when the camera is moving, the background is never stable at all. There might even be objects in the scene that are physically moving. The goal of a good stabilization algorithm is to follow the moving background as closely as possible but to filter local motion in the output sequence that is not caused by the movement of the camera. The registration algorithm starts from an estimate of the global motion parameters. The new frame is then pre-warped and compared to a reference frame. The remaining motion will be small and provides correction factors for the estimated values. After a few iterations of this process, the remaining motion parameter error should be close to zero under conditions that most of the pixels (true for the background) have the same translation and rotation. It is important to perform the registration on the whole frame or on a large enough, configurable region of interest inside the picture. Not doing so may cause the stabilization to operate erroneously. Imagine for example what would happen if an object-oriented stabilization algorithm considers a drop of water or dust on the lens to be the object of interest. This is especially true for rugged environmental conditions such as in the defense market. Considering the whole image content can be a huge computational load for any device to accomplish on moving video at 30 frames per second. This can be simplified by a smart selection of the registration algorithm and working on an input frame with decreased resolution.

Filtering the Global Motion Parameters From the calculated motion parameters, an apparent motion velocity and frequency results. The intention is to follow the slow camera motion but compensate for fast mechanical vibration and unexpected changes. In this

global motion parameters

Filter Registration

Warp

Input Frame Output Frame

Pre-processing

Warp



step, a low-pass filter is applied that maintains the low-frequency motion in the picture and rejects the high-frequency motion at a selected cutoff frequency. The implementation of the algorithm must allow adjustment of the cutoff frequency to be able to adapt to different types of platforms and different expected velocities. In fact, these frequencies can be changed dynamically while the stabilization process is continuing, and can be changed separately for each translation and rotation. This dynamic tuning can either be automatic, based on the average motion history of the picture or manually forced by means of a controlling software application. At the same time, the compensation is monitored for extreme corrections. This insures the end-result is never worse then the unstabilized picture, avoiding a self-oscillating effect that can theoretically occur in any closed loop process.

Warping Image warping is the geometrical mapping of an image from one spatial domain onto another. It looks like a projection of an image to another surface in a 3D space. This is a common process in graphics boards to apply textures (like a wallpaper) to a 3-dimensional object as illustrated below.

This process is used in the stabilization algorithm to map a new incoming frame back to the position of the previous frame. It is also used, as described below to fill borders and to optionally compensate for lens distortions.

Video Output Formatting The video is now stabilized but the missing data in the borders must still be filled in. This can be accomplished in several ways. The simplest solution is to fill this data with a predefined colour. By applying a scaling function to the image, these borders can be kept to a minimum. A second method is to keep the old information of the previous video frame(s). The third one is to warp the old information. The last method is preferred over the two others since this one gives a more natural viewing result for the same computational effort as the second and a much better result than the first.

Optionally, by displaying the stabilized video in a larger frame and preserving the information of previous frames for a longer time, one can visualize a history trail. This kind of picture composition is commonly referred

Fill Color

Fill with old frame

Warp old frame

Warp Transformation



to as mosaicing and is especially useful for UAV reconnaissance applications. While this format may add to the comprehensiveness of the image, it should be noted that it may also introduce artifacts due to moving objects in the scene. In some cases, displaying non real-time information on a real-time system may even be prohibited.

Optional Generic Warp Pre-processing A typical camera lens is not geometrically perfect and can induce all kinds of distortions such as barrel or pincushion distortion.

Application experience shows that the impact of those distortions on the stabilization process is generally so small, that we can neglect them completely for most applications. However, when the distortion becomes too big, the same warping process as described above can be used as a pre-processing step to eliminate the distortions as illustrated below. This step is optional and can be enabled or disabled on demand.

Object Pincushion distortion Barrel distortion

Warp Transformation



Controlling the process Controlling the full stabilization process involves controlling a number of parameters that apply to the individual sub-processes described above :

- Controlling the location and size of a region of interest for the registration process in order to : o force the stabilization to focus on objects of interest inside the picture frame o ignore fixed parts of the input signal e.g. inserted overlay graphics

- Controlling the iteration process of the registration. More iteration may give a better result but increase the processing time.

- Setting the cutoff frequency of the filter process either automatically or user-enforced. This can be of interest when the nature of the vibration is known in advance, or when the vibration frequency is very close to the motion of the vehicle

- Defining the maximum correction of the warping algorithm (which is essentially defining the maximum size of the fill borders) in order to :

o ignore known intentional picture movement in either translation or rotation (e.g. turning of a vehicle, climb or descend of a UAV)

o avoid over-compensation - Enabling / disabling the process to avoid the processing delay e.g. when the vehicle is known to be

stationary or the vibration source is gone. By implementing the stabilization so that it reads these parameters at each operation from modifiable registers, the process can be controlled dynamically. A change in parameters will be effective at the next input frame.



Practical Implementation

Overview The process described above is computationally quite intensive. To perform this in a real-time way, dedicated video processing hardware is required. Some additional functionality is required to implement electronic video image stabilization as a full solution :

- Acquiring the video input signal : While many digital camera interface standards are appearing in the market, a lot of basic cameras are still analog composite output. An accurate digitization is the first important step to a useful picture. If the video is from a remote platform and transmitted over a limited bandwidth link, some decompression may be required to re-create the video signal.

- Visualizing the stabilized video : An operator console is typically composed of video and graphical information such as maps, instrumental information, etc. In some cases, graphical information needs to be superimposed on the video image such as target markers and geo-referencing information. Mixing the stabilized video inside this graphical user interface and providing graphics overlay capabilities is essential in this solution.

- External control of the process : In most cases, enabling stabilization is not always required. When a camera is moved to a different view or while the lens is being zoomed in to a region of interest, the stabilization process would inject an unwanted delay in the operator view. Furthermore, the process can benefit from information provided by other sensors such as gyro or operator initiated tuning. A complete system implementation should interpret different input signals and enable/disable the stabilization as appropriate, or even dynamically control and tune the process.

Component Implementation : Computational Performance of FPGAs The stabilization scheme as described above involves a large number of pixel transformations to be applied on the horizontal camera image grid, especially in the registration and warping phases. While the nature of the transformations is relatively straightforward, the number of individual operations is large and proportional to the resolution of the sensor. This kind of operation benefits from the massive parallel computational capability of field programmable gate arrays (FPGAs). An FPGA is essentially a silicon device that incorporates a large amount of programmable logic functions such as AND, OR, XOR, NOT. Specific function blocks are generated by properly programming the interconnects of these logic components. Different function blocks can be programmed to perform different types of operations and a large number of equal function blocks can be programmed to perform parallel operations on large blocks of input data. The quick evolution of silicon technology has lead to a constant growth of the number of logic gates available in one FPGA device that basically doubles every 18 months. An additional consequence of this is the affordability of these devices, providing increasing performance characteristics. Another recent trend in FPGAs is the occurrence of hybrid technologies which combine embedded processors and related interface functionalities with FPGA architectures in a single device. The programmable nature of an FPGA allows easy modification of the embedded algorithm to perform different functions or easy tuning for specific requirements, without expensive hardware modifications or upgrades. This property guarantees the value of the long-term investment typically associated with defense systems. It also supports the trend for spiral development initiatives of many system integrators and defense departments. The Xilinx Virtex-II Pro family of FPGAs provides a number of features that are well suited for this application :

- a large capacity, capable of handling the large number of transformations required for high resolution motion video stabilization



- a high speed serial interface to quickly move around the large amount of data associated with digital video

- an embedded PowerPC to control the process and handle overall computational functions

Board Implementation : Acquisition and Visualization A practical implementation of real time video stabilization is found in Barco's FlexiVision III product platform. FlexiVision III is a single-board solution for real-time video and graphics mixing, display recording, real-time video streaming and image processing such as stabilization.

Mixing BlockControl Block(PowerPC) 8 bit parallel RIO

Gigabit Ethernet 1

Gigabit Ethernet 2

Master Graphics In

Master GraphicsOut

Universal FrameGrabber Function Block II Function Block III

PC

I or

VM

E

Video In Video In Video In

Ser

ial P

ort

UniversalFrame

Grabber(UFG)

JPEG2000Codec(J2K)

ProgrammableVideo

Processor(PVP)

It is made up of four modular components that can be combined in a variety of ways to perform various functions e.g. video acquisition, stabilization and windowing :

- The Carrier board is based on a RapidIO switch fabric architecture that is tightly coupled with a local on-board PowerPC. The hardware is designed to support three optional mezzanine modules, one of which is a fixed framegrabber. The control block contains the on-board PowerPC and runs an embedded Linux operating system.

- The Universal Frame Grabber (UFG) module is a dual channel analog video frame grabber that is capable of supporting virtually any standard analog video format. This part performs the front-end sensor acquisition when dealing with directly connected analog sensors.

- The JPEG 2000 (J2K) module is a hardware JPEG2000 based codec module. With this wavelet domain-based compression technique, it is possible to select the compression level from lossless to very high, and transport video over industry standard TCP/IP networks with low latency. As this encoding algorithm is quite computationally intensive, it should be performed in hardware for real-time systems. This part allows for networked acquisition of remote sensors, rather then direct connectivity.

- The Programmable Video Processor (PVP) module accepts digitized video from any of the other modules and processes it with its on-board FPGA. The PVP provides the core for the image processing capabilities and is where the stabilization algorithm resides.



Operator Workstation

FlexiVision III carrier

PVP

Graphics Board

Master In

Master Out

UFG

Unstable VideoSource

Video In

Stabilized Video WithGraphics Overlay

For a video stabilization solution, the UFG module may capture any direct analog sensor input, or the J2K module may uncompress a networked JPEG2000 video stream. A PVP module can be loaded with the stabilization algorithm, and the outgoing video can be combined in real-time with computer-generated graphics and overlay to provide a complete operator workstation solution. Latency is extremely important for real-time systems, especially those that perform interactive control, based on the visual information. E.g. a remote pilot of a UAV expects to see an immediate effect if he makes the vehicle turn. A long processing delay causes the image to lag behind the actual maneuvering commands. This would appear very confusing to a human operator and makes the system practically unusable for direct control. The single board implementation described above allows the stabilization and mixing to occur within the time window of 2 to 3 frames, which is practically unnoticeable to a human observer.

System Implementation : External Control & Tuning Full system integration of video stabilization involves software for control and tuning. The figure below is a sample high-level diagram of such a system for vehicle control, and the overall layout is applicable to most applications.

Pilot Control

Vehicle Controlsub-system

Visualizationsub-system

Control

User Interfacesub-system

Control

Graphics

Video

GPS, Gyro, ...

Vehicle Platform

Control

GPS, Gyro, ...

Communicationsub-system

In this diagram, we consider 4 different processes :



- The vehicle control sub-system handles the driving of the vehicle. This can be an automatic system or can be based on user-input through a piloting control interface. Often, this module loads a pre-programmed mission into the vehicle for further autonomous operation. As a feedback, the vehicle control module receives actual positional information from the vehicle.

- The communication module handles all communications with the vehicle in both directions. This could be over radio-link, satellite link, or direct wired control in the case of small ground robots.

- The user interface generates graphics information to be presented to the operator and translates the operator input to control operations of the other modules.

- The visualization sub-system is responsible for combining user-interface graphics and sensor information to generate the actual operator picture.

In this diagram, the stabilization process is part of the visualization sub-system. It receives video through the communication module, via either a direct analogue or digital signal or via a compressed video stream. The stabilization can be controlled by inputs from the other sub-systems :

- An operator may decide to enable / disable the stabilization process by clicking an on-screen button. He can indicate the region of interest on the screen by drawing a box over the video. In these cases, the user interface sub-system controls the stabilization.

- If the vehicle control is aware of the intentional motion of the vehicle, either through pilot control or through automatic guidance, this information can be used to change the way the stabilization process works to optimize the result. E.g.: if the vehicle and it’s attached sensor move to the right, the horizontal motion compensation can be temporarily minimized.

- Positional and movement information from the vehicle itself can be used to optimize the stabilization process dynamically and in an automated way.

Video latency of the complete system now has to be considered over the whole transport chain. Each element may contribute to the total latency of the video visualization. In fact, any of the subsystems described typically contains input- and/or output-framebuffers. A video frame that enters the sub-system line by line is stored into these memory blocks for processes that require a full field or frame on which to operate. As such, each of those buffer memories may add an additional field or frame of delay. Even the display device itself typically contains up to 3 buffers delaying the actual visualization of the picture. Hence the importance of keeping latency minimized in each single block of the system. If a direct connection to the sensor system exists, one typical method to minimize delays is synchronization of all sources. A separate synchronization signal is distributed throughout the system that makes sure all sources are generated at a common pace. A common approach is to use the refresh rate of the user interface graphics and insure it happens at the exact same moment a new video frame is generated.



Future Impact of Current Trends

Sensor Resolution A common trend in all EO-sensor devices is an increase of the picture resolution, mainly driven by the commercial television world. While PAL and NTSC resolutions are still quite common today, new devices are coming into the market with high-definition (HD) resolution. The term HD is commonly used for every sensor resolution which is higher then the common PAL and NTSC, covering a wide range of different resolutions. Full HD refers to the highest resolution (1920 x 1200), but most HD sensor devices are still limited to 1280 x 720. As already indicated in the table on page 4 of this paper, an increase in resolution, without a change in mechanical vibration tolerance, will result in more pixels being shifted. As such, the importance of stabilization grows in order to take advantage of the higher level of detail that an HD sensor provides. Increasing resolution also brings additional difficulty to the processing functions. Note that doubling both the horizontal and vertical number of pixels results in four times the number of pixels to be processed. Full HD results in 7.5 times the pixel quantity of NTSC. Not only does this exclude software PC-based processing, in requires very high performance of dedicated hardware. A high resolution sensor only brings a benefit if it is also visualized to its fullest extent. The trend for higher resolution display devices has been ongoing for a number of years. This also places higher requirements on the output stage of the visualization sub-system. Combining graphics with live sensor data now also requires higher bandwidth mixers. The input stage of the visualization sub-system is also subject to some changes. Higher resolution comes with higher data bandwidth, and to fully exploit its advantage it typically is no longer transported in an analog composite way. The composite PAL / NTSC signals have the disadvantage that some color bandwidth is sacrificed to limit the total signal bandwidth. To avoid this loss, HD signals are typically transported as a high-speed digital serial signal. The raw bandwidth of these signals often exceeds the 1Gb limit of typical contemporary networks (e.g. SMPTE 292M standard defines a nominal data rate of 1,485 Gb per second). As a result, input requirements of the video capturing hardware become more stringent, and high performance data compression becomes a critical need.

Processing Components As device performance keep going up, it becomes possible to embed more complex functionality, or multiple functions, for real-time processing applications. These additional functionalities may serve to offload the operator, automate the system, or simply improve efficiency and performance. A basic motion detection system that highlights and/or tracks moving objects within video frames can help an operator to more easily distinguish points of interest in the video information. This kind of detection processing becomes more difficult as the video platform itself is subject to motion. In such applications, stabilization becomes a pre-processing step for the motion detection algorithm. Opposed to what one would expect, performance does not come with weight, cost and volume. The miniaturization trend continues as more functionality gets integrated into the same package, e.g. hybrid FPGAs that contain PowerPC cores. Higher performance at approximately the same price means a lower relative cost for the same capability. As a consequence, vide processing features such as stabilization become available as embedded functionality, either at the sensor side or at the display side. What used to be a ‘dumb display’ is available today as a smart device that includes advanced local processing functions and computing capabilities. Too often, higher performance has come with higher power demands. Heat dissipation is also closely related with power consumption. Today, one of the main challenges for system designers is thermal management. Technologies such as liquid flow through cooling, spray cooling and Peltier elements enjoy renewed attention in an effort to cope with the typical heat dissipation of modern processing and graphics devices. Luckily, component manufacturers have also understood this integration issue and are now paying increasingly more attention to power consumption, e.g. by having components operate at lower voltages and using more efficient semi-conductor materials.

Realtime Video Stabilization for Moving Platforms

Documents

Transcript of Realtime Video Stabilization for Moving Platforms