SALDRE on DM365 - TI E2E Community
Transcript of SALDRE on DM365 - TI E2E Community
SALDRE on DM365
User Guide
Literature Number: TBD May 2011
IMPORTANT NOTICE
Texas Instruments Incorporated and its subsidiaries (TI) reserve the right to make corrections, modifications, enhancements, improvements, and other changes to its products and services at any time and to discontinue any product or service without notice. Customers should obtain the latest relevant information before placing orders and should verify that such information is current and complete. All products are sold subject to TI’s terms and conditions of sale supplied at the time of order acknowledgment.
TI warrants performance of its hardware products to the specifications applicable at the time of sale in accordance with TI’s standard warranty. Testing and other quality control techniques are used to the extent TI deems necessary to support this warranty. Except where mandated by government requirements, testing of all parameters of each product is not necessarily performed.
TI assumes no liability for applications assistance or customer product design. Customers are responsible for their products and applications using TI components. To minimize the risks associated with customer products and applications, customers should provide adequate design and operating safeguards.
TI does not warrant or represent that any license, either express or implied, is granted under any TI patent right, copyright, mask work right, or other TI intellectual property right relating to any combination, machine, or process in which TI products or services are used. Information published by TI regarding third-party products or services does not constitute a license from TI to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property of the third party, or a license from TI under the patents or other intellectual property of TI.
Reproduction of information in TI data books or data sheets is permissible only if reproduction is without alteration and is accompanied by all associated warranties, conditions, limitations, and notices. Reproduction of this information with alteration is an unfair and deceptive business practice. TI is not responsible or liable for such altered documentation.
Resale of TI products or services with statements different from or beyond the parameters stated by TI for that product or service voids all express and any implied warranties for the associated TI product or service and is an unfair and deceptive business practice. TI is not responsible or liable for any such statements.
Following are URLs where you can obtain information on other Texas Instruments products and application solutions: Products Applications
Amplifiers amplifier.ti.com Audio www.ti.com/audio
Data Converters dataconverter.ti.com Automotive www.ti.com/automotive
DSP dsp.ti.com Broadband www.ti.com/broadband
Interface interface.ti.com Digital Control www.ti.com/digitalcontrol
Logic logic.ti.com Military www.ti.com/military
Power Mgmt power.ti.com Optical Networking www.ti.com/opticalnetwork
Microcontrollers microcontroller.ti.com Security www.ti.com/security
Telephony www.ti.com/telephony
Video & Imaging www.ti.com/video
Wireless www.ti.com/wireless
Mailing Address: Texas Instruments
Post Office Box 655303 Dallas, Texas 75265
Copyright © 2010-12, Texas Instruments Incorporated
iii
Preface
Read This First
About This Manual The scene adaptive local dynamic range enhancement (SAL-DRE) algorithm improves the dynamic range of a scene by adaptive tone mapping such that the contents in both the dark and bright regions are visible. The purpose of SAL-DRE is to modify the brightness/contrast of a video/image in such a way that it looks more similar to how a human would see the actual scene.
Intended Audience
This document is intended for system engineers who want to integrate TI’s algorithms with other software to build a multimedia system based on the DM365 platform.
This document assumes that you are fluent in the C language, have a good working knowledge of Digital Signal Processing (DSP), digital signal processors, and DSP applications.
Read This First
iv
Related Documentation
You can use the following documents to supplement this user guide:
TBD
Text Conventions
The following conventions are used in this document:
Text inside back-quotes (‘‘) represents pseudo-code.
Program source code, function and macro names, parameters, and command line commands are shown in a mono-spaced font.
File names, paths are shown in italic.
Product Support
When contacting TI for support on this codec, please quote the product name (SALDRE algorithm on DM365) and version number. The version number of the codec is included in the Title of the Release Notes that accompanies this codec.
Read This First
v
Trademarks
Code Composer Studio and eXpressDSP are trademarks of Texas Instruments.
All trademarks are the property of their respective owners.
Software Copyright
Software Copyright © 2006 Texas Instruments Inc.
vi
Contents
Read This First ................................................................................................................ iii
About This Manual .....................................................................................................iii Intended Audience .....................................................................................................iii Related Documentation ............................................................................................. iv Text Conventions ...................................................................................................... iv Product Support ........................................................................................................ iv Trademarks ................................................................................................................ v Software Copyright..................................................................................................... v
Contents ..........................................................................................................................vi 1.1 Overview .........................................................................................................0-1 1.2 Data flow .........................................................................................................0-2 1.3 Execution flow .................................................................................................0-3 1.4 APIs 0-8 1.5 External dependencies ..................................................................................0-12 1.6 Quality tuning.................................................................................................0-12
1.6.1 Auto-exposure ................................................................................................. 0-13 1.6.2 Local flickering prevention ............................................................................... 0-13
1.7 Performance considerations ..........................................................................0-13
Revision History
Version Date Comments 1.3 May 5, 2011 • Updated example code p.0-4 and p.0-5 1.2 April 14, 2011 • Updated SALDRE_create() prototype with explanation on
new parameter temporalAverWinSize. • Updated Quality tuning section • Updated Performance section
1.1 Jan 16, 2011 Initial document
vii
This page is intentionally left blank
1
1.1 Overview The scene adaptive local dynamic range enhancement (SAL-DRE) algorithm improves the dynamic range of a scene by adaptive tone mapping such that the contents in both the dark and bright regions are visible. The purpose of SAL-DRE is to modify the brightness/contrast of a video/image in such a way that it looks more similar to how a human would see the actual scene. The dynamic range ratio of the scene in the figure below is 100 and the SAL-DRE algorithm enhances the dark regions while preserving the bright regions.
Examples of processed images through the SAL-DRE algorithm are shown below.
Unprocessed Processed
Contents
0-2
1.2 Data flow
The SALDRE algorithm must operate on raw data obtained by the ISIF. The capture data flow used must be ISIF->DDR->IPIPE depicted in the following figure:
The SALDRE algorithm simply operates between the ISIF and IPIPE altering the data flow to: ISIF->DDR->SALDRE->DDR->IPIPE:
Sensor ISIF
H3A
DDR
IPIPE
DDR DDR
BSC
RSZ2
VSTAB (IMCOP)
2A (ARM)
DDR
NF (IMCOP)
DDR
SW OSD (ARM)
1536x864 (20% more than 1280x720 for VS in H and V direction)
1536x864 YUV420
WxH YUV420
WxH YUV420
AEWB control to Sensor, IPIPE, ISIF
RSZ1
DDR
DDR
LDC
768x432 YUV420
Face Info
(W*1.2) x (H*1.2) YUV420 (From RSZx) WxH = 1280x720, 320x180, 640x360
DDR
RSZ1
DDR
394x216 YUV420
Contents
0-3
The IPNC can enable an alternate data flow ISIF->IPIPE which is to pass the output of the ISIF directly to the IPIPE. This data flow is the default one and consumes less DDR bandwidth. However it cannot be used in conjunction with SALDRE since there is no access to the raw data (the IPIPE outputs directly YUV data).
The latest release IPNC 2.5 automatically switches to the ISIF->DDR->IPIPE data flow whenever the user enables SALDRE through the GUI. If SALDRE was previously disabled, the system restarts the AVServer in order to switch from the default ISIF->IPIPE data flow.
To integrate SALDRE into a prior release, such as 2.0, one must edit the file av_capture\application\ipnc\av_server\src\common\avServerUi.c and modifies the function UI_setConfig() such that config->captureRawInMode is initialized to AVSERVER_CAPTURE_RAW_IN_MODE_DDR_IN whenever SALDRE is enabled.
1.3 Execution flow
The SALDRE algorithm is implemented by the library av_capture\framework\alg\lib\alg_saldre_ti.a .
The library contains the following APIs:
1. SALDRE_create()
Sensor ISIF
H3A
DDR
IPIPE
DDR DDR
BSC
RSZ2
VSTAB (IMCOP)
2A (ARM)
DDR
NF (IMCOP)
DDR
SW OSD (ARM)
1536x864 (20% more than 1280x720 for VS in H and V direction)
1536x864 YUV420
WxH YUV420
WxH YUV420
AEWB control to Sensor, IPIPE, ISIF
RSZ1
DDR
DDR
LDC
768x432 YUV420
Face Info
(W*1.2) x (H*1.2) YUV420 (From RSZx) WxH = 1280x720, 320x180, 640x360
DDR RSZ1
DDR
394x216 YUV420
SALDRE
DDR
Contents
0-4
2. SALDRE_func0()
3. SALDRE_func1()
4. SALDRE_setupFunc2()
5. SALDRE_startFunc2()
6. SALDRE_waitFunc2()
7. SALDRE_endFunc2()
As it can be noted, there is a creation function called SALDRE_create(), which just needs to be called once during system initialization and three processing functions func0, func1 and func2. Out of these three functions, only func1 executes on the ARM, the other two use the DM368 SOC’s hardware accelerators. Note also that func1 processes the output of func0. To take advantage of parallel processing between ARM and the accelerators, func2 can run in parallel with func1 but this involves a more complicated processing flow with ping-pong buffering on the buffers. A simple sequential schedule can first be implemented using the following pseudo code:
byte *srcAddr; Byte dstBuffer[2*WITH*HEIGHT]; SALDRE_create(); while(isThereNewFrameFromCaptureDriver== TRUE) { srcAddr= getFrameAddrFromCaptureDriver(); SALDRE_func0(srcAddr, func0output, …); SALDRE_func1(func0output, 0, …); SALDRE_setupFunc2(srcAddr, dstBuffer, …); SALDRE_startFunc2(); SALDRE_waitFunc2(); SALDRE_end_func2(); IPIPE_RSZ(dstBuffer); }
As this pseudo code shows, all the processing is sequential: the frame obtained from the getFrameAddrFromCaptureDriver() function gets passed along to the other functions in a sequential manner. The function IPIPE_RSZ() processes the raw bayer frame pointed by dstBuffer, resize it, enhances it and converts it into YUV format. It runs one the IPIPE accelerator, which is different than the accelerator used by SALDRE. So another level of parallelism can be added.
Now to take advantage of parallelism between IPIPE, SALDRE accelerator and ARM, a different schedule needs to be implemented, which involves ping-pong buffering for both input and output buffers and pipeline fill-up. One thing to note is that func0 and func1 must be ran sequentially to each other since func1 processes the output of func0. Here is such a pseudo code reflecting new concurrent schedule:
Byte *srcAddr[2];
Byte *dstBuffer[2];
Byte *func0output[2];
Byte *weightBuffer;
Contents
0-5
/* Buffers must be allocated using CMEM so they are continuous */
outputBufSize= SALDRE_getOutputBufferSize(WIDTH, HEIGHT, WIDTH);
dstBuffer[0]= OSA_cmemAlloc(outputBufSize, 32);
dstBuffer[1]= OSA_cmemAlloc(outputBufSize, 32);
func0output= OSA_cmemAlloc((2*WIDTH*HEIGHT)/256, 32);
weightBuffer= OSA_cmemAlloc(2*WIDTH*HEIGHT, 32);
SALDRE_create(WIDTH, HEIGHT, temporalAverWinSize);
saldreInBufferIndex= 0;
saldreOutBufferIndex= 0;
srcAddr[0]= getFrameAddrFromCaptureDriver();
SALDRE_runFunc0(srcAddr[0], func0output, WIDTH, HEIGHT, STRIDE);
SALDRE_runFunc1(func0output, 0, WIDTH, HEIGHT, 0, 128, 64, 6144, 4096);
saldreInBufferIndex= 1;
frame_no= 1;
while(isThereNewFrameFromCaptureDriver== TRUE) {
if (frame_no != 1) IPIPE_RSZ_thread_start(dstBuffer[saldreOutBufferIndex^1]);
srcAddr[saldreInBufferIndex]= getFrameAddrFromCaptureDriver();
SALDRE_runFunc0(srcAddr[saldreInBufferIndex], func0output WIDTH, HEIGHT, STRIDE);
SALDRE_setupFunc2(srcAddr[saldreInBufferIndex^1], dstBuffer[saldreOutBufferIndex], weightBuffer, NULL, WIDTH, HEIGHT, STRIDE, STRIDE);
SALDRE_startFunc2();
SALDRE_func1(func0output, saldreInBufferIndex, WIDTH, HEIGHT, 0, 128, 64, 6144, 4096);
SALDRE_waitFunc2();
SALDRE_end_func2();
saldreInBufferIndex ^=1;
Contents
0-6
saldreOutBufferIndex^= 1;
if (frame_no != 1)
IPIPE_RSZ_thread_wait();
frame_no++;
}
SALDRE_setupFunc2(srcAddr[saldreInBufferIndex^1], dstBuffer[saldreOutBufferIndex], weightBuffer, NULL, WIDTH, HEIGHT, STRIDE, STRIDE);
SALDRE_startFunc2();
SALDRE_waitFunc2();
SALDRE_end_func2();
IPIPE_RSZ_thread_start(dstBuffer[saldreOutBufferIndex]);
IPIPE_RSZ_thread_wait();
srcAddr and dstBuffer are now two entries arrays to implement ping-pong buffering of input and output buffers.
We first fill the pipeline by grabbing the very first frame. This frame is processed by SALDRE_func0 and then SALDRE_func1 . The input buffer flag saldreInBufferIndex is then flipped from 0 to 1 before going to the while() loop which is the steady state processing in which func1 is ran parallel to func2.
In that loop, we first run the IPIPE_RSZ_thread_start() on a frame previously processed by func2. If it is the first iteration, this step must be skipped since the very first frame grabbed hasn’t had the opportunity to be processed yet. Note this IPIPE_RSZ_thread_start() is slightly different than IPIPE_RSZ() used in the sequential schedule because it must switch to a different thread that will run IPIPE_RSZ(). This is to make the call non-blocking. We assume we run on Linux environment so at this point, two threads will run in parallel: the IPIPE_RSZ thread which mainly calls the IPIPE accelerator and the SALDRE thread.
Next in the SALDRE thread, a new frame is grabbed and processed by func0() and func1(). Note that the execution of func1() happens during the start() and wait() calls of func2, effectively achieving parallelism between ARM and SALDRE’s accelerator. Lastly, we wait for completion of the IPIPE_RSZ_thread() at the end of the innermost loop. Basically we need to ensure that all the accelerators have finished their processing before going to the next loop iteration.
A variation of this concurrent schedule is implemented in the IPNC in files av_capture/application/ipnc/av_server/src/video/videoCaptureThr.c and av_capture/application/ipnc/av_server/src/video/videoSaldreThr.c .
The difference between the actual IPNC implementation and above pseudo-code is that the processing flow on IPNC system is split into two threads:
- A SALDRE thread implemented in videoSaldreThr.c. Its implementation is very close to the pseudo code, except that it doesn’t grab any frames and call IPIPE_RSZ_thread(). These two actions are carried out by the video capture thread.
Contents
0-7
- A video capture thread implemented in videoCaptureThr.c: this thread is responsible of grabbing a new frame, call IPIPE_RSZ function and call the SALDRE thread. Note that IPIPE_RSZ doesn’t need a separate thread as it executes inside the video capture thread. Parallelism between SALDRE and IPIPE_RSZ is achieved because SALDRE runs in a different thread than the video capture thread, which contains the IPIPE_RZS function.
Please refer to function VIDEO_captureTskRunDdrIn(). In it, look at the following code section:
if (gAVSERVER_config.aewb_config.saldreEnable == FALSE){
VIDEO_captureRszFunc(pRawBufInfo->virtAddr, pOutBufInfo, rszBufId, numStreams);
}
else {
saldreOutputBufAddr= VIDEO_saldreGetOutputBufAddr();
OSA_semSignal(&g_saldreSemStart);
VIDEO_captureRszFunc(saldreOutputBufAddr, pOutBufInfo, rszBufId, numStreams);
OSA_semWait(&g_saldreSemDone, OSA_TIMEOUT_FOREVER);
}
On the case SALDRE is enabled, the video capture thread gets the output address of the last frame processed by SALDRE and then it triggers the SALDRE thread by signaling a semaphore. From this point of time, the SALDRE thread executes in parallel with the video capture thread. The video capture thread goes on calling the IPIPE_RSZ function, which is VIDEO_captureRszFunc(). A semaphore wait must be called to synchronize all the processing.
Note that waiting for IPIPE completion is implicitly done inside VIDEO_captureRszFunc().
Frames are continuously processed because the function VIDEO_captureTskRunDdrIn() is called inside a while(1) loop in function VIDEO_captureTaskMain(). Additionally, the system maintains a pool of input buffers in order to prevent frame drop in case . The number of input buffers is set by the macro symbol VIDEO_NUM_BUF in file video.h . For optimum performance, please set it 4 otherwise frame rate won’t be the best the system can achieve. Default value of 3 is to allow higher resolution such as 2 MegaPixels, 3 MegaPixels without encountering any memory allocation error. If your target resolution is 720P or 1080P, you can set it to 4.
1.4 APIs
Following are descriptions of the SALDRE APIs.
║ Name
SALDRE_create() – Create the algorithm ║ Synopsis
Contents
0-8
Int32 SALDRE_create(Uint16 maxWidth, Uint16 maxHeight, Uint16 temporalAverWinSize) ║ Arguments
Uint16 width /*maximum width in number of pixels of any Bayer frame that will be processed. */ Uint16 height /*maximum height in number of rows of any Bayer frame that will be processed */ Uint16 temporalAverWinSize /* size of the temporal averaging window in number of frames */
║ Return Value 0: success -1: error
║ Description
Initialize the SALDRE algorithm. Must be called before any other SALDRE functions. To prevent local flickering effects due to sensor noise, the SALDRE algorithm performs some temporal averaging. The parameter temporalAverWinSize sets the number of frames that are averaged over time. The greater this number is, the more averaging is done and the more stable the image is. But the scene doesn't adapt so quickly to sudden illumination changes or spatially large content changes. So if the camera is pointing to a mostly static scene with small motion areas, then a large value such as 16 can be assigned to temporalAverWinSize. Otherwise use smaller values such as 2, which is the default one set in the IPNC codebase. Refer to file videoSaldreThr.c in which SALDRE_create() is called by VIDEO_saldreCreate().
║ Name
SALDRE_getOutputBufferSize
║ Synopsis Int32 SALDRE_getOutputBufferSize(Uint16 roiWidth, Uint16 roiHeight, Uint16 outputStride) ║ Arguments
Uint16 roiWidth /* ROI width of the frame in pixel */ Uint16 roiWeight /* ROI height of the frame in pixel */ Uint16 dstStride /* stride of the output buffer in pixels, usually equal to roiWidth */
║ Return Value sizeOutput in bytes of frame buffer for holding the output
║ Description
This function returns the minimum number of bytes that need to be allocated to hold one frame output by SALDRE_func2(). The input parameters correspond to the region interest's pixel width and pixel height of the frame to process as well as the stride of the output frame. Most of the time the returned size value is equal to 2*dstStride*roiHeight but for some case such as 720P resolution, it is a larger value because the actual output height may be greater than roiHeight.
Later, when calling SALDRE_setupFunc2(), the application must make sure that the arguments imgHeight and dstStride for this function satisfy the expression: 2*imgHeight*dstStride= sizeOutput .
║ Name
SALDRE_runFunc0
║ Synopsis
Contents
0-9
Int32 SALDRE_runFunc0(Uint8* srcAddr, Uint8* dstAddr, Uint16 imgWidth, Uint16 imgHeight, Uint16 srcStride) ║ Arguments
Uint8* srcAddr /* pointer to the raw Bayer frame to process of size 2* srcStride *imgHeight bytes */ Uint8* dstAddr /* pointer to buffer of size 2*imgWidth*Height/256 bytes */ Uint16 imgWidth /* width in number of pixels of the Bayer frame to process */ Uint16 imgHeight /* height in number of rows of the Bayer frame to process */ Uint16 srcStride /* stride in number of pixels of the Bayer frame to process */
║ Return Value 0: success -1: error
║ Description
This function collects some statistics about Bayer frame pointed by srcAddr and write them into the buffer pointed by dstAddr of size 2*imgWidth*Height/256 bytes.
║ Name
SALDRE_runFunc1
║ Synopsis Int32 SALDRE_runFunc1(Uint8 *dstAddrFromFunc0, Uint16 index, Uint16 imgWidth, Uint16 imgHeight, Uint32 config, Int16 GBEStrength, Int16 GCEStrength, Int16 LBEStrength, Int16 LCEStrength) ║ Arguments
Uint8* dstAddrFromFunc0/* pointer to the statistics buffer generated by SALDRE_runFunc0 */ Uint16 index /* if sequential processing implemented, always 0, if concurrent processing implemented, alternate between 0 or 1 */ Uint16 imgWidth /* width in number of pixels of the Bayer frame to process */ Uint16 imgHeight /* height in number of rows of the Bayer frame to process */ Uint32 config /* Ignored for now, set to 0 */
Int16 GBEStrength /*global brightness strength, can be 64, 128, 256 */ Int16 GCEStrength /*global contrast strength, can be 64, 128, 256 */ Int16 LBEStrength /*local brightness strength, can be 4096, 6144, 8192 */ Int16 LCEStrength /*local contrast strength, should be set to 4096 */
║ Return Value 0: success -1: error
║ Description
This function generates tone curves from the statistics generated by SALDRE_runFunc0
Contents
0-10
║ Name
SALDRE_setupFunc2
║ Synopsis ║ Int32 SALDRE_setupFunc2(Uint8 *srcAddr, Uint8*dstAddr, Uint8* weight, Uint8 *toneCurves, Uint16 imgWidth, Uint16 imgHeight, Uint16 strideSrc, Uint16 strideDst) ║ Arguments
Uint8 *srcAddr /* pointer to the raw Bayer frame to process of size 2*strideSrc *imgHeight bytes */ Uint8 *dstAddr /* pointer to the destination buffer of size 2*strideDst *imgHeight bytes */ Uint8 *weight /* pointer to buffer of size 2*imgWidth*Height */ Uint8 *toneCurves /* set to NULL to use tone curves generated by SALDRE_runFunc1()*/ Uint16 imgWidth /* width in number of pixels of the Bayer frame to process */ Uint16 imgHeight /* height in number of rows of the Bayer frame to process */ Uint16 srcStride /* stride in number of pixels of the Bayer frame to process */ Uint16 dstStride /* stride in number of pixels of the destination buffer */
║ Return Value 0: success -1: error
║ Description
Setup the function that will enhance the dynamic range of the source frame, using the tone curves generated by SALDRE_runFunc1() if toneCurves set to NULL. If toneCurves is different than NULL, it must point to valid toneCurves table. Contact TI for details on format of this table.
The application must make sure that the arguments imgHeight and dstStride for this function satisfy the expression: 2*imgHeight*dstStride= sizeOutput where sizeOutput was returned by SALDRE_getOutputBufferSize().
║ Name
SALDRE_startFunc2
║ Synopsis void SALDRE_startFunc2() ║ Arguments ║ Return Value
0: success -1: error
║ Description
Start execution of the function setup by SALDRE_setupFunc2(). Upon return of the function, the SALDRE processing has started in the background.
Contents
0-11
║ Name
SALDRE_waitFunc2
║ Synopsis void SALDRE_waitFunc2() ║ Arguments ║ Return Value
0: success -1: error
║ Description
Wait for the completion of the function started by SALDRE_startFunc2(). Upon return of the function, the SALDRE processing has completed.
║ Name
SALDRE_endFunc2
║ Synopsis void SALDRE_endFunc2() ║ Arguments ║ Return Value
0: success -1: error
║ Description
De-initialize the SALDRE function setup by SALDRE_setupFunc2() . Must be called at the end of each frame processing, before processing next frame.
║ Name
SALDRE_resetFunc2
║ Synopsis void SALDRE_resetFunc2() ║ Arguments ║ Return Value
0: success -1: error
║ Description
This function needs to be called after every time the AV_server is restarted due to change of resolution, codec, enabling/disabling SALDRE.
1.5 External dependencies
The SALDRE library calls the following functions defined in av_capture\framework\alg\src\alg_aewb_control.c:
ALG_aewbGetAEValues()
Contents
0-12
ALG_aewbGetAWBGains() ALG_aewbGetRgb2Rgb()
These functions access some image statistics maintained by the TI 2A or Appro 2A driver. If the system uses a custom driver, these functions will have to be re-implemented.
1.6 Quality tuning
The SALDRE algorithm doesn't require much tuning as long as the functions ALG_aewbGetAEValues(), ALG_aewbGetAWBGains() and ALG_aewbGetRgb2Rgb() are correctly implemented. Some tuning concerning auto-exposure and local flickering prevention may be done per the following guidance.
1.6.1 Auto-exposure
For TI-2A, in case bright regions become saturated after SALDRE is applied, lower the image pipe gain by changing in
av_capture\framework\alg\src\alg_aewb.c
the aeDynamicParams.ipipeGainRange[i].max value to some value lower than 4092.
1.6.2 Local flickering prevention
To prevent local flickering effects due to sensor noise, the SALDRE algorithm performs some temporal averaging. The parameter temporalAverWinSize of SALDRE_create() sets the number of frames that are averaged over time. The greater this number is, the more averaging is done and the more stable the image is. But the scene doesn't adapt so quickly to sudden illumination changes or spatially large content changes. So if the camera is pointing to a mostly static scene with small motion areas, then a large value such as 16 can be assigned to temporalAverWinSize. Otherwise use smaller values such as 2, which is the default one set in the IPNC codebase. Refer to file videoSaldreThr.c in which SALDRE_create() is called by VIDEO_saldreCreate().
1.7 Performance considerations
The following lists timing information for each function in micro-second for 720P resolution:
func0 time_frm = 6500
func1 time_frm = 14627
func2 setup time_frm = 5546
func2 time_frm = 25000
Total frame processing time (func0 + setup func2 + max(func1, func2)= 37046
As you see the total frame processing time exceeds 33 ms, which indicates that the algorithm cannot run at 30 fps. However it is possible to improve its speed by executing SALDRE_func0(), SALDRE_func1() every other N frames while SALDRE_setupFunc2(), SALDRE_startFunc2(), SALDRE_waitFunc2() and SALDRE_endFunc2() must be executed every frame. This way we can achieve > 30 fps.
To see how such a technique is implemented please refer to the implementation of function VIDEO_saldreCreate() and VIDEO_saldreRun() in file videoSaldreThr.c, which can be
Contents
0-13
summarized as follow: from videoCaptureThr.c, VIDEO_saldreCreate(Uint16 frameSkip) is called with argument frameSkip set to the number of frames to skip between consecutive calls of SALDRE_func0() and SALDRE_func1(). Then VIDEO_saldreRun() updates the internal counter numframes which triggers the execution of SALDRE_func0() and SALDRE_func1() whenever the condition !(numframes % frameSkip)) becomes true.
When value frameSkip=0 is passed to VIDEO_saldreCreate(), there isn't enough processing power to run SALDRE at 30 fps and ARM load shoots up to 90%. But when the value frameSkip=1 is passed to VIDEO_saldreCreate(), frame rate becomes 30 fps and ARM load is approximately 58 %. To reduce ARM load further, you can increase frameSkip at the expense of some lag when big illumination changes occur in the scene. If the scene is mostly static, this may not be an issue. No adverse effects have been observed when frameSkip=1 . But if you increase frameSkip to 2, 3, etc, then you will want to decrease temporalAverWinSize parameter, which is passed to SALDRE_create() in file videoSaldtreThr.c .
In addition, make sure that the number of input buffers is set by the macro symbol VIDEO_NUM_BUF in file video.h . For optimum performance, please set it to 4 otherwise frame rate won’t be the best the system can achieve. Default value of 3 is to allow higher resolution such as 2 MegaPixels, 3 MegaPixels without encountering any memory allocation error. If your target resolution is 720P or 1080P, you can set it 4.