website ： http ： // Image Sensor Pipeline CCD & CMOS Module.

website ： http ： // www.smit.com.cn

Image Sensor Pipeline

CCD & CMOS Module


CCD ISP

CMOS ISP

VD

CCD module Video Decoder

CMOS SOC


CMOSCCD

Color SpaceColor transformResize色彩调整测试设备与人员国家图像质量标准新技术点

CCDInterlaceDeinterlaceDenosie色彩增强

CMOSLens(very important,but no mention)Bad pixelInterpolationGammaBacklightingColor Correction3AvignettingEdge Extraction and SharpnessDetection and CompensationAnti FlickerTemporal Noise ReductiondistortionsDeblur(no mention)


Outline• Sensor Performance • Color Space • Color Transform• Interlace & deinterlace• Denoise&Deblur• Scale & Crop• New scheme


典型的安防市场用 CCD 模组

典型的安防市场用 CMOS 模组


Sensor Performance Summary• Sensor Pixel Size• Read Noise• AD Converter• Signal-to-Noise Ratio• Full Well Capacity• Dynamic Range• Unity Gain• Low Light Sensitivity Factor• Noise Sources• Pixel Density


Sensor Size

Typical sensor size of 3, 4, and 5 megapixel digital compact cameras

Typical sensor size of 6 megapixel digital SLRs

背景是 35mm 标准胶卷


Sensor Pixel Size

Full Well Capacity


Color Array

The New Foveon Sensors

CFA Bayer


Super CMOS SensorNormally sensor pixels are square. Fujifilm's Super CCD sensors have octagonal pixels, as shown in this diagram. Therefore, the distance "d2" between the centers of two octagonal pixels is smaller than the distance "d1" between twoconventional square pixels, resulting in larger (better) pixels.


However, the information has to be converted to a digital image with square pixels. From the diagram you can see that, for a 4 x 4 area of 16 square pixels, only 8 octagonal pixel measurements were used: 2 red pixels, 2 blue pixels, and 4 greenpixels (1 full, 4 half, and 4 quarter green pixels). In other words, 6 megapixel Super CCD images are based on the measurement by only 3 million effective pixels,similar to the above interpolated example, but with the advantage of larger pixels. In practice the resulting image quality is equivalent to about 4 megapixel. The drawback is that you have to deal with double the file size (leading to more storage and slower processing), while enjoying a quality improvement equivalent to only 33% more pixels.

简单的说，就是同样感光面积的的 Sensor ，能提供约多 44% 的有效像素，但是它需要多处理 1 倍大小的图像。得不偿失？？？？？？？？？


Sensitivity (ISO)

ISO 100 - Red Channel ISO 800ISO 100 ISO 800 - Red Channel

Increase sensitivity-amplified -Noise increase


Sensor Linearity

Sensors respond in a linear way (*) to light while human vision responds in a non-linear way which is often approximated by a power curve of about 0.45. So what the sensor measures as "127" is perceived by the human vision as about "186".


Sensor Linearity

这就是需要做 gamma 校正的原因，当然实际的校正曲线往往是 S 型的曲线这是因为为显示设备压缩动态范围的需要。


AD Converter

Sensor Dynamic Range ADC Type Image Tonal Range

Low (e.g. around 256:1)8 bit 8 bit

10 or 12 bit 8+ bit (*)

High (e.g. around 4,000:1)

8 bit 8 bit

10 or 12 bit 10 or 12 bit if RAW


Dynamic RangeWell Capacity (electrons) / Read Noise (electrons),

理想 Dynamic Range = Full Sensor ，在完全黑暗中，读出应该为 0.根据 sensor 的噪声模型，最小信号为 1 时，噪声为 1.假设动态范围为 1000 ，那么需要用 2^12bits 的 AD 表示这颗 sensor 。另外： sensor 的放大器，和 AD 会带来噪声，所以在不同的 ISO 值会有不同的噪声叠加值。所以，不同的 ISO 情况下，有不同的动态范围。

所以 read noise, full well capacity, and dynamic range. 三者关系复杂

Photons

Noise

Signal-to-noiseRatio

9 3 3

100 10 10

900 30 30

10000 100 100

40000 200 200

90000 300 300

Table 1a

Poisson Statistics


High Dynamic Range影响动态范围的参数：1.噪声大小（越小越好）2.数据位宽（越宽越好）3.Pixel 物理面积大小（越大越好）

说明： Pixel 物理尺寸越大， HDR 性能越好AD 位宽越高，性能越好

说明：在高 ISO 时， HDR 性能急速下降，因为 ISO 增加， AD 增益加大，噪声增加。


The More Pixels, the Worse the Image!

"As an independent test lab for digital cameras, we have observed a decrease in the image quality of digital cameras during the last 3 years,“ saysImage Engineering.www.image-engineering.de

Newer cameras have an increasingly worse image quality and the reason for that is obvious:"The more pixels a camera has, the better it is” was true in the beginning of digital photography when compact cameras had a VGA resolution (640 x 480)

When the pixel count exceeded 6 Megapixels in 2004, this was no longer the case but consumers still buy cameras with the highest pixel count.


Color Space

•YUV

•YIQ

•YCbCr

•HSV

•HSB

•RGB

•Lab

•CMYK

luma/chroma(TV) Hue/Saturation Mixtrue


BT656,PAL/SECAMATV

TV Color Space

A B

To Be Created

DBT601SDTV,BT1120

BT709HDTVDTV

C

YUVYIQ

YCbCr

NTSCATV


YUV

Y' value of 0

Y' value of 0.5

Y' value of 1

Note :From http://en.wikipedia.org/wiki/YCbCr#ITU-R_BT.601_conversion

YUV YIQ


YCbCr

Y' value of 0

Y' value of 0.5

Y' value of 1

Note :From http://en.wikipedia.org/wiki/YCbCr#ITU-R_BT.601_conversion


RGB/CMY(K)RGB, 表示范围大，适用主动发光物体色彩空间，如显示器。

CMKY ，表示范围小，使用与被动显示物体，如打印相片


RGB/HSB(HSV)RGB, 非常适合产生 R,G,B 分量色彩。

RGB, 非常适合调整饱和度，亮度，色度。


Color TransformColor Transform ：基本上任何一款视频处理芯片都需要。YUV-RGB ，注意处理精度问题HSB-RGB ，算法复杂度，色彩损失


YUV Transform

YUV 的色彩排列很重要，每颗芯片可能有自己定义的数据输入格式，这些格式在FOURCC 中定义。 YUV 转换涉及到 DownSampling 和Upsampling 技术

YUYV

YVYU

YV12Y655

Y41P

GREY

http://www.codecs.onerivermedia.com


RGB Transform

RGB 的数据格式定义：主要是某些数码相机中，输出的RGB 格式排列有不同。

RGBA

RAW

BI_RGBRGB

BI_RLE8

RLE

http://www.codecs.onerivermedia.com


Q&A about Color Space &

Transform


Interlace & Deinterlace

• What is interlace?Is interlace a bug? Why interlace ?• How to Deinterlace?• Telecine


AD. :Interlace! Since 1930s!


From ： http://en.wikipedia.org/wiki/Deinterlacing

Resize

Deinterlace

This process of dividing frames into half-resolution fields at double the frame rate is known as interlacing.


（ 1 ） Clever processing for TV system Save half bandwidth.(Data compress)

（ 2 ） Most modern displays, such as LCD, DLP and plasma displays, are not able to work in interlaced mode, because they are fixed-resoution displays and only support progressive scanning. (Bad luck to ATV)

（ 3 ） In order to display interlaced signal on such displays, the two interlaced fields must be converted to one progressive frame with a process known as de-interlacing.

Why interlace


How to deinterlace

Deinterlace

3D deinterlace• Memory cousume

复杂，效果最好复杂，效果最好

Blend(to blur)• Area Base

• To blur the mice teeth• Discard

Motion CompensationMotion detection

Weave(do nothing)Field combination

deinterlacing

BOBDisplay every fields by

50FPSField extension deinterlacing

• Lose nothing

Note:None of them is perfect!!!!!!!!!!!(The main argument is that no matter how complex the deinterlacing algorithm may be, the artifacts in the interlaced signal cannot be completely eliminated because some information is lost between frames.)


When is deinterlace bad?

（ 2 ） Film to TV(just de telecine)

（ 1 ） when video source is not interlaced


telecine

FilmTV (telecine)

Interlaced video can carry progressive scan signal, and deinterlacing process should consider this as well. (More unfortunately)

24Fps25Fps on PAL TV system 24Fps59.94Fps on NTSC SECAM

system


2:2 2:3


Does deinterlacing work?

• Interlace will be disappeared in future!

• The European Broadcasting Union has argued against the use of interlaced video in production and broadcasting


Q&A about Deinterlace


Scale & Crop

How to Scale?,( 等比，非等比 )How to Crop?Rotation? (no need)


ResizeResizeScale

Crop


原图

近邻取样缩放到 0.6 倍近邻取样缩放到 1.6 倍

二次线性插值缩放到 0.6 倍二次线性插值缩放到 1.6 倍

三次卷积插值缩放到 0.6 倍三次卷积插值缩放到 1.6 倍

Scale


原图

近邻取样缩放到 8 倍二次线性插值缩放到 8 倍

三次卷积插值缩放到 8 倍二次线性插值 ( 近似公式 )

Scale


Crop


Denoise & Deblur


• 背景噪声，白噪声• TBD


Video Enhancement


• Luminance peaking ，补偿信道过程中高频损失，主要在 YUV 域中处理 Y 分量

• 数字亮度瞬态增强技术 (DLTl ） (sharpness)• 黑电平扩展，只处理 Y• 白电平扩展，只处理 Y• 对比度，亮度，处理 Y• 色度，饱和度调整 , 之处理 UV• 肤色校正


参考 PW328


肤色校正原理，肤色在 YUV 空间中分布范围：

正负 20 度之间

调整方法：


YCbcr444

Luminance Peaking

YCbcr444 YCbcr444

DLT1

BLE/WLE

YCbcr444

BLE/WLE

BLUE leGamma

YCbcr444->420/422


CMOS Sensor Image Pre_processing

ISP


New Scheme


• ROI• SVC/MVC/3D• Panorama Image(1.3M=4D1)• HDR


谢谢！

website ： http ： // Image Sensor Pipeline CCD & CMOS Module.

Documents

Transcript of website ： http ： // Image Sensor Pipeline CCD & CMOS Module.