Dealing with the Complexities of Camera ISP Tuning...Dealing with the Complexities of Camera ISP...

Dealing with the Complexities of Camera ISP Tuning

Clément Viard, Sr Director, R&DFrédéric Guichard, CTO, [email protected]

1© DxO Labs 2016 | PREPARED FOR: AutoSens conference

Dealing with the Complexities of Camera ISP Tuning

2

> Basic camera image processing

> First revolution: optical and sensor defects corrections

> Second revolution: miniaturization

> ISP, A key differentiator in image and video processing

© DxO Labs 2016 | PREPARED FOR: AutoSens conference

3

Introduction

« Basic » digital image processing in a camera


RAW data from Sensor


After « demosaicing »


After exposure adaptation


After « white ballance »


After color rendering


After sharpening


10

A digital revolution in cameras

Optical and sensor defect corrections


Geometric distortion


Volume anamorphos – perspective error with wide angle lenses


Optical vignetting


Chromatic aberation


Tone mapping


Tone mapping (with HDR sensor)


Tone mapping (with HDR sensor)

21© DxO Labs 2016 | PREPARED FOR: RENAULT

Clipping – Recovering saturated area


Roll

Yaw

Pitch

X

Y

Z

Stabilisation and Electronic Rolling Shutter (ERS) effect

Translation Z

Yaw (Rotation)

Translation X

Pitch (Rotation)

Translation Y

ERS horizontal

ERS vertical


Range of Digital Corrections with Advanced ISPs

25

> Optical aberrations> Distortion> Blur (spherical aberration, field

curvature, coma, astigmatism, motion)> Chromatic aberrations> Vignetting> Flare, veiling glare

> Light & Sensor> Noise> Dynamic range> Contrast> Atmospheric haze

> Sensor limitations> Field non uniformity (color, black

level,…)> Defective pixels, Dust> Clipping> Metamerism

> Others> Video stabilization> Electronic rolling shutter


26

Miniaturization – the second camera revolution

Will they eventually reach same performance?


Sensor Miniaturization Challenge

27

22 times less light 4.5 stops


Performance improvement over 10 years

28

> CMOS Sensor improvement : +1.5 stops

> Digital processing gain: +3 to +4 stops

DxOMark sensor score, APS-C cameras

+1.5 stops


Nikon D70s, ISO 3200 – jpg


Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v3


31



32



33



34

Nikon D70s, ISO 3200 – jpg


Sensor color response changes with CRA

Photosite

Color response as a function of the angles350 450 550 650 750


36

ISP

A key differentiator in image and video processing


« ISP » as a key differentiator

37

> Enables camera hardware and scene artefact corrections

> Only remaining differentiator since access to best sensor and lens is now ubiquitous

> Paradigm shift, cameras are designed considering possible digital corrections

> Computing power requirement consistently increasesDelivering 2,000 Ops/pixel with 240 Mpix/s (24 frames at 10Mpix per second) requires 500 GOps/s

> Very significant investment within the Mobile industry(e.g. iPhone camera engineering team ~ 800 people)


Same HW, same ISP, same BOM same image quality?

DxO Mark Mobile score (Photos and Videos)38© DxO Labs 2016 | PREPARED FOR: AutoSens conference

« Camera Tuning »

39

For correcting each image, ISP requires 10,000+ register settings to be adapted to the situation.

> Per serie Calibration

> Per unit Calibration

> On the fly parameter estimations


Image Quality Evaluation Challenge

> What is “Image Quality”?> Perception of how a picture “looks good”> In essence a subjective matter…that can be modeled with engineering tools

> What influences “Image Quality”> Shooting conditions: illumination, illuminant, dynamic> Content has a strong influence on Image Quality perception and image processing> Image Quality perception is different between imaging expert, professional

photographers and consumer


Image Quality Evaluation Challenge - Subjective vs. objective

> Objective evaluation> Objective means a measurement that is neutral, operator independent: “2

cm, 15 meters, 150 kg, 2 g, -5°C, 50°C, …”> A device must provide figures (metrics) that are related to image quality> Normalizations may be necessary to have comparable metrics (cameras

with different resolutions)> Only addresses a set of metrics (some artifacts may be ignored)

> Subjective evaluation> “Subjective” means real people give an opinion like: “big, small, heavy, light,

cold, hot, …”> People can judge the quality of photographs> Methodology is key to get non-biased results


Camera IQ assessments

42

Temporal behaviorDynamic behavior Photogrammetry and 3D

Thousands of videos/photos are required to characterize IQ along differentdimensions

Specifying AND verifying image quality targets are tough challenges


Just Noticeable Difference (JND), a Matter of Statistics

> The smallest statistically measurable difference of perception, e.g., smallest perceivable distance between 2 parallel lines

> Typically, defined when half of the people perceive a difference and the other half are guessing (50% JND)

50% perceive

a change 50% guessing


Anchored Scaling Concept

C D E F

Test image

??

Anchors


Objective/subjective correlation example: Sharpness JND mapping

After extensive visual experiments, one could show that the Acutance objective metric linearly correlates with perception of sharpness (expressed with JND unit)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

5

10

15

20

25

30

35

Acutance

JN

D

0

20

40

60

80

100

120

0.0 0.1 0.2 0.3 0.4 0.5

MTF

(%

)

Frequency (cycle/pixel)

CSF

MTF


Back to the real world!

> Unfortunately the assumptions to build a complete multi-variate analysis based scoring system are not met > Only few metrics are linearly correlated to perception> They are not strictly orthogonal to each others> New issues comes with every technological improvement

> Since we can’t build a unified scoring system based on a set of perceptual metrics, a Multi-Modal approach is required


Recommendation for building a relevant Image or video score

47

> Multi-modal testing approach> Objective measurements in lab environment> Perceptually-correlated metrics when available> Perceptual analysis from natural scenes

> Multi-level testing results> Overall, Photo, and Video scores> Top-level scores (open scale) non technical> Detailed reports for engineering


Example of natural scene set used for mobile camera application


Conclusion

49

> Eventually, all industries would benefit from Smartphone technology breakthroughs

> ISP can fix many optical and sensor defects

> ISP technology improvements are a key enabler for miniaturization

> Mastering tuning complexity and IQ evaluation is a key differentiator

> DxO has developed robust methods to deal with this problem


1 / DxO USA 1407 Minnesota Street San Francisco, CA [email protected]: +1 415 805 8567

2 / DxO France3, rue Nationale 92100 Boulogne-Billancourt France [email protected] Tel: +33 1 55 20 55 99

3 / DxO Korea16th Floor, Gangnam Building 1321-1 Seocho-dong, Seocho gu Seoul [email protected]: +82 2 2190 3825

4 / DxO TaiwanTai Centre 2/F-A170 Dunhua North Road Taipei 10548Taiwan Tel: +886 2 5574 0429

5 / DxO Japan3F Metlife Kabutocho Bldg 5-1 Kabutocho Chuo-ku Nihonbashi, [email protected]: +81 (0)3 6861 7565

Thank you!

50

[email protected]


Dealing with the Complexities of Camera ISP Tuning...Dealing with the Complexities of Camera ISP...

Documents

Transcript of Dealing with the Complexities of Camera ISP Tuning...Dealing with the Complexities of Camera ISP...