Image Processing System Applications: from Barbie Cams to the

Image Processing System Applications:from Barbie Cams to the Space Telescope

Robert Kremens, Ph.DRochester Institute for Technology

Center for Imaging ScienceDigital Imaging and Remote Sensing Group

andPixelphysics, Inc.

May 2001

Outline

• Fundamentals of Image Processing for Digital Cameras

• Solid State Image Sensors - CCDs, CMOS, etc.

• System Requirements for Several Applications

– Break

• Hardware Analysis: The Jam Cam

• Hardware Analysis: The Kodak DC210

• Hardware Analysis: The Chandra Orbital X-ray Telescope

Fundamentals of Camera ImageProcessing

Robert KremensMay 2001

Digital Camera Image Processing Pipeline

Color Filter Array (CFA)

AnalogProcessing& A/D Conv.

White Balance

Scene Balance

CFAInterpolation

Blurring*Gamma

CorrectionRGB to

YCC Conv.

ChromaSubsample

UnsharpMasking

(Edge Enh.)

JPEGCompress

FinishedFile Format

White Balance

• Usually performed on raw CFA pixels• White balance attempts to adjust for variations in the illuminant (D65,

Tungsten, etc.) by adjusting analog amplifier gain in the R,G,Bchannels

• What is white?– R=G=B= ~255

Implementing White Balance

• Method A - Predetermine the illuminant.– Acquire image.

• Camcorder method - white lens cap pointed at light source.• Can be a problem if scene has no white.

– R, G and B adjustment values calculated to make them scale to ~255.– Subsequent images have incoming raw pixels multiplied by

adjustment value.• Can be done on the fly with hardware, or in analog stages (preferable).

Cou

nt

Pixel Level0 255

Implementing White Balance (cont’d)

• Method B - Adjust each image after acquisition– Find area with R~G~B at highest intensity.

• Examine Full Image - time consuming.• Predetermined small image area - what if no white?• Subsampled image - faster.

– Determine adjustment values for R, G and B.– If G isn’t high enough, there is no white.

• Still possible to scale to gray by adjusting R and B?• Or just leave it alone.

• Flash illumination removes much of the need for determining theadjustment parameters, since the color spectrum of theillumination source is known.

Digital Camera Image Processing



White Balance

Scene Balance

CFAInterpolation

Blurring*Gamma

CorrectionRGB to

YCC Conv.

ChromaSubsample

UnsharpMasking

(Edge Enh.)

JPEGCompress

FinishedFile Format

Scene Balance

• Adjusts the color balance of an image so that neutral images areseen as neutral.

• Adjust color planes throughout their range; can use adjustmentthat is a function of pixel value.

• Unless subject is holding a neutral density chart, this is muchmore of an art than a science.

Implementing Scene Balance

• Look for areas of image with approximately equal R, G and Bvalues.

– Look at entire image - time consuming.– Look at sub-sampled image - can miss data.– Look at blocks of image.

• Create areas of image by averaging over 20x20 pixels.

• Histogram and scene classification are basic methods - forcecolor histogram to be ‘correct’

• Adjustment (multiplication) values must not affect overallbrightness of image.

– If R an B need to be increased, G should be decreased also.

• Be careful of interaction w/ White Balance.




White Balance

Scene Balance

CFAInterpolation

Blurring*Gamma

CorrectionRGB to

YCC Conv.

ChromaSubsample

UnsharpMasking

(Edge Enh.)

JPEGCompress

FinishedFile Format

CFA Interpolation

• CFA interpolation creates 3 separate bit planes for each pixellocation of the sensor.

• Weighted averages are typical, but algorithms vary depending onfilter array patterns.

• Can be clever and use adaptive algorithms to reduce sub-sampling effects and undesirable artifacts (‘zippers’)

Implementing CFA Interpolation (Median method)

00 01 02

10 11 12

20 21 22

30 31 32

03

13

23

33

• Mean Interpolation (Green)– G11 = (G01 + G10 + G12 + G21)/4

• Median Interpolation (Green)– G11 = [(G01 + G10 + G12 + G21) - MAX(G01 + G10 + G12

+ G21) - MIN(G01 + G10 + G12 + G21)]/2

• Red and Blue may interpolated differently than green.– R11 = (R00 + R02 + R20 + R22) / 4– B11 = B11– R12 = (R02 + R22) / 2– B12 = (B11 + B13) / 2– B22 = (B11 + B13 + B31 + B33) / 4

Some Observations on CFA Interpolation

• The math is not complicated (adds, compares and shifts)– Data re-organization is key to speed.

• Barrel shifters and/or byte extraction instructions.• MMX style pack and unpack.

– SIMD instructions (such as in MMX) can greatly accelerate math.– A good candidate for hardware acceleration.

• Arithmetic compares, adders and shifters are easy to implement.

• Interpolation schemes can cause image artifacts.– Edges, corners and stripes present a problem.

Color Spaces and Standards

• How is the image represented in R,G,B space?– Attempt to maximize color gamut and psycho-visual quality while minimizing

non-linear effects.

• CCIR 601 - now ITU-R BT.601– Digital Video Standard– Y’CrCb (Y’Cr’Cb’) Color Space, 4:2:2 Subsampling

• Y excursion 0 - 219, offset = 16 (Y = 16 to 235)• Cx excursion +/- 112, offset = 128 (Cx = 16 to 240)

– No assumptions about white point

• CCIR 709 - now ITU-R BT.709– HDTV Studio Standard– Y’CrCb Color Space

• Y excursion 0 - 219, offset = 16 (Y = 16 to 235)• Cx excursion +/- 112, offset = 128 (Cx = 16 to 240)

– Specifies White Point (x = .3127, y = .3290, z = .3582) (D65)– Specifies dark viewing conditions.

Color Spaces and Standards (cont’d)

• sRGB (called NIFRGB by Kodak)– Default color space for HP and Microsoft– Same as CCIR 709 except

• Specifies DIM viewing environment• Full 0 - 255 encoding of YCrCb values

• Photo YCC– Also uses CCIR 709

• White is 189 instead of 219• Results in RGB values from 0 - 346 when reconverted• Chroma channels are unbalanced (supposedly follows distribution of colors in a real

scene)




White Balance

Scene Balance

CFAInterpolation

Blurring*Gamma

CorrectionRGB to

YCC Conv.

ChromaSubsample

UnsharpMasking

(Edge Enh.)

JPEGCompress

FinishedFile Format

Gamma Correction

• Gamma describes the nonlinear response of a display device (CRT) toan applied signal.

• RGB values must be corrected for Gamma before they aretransformed into a video space.

L’709 = {4.5L, L<=0.0181.099L0.45-0.99, 0.018<L

Ligh

t In

tens

ity

Video Signal




White Balance

Scene Balance

CFAInterpolation

Blurring*Gamma

CorrectionRGB to

YCC Conv.

ChromaSubsample

UnsharpMasking

(Edge Enh.)

JPEGCompress

FinishedFile Format

R’G’B’ to Y’CrCb Conversion

• Conversion to YCrCb occurs for 2 reasons:– Chroma can be subsampled.

• Eye is more responsive to intensity changes (G) than color changes (R,B)• Can compress color channels (R,B) for smaller stored image

– Video output potential.

• Well known conversion matrices convert Gamma Corrected RGBto Y’CrCb.

Y’CrCb

= 0.257 0.504 0.098 0.439 -0.368 -0.071-0.148 -0.291 0.439

R’G’B’

0*128128

+

* 0 for UPF format, 16 for CCIR 601

Implementing RGB - YCrCb Conversion

• Color space conversion can be implemented in several ways.– Straight software implementation

• Flexible, but inefficient

– Hardware assist• Single cycle MAC - 9 clock cycles• SIMD instructions - 3 clock cycles

– Straight Hardware• Fast - 3 clock cycles• Costly

– 3 Dimensional Lookup Tables

Red

Green Blue

Implementing Nonlinear Color Space Conversion -3D Lookup Tables

• CMY Space is very device dependent and the conversions fromRGB, L*ab or YCrCb are not linear.

0,0,255

255,0,0

0,255,0

255,255,255


• A 3-D Lookup Tabledefines the conversionfor specific colors

• A fully populated tablewould have over 16Mentries (256x256x256)

• A subset of entries ischosen to populatethe table.

0,0,255

255,0,0

0,255,0

255,255,255


• The actual values for the conversion are interpolated using variousmechanisms.

– Tri-linear interpolation (10 */ 7+-)– Prism Interpolation (8 */ 5+-)– Tetrahedral Interpolation (6 */ 3+-)– Pyramid Interpolation (7 */ 4+-)– Fuzzy Logic methods

• Simple Table ExampleC,M,Y Y,Cr,Cb

0,0,0 219,120.2,128.0

0,0,128 192,120.5,0.8

0,0,255 .

0,128,0 .

0,128,128 . … .

255,255,255 0.2, 120.5,127




White Balance

Scene Balance

CFAInterpolation

Blurring*Gamma

CorrectionRGB to

YCC Conv.

ChromaSubsample

UnsharpMasking

(Edge Enh.)

JPEGCompress

FinishedFile Format

Chroma Subsampling

• The human eye is much more sensitive to intensity variations thancolor variations.

• Some color information can be discarded without loss of imagequality.

(Y) Luminance (RS-170)

+

Chrominance (I & Q)

=

Chroma Subsampling

• 4:2:2 and 4:2:0 are typical subsampling ratios– 4:2:2 is typically used in video.– 4:2:0 is prevalent in still photography.

CbCb

Cb Cb Cb

Cb

Cb

Cb

CrCr

Cr Cr Cr

Cr

Cr

Cr

YY

Y Y Y

Y

Y

Y

Cb

Cb Cb

Cb

Cb

Cb

CrCr

CrCr Cr

Cr

Cr

Cr

YY

Y Y Y

Y

Y

Y

CbCbCb

Cb

CbCb

CrCr

CrCr Cr

YY

Y Y Y

Y

Y

Y

4:4:4 (No subsampling)

4:2:2 4:2:0

What about other image processing?

• In consumer cameras, a large amount of post-capture processingtakes place to enhance the quality of the image.

• Blurring and subsequent unsharp-masking are common imageimprovements in processing chains

• Blurring with a 3 X 3 convolution kernal after CFA interpolationreduces artifacts (Moire patterns, ‘zippers’, color banding)

• Implementation of 3 X 3 convolution with various kernals is wellknown

• Unsharp masking:– Add a edge-enhanced image to the original image to sharpen lines R = O + ααS where on a pixel-by-pixel basis original image (O) is added to a

fraction (αα) of a sharpened or edge-extracted image (S)– Requires sharpened copy of image - can be performed in 3 row blocks

for 3X3 sharpening kernal

What is the future direction of the digital cameraprocessing chain?

• Increased sensor size will require higher computing power tomaintain image quality

• Multiple output CCD’s can use parallel processing for some imagefunctions

• Smaller pixel size and decreased exposure latitude will requiremore accurate image processing to achieve accurate images

• Sensor development will halt around 3-4 Mpixels barring surprisesin process development

• Movie modes (MPEG or AVI stream) may be desirable as storagedevices increase in capacity

• Appearance of high capacity random access magnetic recording,video, audio, HDTV, high speed radio packet network, could usherin a new devices

Modern Sensor CharacteristicsSolid State Imager Basics

Robert KremensMay 2001

31

Outline

• What Are The Basic Characteristics of Solid State Imagers?• What are the Different Solid State Imager Implementations?• Where are We Headed in The Future?

Imager Characteristics: Pixel Size & Pitch

• Smaller pixels create marketing trade-offs– More pixels per specific die size (increased resolution)– Smaller die size for same number of pixels (lower cost)

• Creating smaller pixels is an engineering tradeoff– Smaller pixels have less dynamic range (total volume of charge

collecting depletion region is smaller)– For some transport mechanisms, smaller pixels will be noisier.– Unless gate metalization is also shrunk, sensitivity will suffer (more of

pixel is covered by metal)

• Current commercially available minimum pixel size is 3.6 µµm(obtained in several ‘consumer’ 2-3 megapixel CCDs)

Imager Characteristics: Fill Factor

• Fill factor (aperture ratio) is the ratio of usable sensor area to thetotal pixel area.

• Current Fill Factors range from 25% up to near 100% dependingmainly on the transfer technology used.

Pixel Area

Active Area

Pixel Pitch

Impediments to a good fill factor

- Gate metalization

- Interline shift registers

- Active pixel amplifiers

- Anti-blooming structures

Imager Characteristics: Dark Current

• Dark current is thermally induced charge carriers generated byimpurities and silicon defects.

• It manifests itself as a DC offset, which in turn lowers the dynamicrange of the device.

• Dark current is non-uniform from pixel to pixel resulting in fixedpattern noise - only continuously clocked CCDs avoid this effect.

Imager Characteristics: Noise and Defects

• Fixed Pattern Noise - constant “speckle” under uniformillumination conditions

– Dependent on transfer method and usage– Not easily removed by process or design changes

• Thermal Noise - shot noise– Independent of transfer method (silicon is silicon)

• Readout noise - clocking noise, active circuitry noise– Shading

• Sensor Defects– Bad pixels, missing columns or rows, non-uniformity of response,

mis-aligned color filter array

• Reset noise - (kTC noise) discharge resistance thermal noise– Not easily removed by process or design changes

Imager Characteristics: Reset and Fixed PatternNoise Removal

Imager +-

Delayed Output

SH Pulse

Output SignalAmp Delay

n Delayed Data Sampling (DDS)

Imager +-

SH1

SH2SH1 Pulse

SH2 Pulse

Output SignalAmp

n Correlated Double Sampling (CDS)

Imager Characteristics: Readout Speed

• Measured in pixels/sec range - 50K to 10M• Integration Time

– Too little - Affects sensitivity (not enough electrons stored)– Too much - Affects noise floor (thermal noise also accumulates)

• Read Out Rate– Too fast - Affects charge transfer, burdens clock driver circuits– Too slow - increased integration time, can’t output standard video

• Quick Calculation– Want VGA (640x480) at 30 frames/sec ~ 300Kpixels/sec– Assume 4 phase horizontal and 4 phase vertical clocks– 640 x 480 x 30 = 9.2MHz (Clock generator would need 36MHz)

Imager Characteristics: Dynamic Range

• Dynamic range is the pixel well capacity (in electrons) divided bythe r.m.s. noise floor (in electrons).

• Sometimes expressed in dB• 8 bits = 48.2dB 10 bits = 60.2 dB• Dynamic range is related to both processing (fixed) and voltage

bias levels (easily screwed up)• Typical Numbers

– ‘Full well’ capacity 30,000 - 2,000,000 e-– Noise <5 - 300 e-– Dynamic Range 40 - 100dB

Imager Characteristics: Blooming & Smearing

• Blooming occurs when a charge collector overflows into itsneighbor.

– More of a problem with CCD transfer mechanisms– Anti-blooming structures shunt excess charge away from active area

• Smearing is the effect of continued integration during the read outphase

– Can occur with all un-shuttered sensor/camera designs

-+

-+---

- - -------- -

-

++++

Charge Coupled Devices (CCD)

• CCDs are bucket brigade devices– Multiphase clocks transfer the charge

1 Pixel 1 Pixel

Φ1 Φ2 Φ3 Φ4 Φ1 Φ2 Φ3 Φ4V- V+ V+ V+ V- V+ V+ V+V- V- V+ V+ V- V- V+ V+V+ V- V+ V+ V+ V- V+ V+V+ V- V- V+ V+ V- V- V+V+ V+ V- V+ V+ V+ V- V+V+ V+ V- V- V+ V+ V- V-V+ V+ V+ V- V+ V+ V+ V-

n Blooming is a problem with CCDs, anti-blooming structuresare added which increases processing complexity and cost.

Full Frame CCDs

• Full Frame CCDs transfer each row incrementally.

n Full Frame CCDs exhibit significant smearing if it is notshuttered during readout.

n A Full Frame CCD can have an excellent fill factor.

Frame Transfer CCDs

• Frame Transfer CCDs combat smearing by rapidly transferringpixels through the active area.

Shielded

Area

n Smearing is still an issue.n Extra silicon is needed.n Fill factor can be very good.n Next image can be acquired while

previous is being read out.

Interline Transfer CCDs

• Interline Transfer CCDs have transfer shift registers along eachcolumn.

n Smearing is eliminated at the cost of reduced Fill Factor.n A second image can be acquired during readout.

Shielded

Progressive Scan CCDs

• Camcorders and video are a significant market driver for CCDs -hence many CCDs have interlaced readout.

• Progressive scan devices are non-interlaced, but can be frametransfer, interline transfer or any other transfer method.

Progressive ScanInterlaced Scan

Charge Injection Devices (CID)

• Charge Injection Devices have individually addressable pixels andnon-destructive readout.

n CIDs do not haveblooming problems.

n Fixed pattern noise ishigh dynamic range islow.

n CIDs are RAD hard.n Electronic window and

zoom capabilities.

+-

Horizontal Scan

Ver

tical

Sca

n

Passive Pixel CMOS

• Passive Pixel CMOS Sensors are very low cost.

Horizontal Scan

Ver

tical

Sca

n

n Fixed Pattern noise is very high,dynamic range is low.

n VERY low power.n Small output signal level - charge

is placed on entire row for readout(many pF)

n Electronic windowing andzooming.

n Fill Factor can be nearly 100%.n On board clocking can greatly

simplify interface.n Not prone to blooming.

Active Pixel CMOS

• Active pixel CMOS sensors trade off Fill Factor for individual pixelelectronics and reduced noise.

Horizontal Scan

Ver

tical

Sca

n

n Current Fill Factor ~35%.n Same advantages as passive

CMOS, but significantly lessnoise.

n On board electronics - amps,A/Ds, Processing!

n Excellent dynamic range.

Active Pixel CMOS (cont’d)

• There are two approaches to solving the Active CMOS Fill Factorproblem.

– Make the array bigger. Causes problems with the optics, makingimage field wider, bad for fixed focus devices.

– Fabricate microlenses over each pixel.

– Microlenses have been manufactured, but are difficult to manufacturein high volume and add a significant cost.

– Anti-reflection coating these highly curved surfaces is difficult.

Side by Side Comparison of the visible sensortechnologies

CCD CID Passive PixelCMOS

Active PixelCMOS

Pixel Size ++ + ++ +

Readout Noise ++ -- -- +

Fill Factor ++(1) - ++ -

Dynamic Range + - - +

RAD Hardness -- ++ ++ +

Single Supply -- - ++ ++

System Power -- + ++ ++

System Volume - + ++ ++

SystemIntegration

-- -- ++ ++

System Noise + -- - ++

System Cost -- -- ++ +

Ease of Use -- -- ++ ++

ElectronicShutter

+ - ++ ++

ElectronicWindowing

-- ++ ++ ++

ElectronicZoom

-- ++ ++ ++

(1) - Frame transfercan be very good,interline transfer ispoor.

CCDs are radiation soft, but otherwise tend to bethe highest performance optical array sensors

Sensor Family Strength Weakness Family Deficiencies

CCDMature technology, mass production lessons

Radiation soft (20 - 30 krad)Radiation soft

Large size available (4K X 4K) Capacitive device, high power consumption (W to many W)

Low QE on shuttered devices

Very low noise Cannot operate cryoWidest experience base Specialized production lines

requiredFull frame CCD Lowest noise devices Smear with moving objectsBackthinned full frame CCD

Highest QE (~80%) of any visible detector

Difficult process, low yield

Very low noise (~2e)

Interline CCD Smear eliminated Low fill factor due to added structure

Frame transfer CCD

Smear eliminated Up to 1/2 silicon area wasted in storage register

High frame rates possibleInterline CCD with microlens

Improved fill factor (~60%) Microlenses radiation soft (plastic)Microlens works best with high f/ system.

CMOS sensors have not been manufactured in largesizes and lag CCDs in image performance

Sensor Family Strength Weakness Family DeficienciesCMOS Many lessons from CMOS

manufacturingHigh fixed pattern noise Small devices (1K X 1K)

Becoming highest volume sensor

High read noise Low QE, low fill factor

On-chip integration with amps, CDS, A/D,

Low fill factor Noisy (read and fixed pattern)

Very low power (10 - 100 mW) Poor sensitivity due to gate structure on top of pixel

Limited shutter availability

Very high frame rates possible (60 MHz pixel clock X 8 outputs)

Operating parameters change with radiation exposure

Radiation hard (~200 - 1000 krad)

Multiple outputs complicate electronics package

Single power supply No shutter capability - smearCMOS passive pixel sensor

Simplest structure CMOS deviceHigh sensitivity - large fill factor 70-80%

CMOS amp per pixel (APS)

High signal output, lower read noise

Fixed patern noise problems due to amplifier mismatch

CMOS APS with microlens

Improved fill factor (~40 - 50%) for higher sensitivity

Microlens works best with high f/ system.

CMOS amp per row (APR)

Higher signal output than PPS, simpler, better fill factor (~50%), less fixed pattern noise

Signal still small because of large column capcitance

Several CMOS alternatives exist

Sensor Family Strength Weakness Family DeficienciesCID Simple structure eases

manufacturingLow signal output

Extendable to large arrays High fixed pattern noiseVery radiation hard (~1000 krad)Random addressableHigh readout speed

Hybrid (CMOS multiplexer - bonded detector panel)

High readout speed Large scale devices only recently produced

100% fill factor, high QE Difficult to butt?Multiple wavelengths by altering photosensor plane

CMOS sensors have advantages in radiationtolerance and readout speed

• Natural extension of the original Reticon photodiode readoutarrays.

• Each pixel may have active transistors to amplify and buffersignal. This is very desirable form a S/N standpoint.

• No charge transfer across the sensor.• APS / CMOS can be cooled to low temperatures to reduce dark

current - no CTE limitations. Passive cooling in space providesample noise reduction.

• Since there is no long-range charge transport, dislocations frommassive charged particle does not degrade sensor performance.

• CID (w/o active pixel) has high read noise on the order of 300 e-.Increased sensor size will increase read noise (due to column/rowcapacitance increase). The CID is a passive pixel sensor.

CMOS sensors still have process-related imagequality problems

• Widely variable, poor sensitivity across array in many early CMOSsensors. CMOS sensors got a ‘bad rap’.

• Non uniformity of response and resultant fixed pattern noise dueto differences in size of photodiodes and storage capacitors.

• Recent designs with smaller features may allow improved fillfactor, spectral response and sensitivity for a given pixel pitch(Motorola and others).

– 0.35 µµm design rules currently used in US on 6 and 8 inch wafers.– 7.8 µµm pixel pitch with pinned photodiode.– Electronic exposure including rolling mode (usual for CMOS) and fully

shuttered mode.– QE ~ 22%, fill factor 35%

CMOS sensors performance issues can be solvedtoday with more transistors per pixel

• 5-transistor per pixelsensors have similar noiseperformance to CCDs, but(presently) low fill factor.

• Need larger pixels (20 mm)and smaller feature size(0.18 mm design rules).

4 transistors per pixel

CMOS noise performance may not be an issue

• Most noisy sensors are of‘commercial’ variety -computer cams, Barbiecameras, etc.

• Read noise can be reduced,fixed pattern noise correctedfor post-capture.

• When feature size isreduced, design must payattention to maximum signalfrom pixel to avoid overload.

• Also need is the plot ofnumber of transistors vs. fillfactor (or QE) parameterizedfor pixel size.

CCD vs. Active Pixel CMOS

• CCD imagers are still the reigning champion ineverything except low cost, low quality (toys &security) and RAD hard (space) applications.

• Active pixel imagers are a revolution in progress.– Powerhouses in the imaging, microprocessor and

memory fields are all throwing gobs of money intoActive Pixel R&D.

• CCDs may soon give way to AP CMOS imagers inthe consumer arena, but CCDs will remaindominant in the high end scientific applications(astronomy).

– What will happen to all those specialized CCD fabswhen the consumer market dries up?

Bob KremensMay 2001

System RequirementsConsumer Cameras to the Space Telescope

The electronic image systems in use today presenta conflicting set of design parameters for sensorsand electronics• Consumer Cameras (Sony Mavica series, Kodak DCS 2XX, etc.)

– Low cost– Large number of relatively small pixels

• Need high resolution to rival film• Small pixels OK for 8 bit dynamic range (more DR desirable, but…)

– Low power for long battery life– Rapid readout for shortened click-to-click time

• Professional digital cameras (Kodak DCS660, Fuji– Cost not as significant an issue– Very large number of larger pixels

• Need high resolution ( to match high end 35mm and 70mm formats) andhigh dynamic range

• 12 bit color planes (‘36 bit images’) are de facto standard• Need high readout speed to suit pro shooting style

Scientific applications are not cost sensitive butpresent other sets of challenges

• Scientific applications– Cost usually not an issue– Require highest dynamic range and lowest noise– Astronomical (ground and space based) imaging

• Huge number of pixels desirable (replace 20-30 cm film)• Extremely low noise for extended exposures

– Cooling to LN2 temperatures or less ‘expected’

• Wide, flat spectral response• Non-destructive readout

– Allows longer exposures on dimmer objects

• Freedom from blooming and adjacent pixel overload effects– Bright objects often next to dim objects of interest

• Radiation hardness necessary for space applications

The electronic image systems in use today presenta conflicting set of design parameters for sensorsand electronics (2)• Remote Sensing applications (satellites and aircraft)

– Can use linear or array sensors• Aircraft or satellite moves, can scan the area like a ‘pushbroom’

– Require sensitivity to reduce constraints on optics and stability toavoid constant re-calibration

– May require radiation hardness in space applications

Some example sensing systems: Astronomy

Ground based astronomy

Lincoln Laboratory 2K X 4K3-side edge buttable CCD

Can be made into an 8K X2NK array (N = 1 to …)

Backthinned (no front-surface structure) for ultrahigh quantum efficiency(~80%)

Optimized for low noise andslow speed readout with 16bit digitizer systems

Some example sensing systems: Astronomy

Sloan Digital Sky Surveyinstrument

Uses 30 2K x 2K SITebackthinned arrays.

Arrays cooled to -80 C.

That’s 126 megapixels perframe!

Some example sensing systems: Professionalcamera back

Phaseone-Lightphase

16 Mpixel large pixelPhillips CCD

14 bit digitizer for~42 bit ‘deep’images

Firewire IEEE 1394readout to PC orMac

16 megapixelcamera back

System HardwareA Digital Consumer Camera

-The Jam Cam-

Bob KremensMay 2001

The KB-Gear JamCam versions have sold close tomillions (?) of units

• VGA resolutionCMOS sensorcamera

• RS-232 and USBdatacommunications

• Fixed focus, fixediris, no shutter

• Simple ‘camera’type interface

• TWAIN drive, MSPicture-It!,ArcSoftPhotoFantasybundled software

What are the basic components of a digital camera?

• Imaging Optics - lens or mirror and mount• Image Sensor• Power Supply - potentially 7 - 9 power supplies for CCD camera

with flash and LCD/backlighter• Processing electronics - CPU / DSP, memory, program store• Exposure and focus control - shutter, iris• (Removable media) -• Camera communication - serial, USB, IrDA, Firewire, radio

The future market and profit leaders may not betoday’s camera/electronics companies

• Fine details of image quality (where great IP is required) notnecessarily an issue with low-end cameras.

• All present low-end cameras have ‘acceptable’ images for the useintended

• These cameras have fundamentally different architecture fromprevious ‘high-end’ cameras

• The popularity of these cameras modify some fundamental digitalcamera ‘truths’ that have been held since 1995

• Ample opportunity for other camera ideas/modalities to come todominate the electronic marketplace.

• 1.3 Mpixel camera produces quite acceptable 4” X 6” prints (Whythe need for more pixels?)

The present camera architecture makes someassumptions which may be invalid

• Assumptions- Removablestorage, in-cameraprocessing,finished files,image reviewon camera.

Image SensorZoom Lens/Focus/Aperture

Assembly

Image processing Microprocessor +auxillary ASICS

Analog Front End+ Analog

Processing

System ROM

System RAM

Removable Flash Media forImage Storage

Image Display LCD

InterfaceUSB

InterfaceRS-232

InterfaceRS-170Video

I/O Ports

Memory Bus

Motor Control

Sensor TimingGenerator

The architecture of the JamCam is similar to olderdigital cameras produced by first-line cameracompanies

Off-camera imageprocessing, fixedstorage, no imagedisplay.

CMOS IntegratedSensor

Digital Output

Fixed FocusFixed AperutreLens Assembly

Microprocessor withEmbedded USB/RS-232

Controller

SRAM

System ROM

USBPhysicalDriver

Fixed Flash ROMImage Store

Status Display LCD

The JamCam provides a totally acceptable cameraexperience - and this is all the consumer asks for!

• Extremely simple to use– Shutter button, mode switch– Mode switch selects resolution– Modes:

• Capture (normal power-on)• Delete (all)• Set resolution (VGA, 1/2 VGA. 1/4 VGA)

– Display indicates number of remaining pictures ala throw-awaycamera

– RS232 and USB interfaces– Fast turn on (~1.5 seconds)– Fast image store (~1.5 - 7 seconds)– Infinite battery life with 9V cell

The computer-centric JamCam model has the PC asthe center of entertainment.

• Computer-centric world has home CPU performing functions ofDVD, MP3, still image player, audio/radio/CD player,etc.

• Single I/O device moves to living room with display unit (HDTV).• Uses high CPU power of PC device to perform entertainment and

web tasks simultaneously.• Radio/IR links to cameras, remote I/O devices

In the present digital camera model, the portabledevice is (or can be) an ‘entertainment center’.

• The camera, as a powerful portable computing device with largelocal memory can also be a:

– Full featured still camera– Audio player (MP3)– Movie camera (short MPEGs)– Radio– Cell/radio Telephone– Audio recorder

• Will the future consumer desire an all-in-one portable‘entertainment center’?

The JamCam is composed of six major components

• The CMOS sensor minimizes thenumber of analog front-endcomponents

• Flash ROM, DRAM and ROM memorycomponents couple with processor.

• One Atmel microprocessorcomponent provides amplecomputing power for this size sensorand the limited processing done on-board.

• Two PLD ‘glue’ parts interface theLCD display and pushbuttons.

• Double sided SMT PCB!!• PCB-mounted image sensor and lens

assembly

The JamCam is memory rich and processor-poor

• 2 MByte Flash image storage• 2 Mbyte ROM• 256KByte static ram

• One low power, 8 bit RISCprocessor with embedded USBand other interface functionality

Limiting the number of mechanical componentsincreases ruggedness, reduces power consumptionand cost• Shutter and iris less lens assembly• Adequate exposure range is provided by electronic shuttering of

sensor• This system is similar to a primitive ‘box’ camera, with some

exceptions:– Short focal length lens provides large depth of field– Large exposure latitude provided by wide range electronic shutter on

sensor

• Image acquisition limitations:– Fixed depth of field

• Not an issue in consumer cameras

– Limited focus range• 2’ - infinity: similar to best AF film point and shoot cameras

– Limited exposure range• Shutter speed typically

The component count in the JamCam may benearing a minimum

• Double sidedPCB

• One ‘major’CPUcomponent

• Processingtime (tocompressedfinished file):

38.5 kpixels/sec.

Component Vendor Function NumberAT43320 Atmel AVR Risc Core with USB Hub and Embedded USB 1AT49LV1614 Atmel 1M X 16 sectored flash RAM 1AT27BV1024 Atmel 1M X 16 non-window EPROM 1ATF16LV8 Atmel 200 gate EPLD 2TC55V2001 Toshiba 2 Mbit (256K X 8) SRAM 1DS14C232 RS232 driver 174HC273 Glue 474HC4040 Glue 174HC00 Glue 174HC244 Glue 1CD4013 Glue 1LCD ? DisplayResistors 22Capacitors 29Xsistors 4Gates Individual gates 212 MHz Xtal 1Switches SPST 2USB conn USB connector 11/8" Mini 1/8" mini stero phone 1HDCS-2000 HP CMOS 640 X 480 image sensor 1

TOTAL 78

System HardwareA Mid-Range Consumer Digital Camera

Kodak DC210

Bob KremensMay 2001

The Kodak DC210 was a ‘second generation’ 1Mpixel digital camera that sold well

2:1 f/ 4auto-focuszoom lens

Flash and ambient lightexposure sensors

Powerfulflash unit

Consumer digital cameras integrate optics, imagesensors and image processing electronics

• Extremely compact three-dimensional packaging

• Flex board, double side SMTessential for density on mid-rangeand high-end cameras

• Separation of boards according tofunction is common to reducenoise

– Flash unit almost always separate– CCD/sensor PCB attaches to rear of

optics package with flex cable– LCD and backlighter (discharge

lamp) separate

What are the basic components of a digital camera?

• Imaging Optics - lens or mirror and mount• Image Sensor• Power Supply - potentially 7 - 9 power supplies for CCD camera

with flash and LCD/backlighter• Processing electronics - CPU / DSP, memory, program store• Exposure and focus control - shutter, iris• (Removable media) -• Camera communication - serial, USB, IrDA, Firewire, radio

The optics/imager/focus/zoom package is w wonderof miniaturization

• Sensor attaches to rear of optics package - fairly universal in thisarena

The goal of the image processing package is theproduction of a ‘finished’ file on a removable media.

There are severalways to assemblethese componentsfrom basiccomponents orcustom ASICdevices

Depending onperformance levelof camera and sizeof image sensor, asingle CPU mayperform all camerafunctions

ImageSensor

AnalogPreprocessing

Basic ImageCreation

Focus andExposureSensors

Camera Control

User InputButtons, Dials

Display

Image IntellectualProperty

JPEG or OtherCompression

RemovableMedia

Aperture,Zoom andFocus Control

Main CPU

The image processing electronics must beinexpensive, low power and provide adequateprocessing• Consumer cameras are still low (no?) margin products - pennies

count!

• Short battery life has been a serious user complaint for first andsecond generation digital cameras - much effort has beenexpended to increase battery life

• The requirements of high processing horsepower and low powerconsumption tend to be mutually exclusive - designers oftenforced to compromise and idle components when not used.

The electronic architecture must also be familiarenough to develop the camera application easily

• Product lifetime is very short in this market (< 1year)• Development cycle is correspondingly short so exotic

architectures or components are less popular even whenproviding superior performance

• Complex interaction between system elements and processorsmay require real time operating system (RTOS), multi-taskingcapability and complex development system support

• Large amounts of code necessary for implementation of companyintellectual property requires source code reuse and control

• Good development tools required!• Often, the best development tools are available for ‘general

purpose’ processors, so these are found in many camera designs(Motorola PowerPC, Hitachi SH-DSP, SPARC, etc.)

The camera electronics performs other importantinternal functions in addition to image processing

These systems are incrediblycomplex and the imageprocessor is only a part of the‘picture’ (multi-processor)

Much near-real-timeprocessing for auto-focus,auto-exposure, powermanagement and flash control(RTOS)

Camera must remainresponsive to user input inspite of other operationsoccurring simultaneously(multi-tasking)

ReadyLight

ShutterLensPosition

MotorDrive

IrDaDrive

Processor (1) Processor (2)

Analog (RF, serial, power,Flashlamp, DC-DC

Converter)

CCD Driver / Interface

Power/InterfaceExposureSensor

StatusLCD

FlashLamp

PhotoflashCap.Batteries

RearSwitch

CCDCarrier

LCD Interface

Kodak has chosen a ‘computer-centric’ architecturefor most of their consumer cameras

It is clear that theKodak computer +camera approach is onethat facilitates change,code control and easeof development

Ancillary camerafunctions relegated to 8bit micro with link tomain processor

Lots of memory - like areal computer!

1160 X 872 CCDGain, A/D, CCD Timing

GenerationInput Signal Conditioning

0.5 MB Dual Port VideoDRAM

LCD and Composite VideoBuffer

Control and Routing ASICsData Routing and Camera

ManagementLCD Rear Panel Display

0.5 MB Flash RamCCD Parameters, System

Code

4 MB CompactFlashMemory Card

Removable Picture Storage

DSP - RISCMicroprocessor

Image Processing,Compression

2 MB Working DRAMLocal Storage, Working

Memory

8 Bit MicrocontrollerExposure control, Controlinputs, voltage monitor,

Status LCD driverStatus LCD

System HardwareThe Chandra Orbital X-Ray Observatory

Bob KremensMay 2001

Astronomers study the full spectrum ofelectromagnetic radiation from the cosmos

Chandra is quite a different camera, but has thesame systems and requirements of the Jam Cam

Basic cameracomponents:

• Imaging Optics• Image Sensor• Power Supply• Processing electronics• Exposure and focus

control• (Removable media)• Camera

communication

X-rays are energetic photons, and can be detectedby most solid state optical sensors

• Incident X-ray deposits energy in pixel via the photoelectric effect.

• Chandra observations are centered in the region under 10 KeV

• A photon will generate a charge carrier for each 2.6 eV of energy,e.g. 2.6 KeV X-ray generates 1000 electron-hole pairs

• The incident flux is very low: incoming photons can be counted.The detectors are read out rapidly and the image ‘accumulated’.Each photon’s ‘signature’ (energy and position) can be preciselyknown if the detector is energy sensitive

• This is a very different situation from most optical photographyapplications, where zillions of photons impinge on the detectorsimultaneously and the detector is read once per image

X-ray optics are operated at glancing incidence

Chandra’s optics are ultra-precise cylindrical mirrorsof parabolic andhyperbolic section

Aperture of telescope ~1.2 m

It was very difficult toprovide the accuracynecessary for theseconcentric cylinders optics(Eastman Kodak)

This instrument wascharacterized beforeorbital insertion!!!

Chandra has two focal plane instruments

• High Resolution Camera - (HRC)high spatial resolution and veryhigh sensitivity, but limitedenergy resolution

– Stacked microchannel platesand crossed wire grid readoutarray

– Pulse from incident X-rayimpinges at intersection of twowires, defining location ofincident photon

• Advanced CCD ImagingSpectrometer - (ACIS) -simultaneous energy andspatial location using CCDs

The CCD in the ACIS focal plane is read out rapidly

Most CCDs in astronomicalapplications are read outinfrequently

The 10 CCDs in Chandra’sACIS instrument are read outrapidly and continuouslywhen on target to insure onlya single photon will becaptured in each pixel

This readout method assuresthat energy and spatialinformation may be obtainedfor each photon - photonsare rare!!!

The CCDs are operated in ‘single hit’ mode andcreate spectra and images simultaneously

Eight frame transferCCDs are used in the0.5o focal plane

Quick frame transferto storage area - thenreadout to determineenergy and position

Total pixels:

The block diagram for the ACIS instrument lookssimilar to a consumer digital camera!

• Except for thermal control in Chandra, a CCD camera is a CCDcamera

The data stream is processed by the Earth station

• Not much image processing is done on the ‘camera’ - raw datastream is telemetered to Earth for subsequent image re-creation

Chandra is proving to be as spectacular aninstrument as the HST

• This is the first decent glimpseof the universe at thesewavelengths, uncovering newphysics and providing a host ofsurprises

• This instrument has tag-teamedwith the Hubble SpaceTelescope to probe the universeover a wide spectral range.

• Especially important are thedifferences in the visual and X-ray images of the same object

Compact galaxy group HGC62 in 50,000second exposure from Chandra

Image Processing System Applications: from Barbie Cams to the

Documents

Transcript of Image Processing System Applications: from Barbie Cams to the