Developing Sensor Applications on Intel® Atomâ„¢ Processor-Based
IMAGE SENSOR PROCESSOR (ISP)
Transcript of IMAGE SENSOR PROCESSOR (ISP)
![Page 1: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/1.jpg)
PUBLIC USE
MARIE-ANNE LE MENN
ADAS MICROPROCESSOR APPLICATION ENGINEER
FTF-AUT-N1807
MAY 19, 2016
FTF-AUT-N1807
IMAGE SENSOR PROCESSOR
(ISP)
![Page 2: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/2.jpg)
PUBLIC USE1 #NXPFTF PUBLIC USE1 #NXPFTF
AGENDA
• The purpose of the ISP
−S32v234
−Camera interface
− Image sensor Processing
− Image pipeline
• Its architecture
• How to program it
![Page 3: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/3.jpg)
PUBLIC USE2 #NXPFTF
THE PURPOSE OF
THE ISP
![Page 4: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/4.jpg)
PUBLIC USE3 #NXPFTF
S32V234:
ADAS PROCESSOR
![Page 5: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/5.jpg)
PUBLIC USE4 #NXPFTF
S32v234
• ADAS processor:
−Sensor fusion
−Front view camera
−Surround-view
![Page 6: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/6.jpg)
PUBLIC USE5 #NXPFTF
![Page 7: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/7.jpg)
PUBLIC USE6 #NXPFTF
![Page 8: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/8.jpg)
PUBLIC USE7 #NXPFTF
![Page 9: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/9.jpg)
PUBLIC USE8 #NXPFTF
![Page 10: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/10.jpg)
PUBLIC USE9 #NXPFTF
S32v234: Block Diagram
![Page 11: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/11.jpg)
PUBLIC USE10 #NXPFTF
S32v234: Block Diagram
![Page 12: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/12.jpg)
PUBLIC USE11 #NXPFTF
S32v234: Block Diagram
![Page 13: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/13.jpg)
PUBLIC USE12 #NXPFTF
S32v234: Block Diagram
![Page 14: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/14.jpg)
PUBLIC USE13 #NXPFTF
S32v234: Block Diagram
![Page 15: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/15.jpg)
PUBLIC USE14 #NXPFTF
S32v234: Vision Application
Image Pre-Processing
• ISP
• Multi-streams connectivity
Features Extraction
• Corners
• Edges
• Intensity gradients
• Shapes
Features Classification& Prediction
• SVM
• Adaboost
• K-nearest neighbor
Multi Image Processing
• Tracking, motion estimation
• Optical flow & disparity
• Stitching
Object Recognition & Fusion
• Object recognition/pedestrian
• Augmented reality
• Face detection
GFX Overlay & Video Distribution
• Safe fusion
• Graphic overlay & display
• 2D vs. 3D projection
CPU Platform
Cortex - A53
32kB I-cache
2 way
NEON
32kB D-cache
4 way
Cortex - A53
32kB I-cache
2 way
NEON
32kB D-cache
4 way
Cortex - A53
32kB I-cache
2 way
NEON
32kB D-cache
4 way
Cortex - A53
32kB I-cache
2 way
NEON
32kB D-cache
4 way
L2 Cache – 256kB 16 ways SCU
Dual Camera Interfaces
2 x MIPI CSI2
Image Signal Processing
HDR
Color Conversion
Tone Mapping
Parallel 20 bit
Image Cognition Proc.
L-mem
Sequencer
L-mem
32 CU 32 CU
DMA
APEX2 CL
Image Cognition Proc.
L-mem
Sequencer
L-mem
32 CU 32 CU
DMA
G2-APEX-642
Image Proc. PlatformGfx & Display
GPU OpenGL ES 3.0
DCU 18/24 bits RGB
Video Codec H.264
8-12 bits Encoder
8bit Decoder
Video & Display Platform
High bandwidth operations
Scalable MIMD local memory
Soft ISP
Scalable RISC – data fusion
SIMD co-processor - neon
Memory hierarchy and coherency
Graphic
Video codec
Smart display
![Page 16: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/16.jpg)
PUBLIC USE15 #NXPFTF
CAMERA
INTERFACES
![Page 17: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/17.jpg)
PUBLIC USE16 #NXPFTF
Camera Interface: MIPI-CSI
• MIPI-CSI2 standard
• 4 lanes up to 1.5Gbps each
• 4 virtual channels
• In theory: up to 6Gbps for 1 MIPI
interface
• In reality:
− No camera with such bandwidth
− LVDS limitation
S32v234 has 2 MIPI-CSI interfaces
![Page 18: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/18.jpg)
PUBLIC USE17 #NXPFTF
Camera Interface: MIPI-CSI
• Typical sensor resolution:
−1280x800 @ 30/60 fps
−1920x1080 @ 30/60 fps
• Examples of camera:
−Sony: IMX224MQV
−Omnivision: OV10640
![Page 19: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/19.jpg)
PUBLIC USE18 #NXPFTF
Camera Interface: Multiple Cameras
![Page 20: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/20.jpg)
PUBLIC USE19 #NXPFTF
Camera Interface: Ethernet
• 1Gbps interface
• Easy synchronisation via AVB
• Embedded decoder for up to 4 video streams
![Page 21: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/21.jpg)
PUBLIC USE20 #NXPFTF
Camera Interface: Ethernet
![Page 22: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/22.jpg)
PUBLIC USE21 #NXPFTF
Camera Interface: Parallel
• Up to 100MHz pixel clock
• Pin out options:
−2 interfaces x16-bit
−1 interface x20-bit + 1 Interface x12-bit
Signals:
• Data [0:19]
• Vsync, Hsync, PixClk
• I2C
• Power-down
![Page 23: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/23.jpg)
PUBLIC USE22 #NXPFTF
IMAGE SENSOR
PROCESSING
![Page 24: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/24.jpg)
PUBLIC USE23 #NXPFTF
Raw Camera Image
![Page 25: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/25.jpg)
PUBLIC USE24 #NXPFTF
Raw Camera Image
Why does it look greenish?
![Page 26: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/26.jpg)
PUBLIC USE25 #NXPFTF
Bayern Pattern
![Page 27: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/27.jpg)
PUBLIC USE26 #NXPFTF
White Balancing
Relative intensity has been normalized for each temperature (in Kelvins).
![Page 28: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/28.jpg)
PUBLIC USE27 #NXPFTF
White Balancing
![Page 29: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/29.jpg)
PUBLIC USE28 #NXPFTF
Raw Camera Image
Why is there 2 frames?
![Page 30: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/30.jpg)
PUBLIC USE29 #NXPFTF
HDR: High Dynamic Range
![Page 31: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/31.jpg)
PUBLIC USE30 #NXPFTF
HDR: High Dynamic Range
High
exposure
Low
exposure
Combination of
the 2 exposures
![Page 32: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/32.jpg)
PUBLIC USE31 #NXPFTF
Image Sensor Processing
• White balancing
• De-bayering
• HDR
• Black level
• Noise filtering
• Vignetting
• Colour conversion: YUV, Gray scale
![Page 33: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/33.jpg)
PUBLIC USE32 #NXPFTF
ISP Solutions
MCU
or
Bare Sensor
+
MCU
BOM Cost
ISP
Heat Problem
+Integrated ISP
extra: $3-4
Bare Sensor
+
Companion
+
MCU
ISP
Companion chip:
$3-4
S32v234: integrated solution
Sensor
![Page 34: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/34.jpg)
PUBLIC USE33 #NXPFTF
Advantages
• Manage power on sensor
• Optimize BOM and board
• Smaller size through integration
Save Money
![Page 35: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/35.jpg)
PUBLIC USE34 #NXPFTF
Image Pipeline
![Page 36: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/36.jpg)
PUBLIC USE35 #NXPFTF
Image Pipeline
![Page 37: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/37.jpg)
PUBLIC USE36 #NXPFTF
Image Pipeline
![Page 38: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/38.jpg)
PUBLIC USE37 #NXPFTF
Image Pipeline
![Page 39: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/39.jpg)
PUBLIC USE38 #NXPFTF
When to Use the ISP?
• Ethernet: ISP is already integrated with the camera
• MIPI-CSI & parallel interface:
−External ISP required for more than 2Mpix
− Internal ISP will save money for mono or stereo cameras
![Page 40: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/40.jpg)
PUBLIC USE39 #NXPFTF
ITS ARCHITECTURE
![Page 41: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/41.jpg)
PUBLIC USE40 #NXPFTF
WHAT IS A KERNEL?
![Page 42: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/42.jpg)
PUBLIC USE41 #NXPFTF
Kernel
• Do not mistake:
−Linux kernel
![Page 43: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/43.jpg)
PUBLIC USE42 #NXPFTF
Kernel
• Do not mistake:
−Linux kernel
− Image processing kernel
![Page 44: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/44.jpg)
PUBLIC USE43 #NXPFTF
Kernel
• Do not mistake:
−Linux kernel
− Image processing kernel
−APEX kernel
− ISP kernel• Both can execute image processing kernel but not only
• Different:
• Programing method
• Capabilities
• Engines
![Page 45: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/45.jpg)
PUBLIC USE44 #NXPFTF
ISP Kernel
• Assembly code
• Running on the Image Processing Units (IPU) of the ISP
• Each IPU can run a different kernel
• Examples:
−De-bayering
−Noise filtering
−Colour conversion
−…
![Page 46: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/46.jpg)
PUBLIC USE45 #NXPFTF
ISP Kernel
• A lot of image processing steps (kernels) require multiple lines:
De-bayering Sobel filtering
IPUs can fetch multiple lines at the same time
![Page 47: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/47.jpg)
PUBLIC USE46 #NXPFTF
ISP SUBSYSTEM
![Page 48: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/48.jpg)
PUBLIC USE47 #NXPFTF
ISP Sub-system
IPUS
=
Scalar Image Processing Unit
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
![Page 49: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/49.jpg)
PUBLIC USE48 #NXPFTF
ISP Sub-system
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…IPUV
=
Vector Image Processing Unit
![Page 50: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/50.jpg)
PUBLIC USE49 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Each input/output
can fetch/output
multiple data at the
same time
![Page 51: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/51.jpg)
PUBLIC USE50 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
![Page 52: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/52.jpg)
PUBLIC USE51 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
M0
Core
![Page 53: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/53.jpg)
PUBLIC USE52 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Data
RAM
M0
Core
Instr.
RAM
![Page 54: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/54.jpg)
PUBLIC USE53 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
Crypto
protected
![Page 55: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/55.jpg)
PUBLIC USE54 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Configuration and
synchronization signals
![Page 56: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/56.jpg)
PUBLIC USE55 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Fast DMA
DDR
![Page 57: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/57.jpg)
PUBLIC USE56 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Fast DMA
DDRConfiguration and
synchronization signals
done
start
![Page 58: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/58.jpg)
PUBLIC USE57 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
Instr.
RAM
M0
Core
done
start
Programmable
Engines
Fast DMA
DDR
done
start
![Page 59: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/59.jpg)
PUBLIC USE58 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Fast DMA
DDR
done
start
Frames are
processed on the
fly by the ISP
through the IPUs
MIPI-CSI
Camera
![Page 60: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/60.jpg)
PUBLIC USE59 #NXPFTF
IPU
![Page 61: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/61.jpg)
PUBLIC USE60 #NXPFTF
Scalar Engine (IPUS)
![Page 62: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/62.jpg)
PUBLIC USE61 #NXPFTF
Scalar Engine (IPUS)
• Input window: 3x3 or 9x1
• Up to 3 Inputs:• Ex: R, G, B
![Page 63: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/63.jpg)
PUBLIC USE62 #NXPFTF
Scalar Engine (IPUS)
• Instruction memory
• Decoding unit
• No data memory (no load/store), only registers
![Page 64: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/64.jpg)
PUBLIC USE63 #NXPFTF
Scalar Engine (IPUS)
Working matrixMatrix ALU (3x3 -> 3x3)
![Page 65: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/65.jpg)
PUBLIC USE64 #NXPFTF
Scalar Engine (IPUS)
Mask matrix: To select
on which elements of
the matrix the operation
or accelerator should be
applied to
3X3 Accelerators:
• Sum, scaled, clipped
• Max, median, min
Other accelerators:
• Histogram
• PRNG
![Page 66: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/66.jpg)
PUBLIC USE65 #NXPFTF
Scalar Engine (IPUS)
Up to 2 output
![Page 67: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/67.jpg)
PUBLIC USE66 #NXPFTF
Vector Engine (IPUV)
5x5 input
windows
4-way SIMD ALUHandles operation with:
• Horizontal vectors
• Vertical vectors
• Scalars
![Page 68: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/68.jpg)
PUBLIC USE67 #NXPFTF
IPU Engines
• 8 x IPUS & 4 x IPUV
• 500MHz per Engine: 6000MHz total
• Pixels are 16-bit fixed point data in signed or unsigned mode
• Input and output managed by the StreamDMA engines
![Page 69: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/69.jpg)
PUBLIC USE68 #NXPFTF
Input Matrix: IPUS
In2 In1 In0
In4 In2 In3
In7 In6 In5
In8 In7 In6 In5 In4 In3 In2 In1 In0
3x3
9x1
![Page 70: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/70.jpg)
PUBLIC USE69 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
Input image
Input matrix
![Page 71: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/71.jpg)
PUBLIC USE70 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
1st Input
![Page 72: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/72.jpg)
PUBLIC USE71 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
2nd Input
![Page 73: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/73.jpg)
PUBLIC USE72 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
… 5th input
![Page 74: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/74.jpg)
PUBLIC USE73 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
1st Input of the 2nd line
![Page 75: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/75.jpg)
PUBLIC USE74 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50
3x3 mode 1st Input
![Page 76: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/76.jpg)
PUBLIC USE75 #NXPFTF
HW Replacement Modes for StreamDMA Engines
StreamDMA has a lot of
additional features:
• Scaling
• Multiple data format
support
![Page 77: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/77.jpg)
PUBLIC USE76 #NXPFTF
Input Matrix: IPUV
I
n
2
I
n
1
I
n
0
I
n
4
I
n
2
I
n
3
I
n
7
I
n
6
I
n
5
????????
![Page 78: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/78.jpg)
PUBLIC USE77 #NXPFTF
SEQUENCER
![Page 79: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/79.jpg)
PUBLIC USE78 #NXPFTF
Sequencer: Block Diagram
• Mapped into Host address space
• Peripherals can also be controlled by Host
CPU: IPUs and FastDMA
• Servicing the Debug Unit for the IPUs
![Page 80: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/80.jpg)
PUBLIC USE79 #NXPFTF
HOW TO PROGRAM
IT
![Page 81: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/81.jpg)
PUBLIC USE80 #NXPFTF
ISP GRAPH
![Page 82: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/82.jpg)
PUBLIC USE81 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Fast DMA
DDR
done
start
Frames are
processed on the
fly by the ISP
through the IPUs
MIPI-CSI
Camera
![Page 83: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/83.jpg)
PUBLIC USE82 #NXPFTF
Graph
Camera
Input
Buffer
1
Kernel
1
Buffer
2
Kernel
2…
![Page 84: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/84.jpg)
PUBLIC USE83 #NXPFTF
Graph
Camera
Input
Buffer
1
Kernel
1
Final
Buffer… Final
Kernel
![Page 85: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/85.jpg)
PUBLIC USE84 #NXPFTF
Graph
Kernel
1
Buffer
1
Buffer
2
![Page 86: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/86.jpg)
PUBLIC USE85 #NXPFTF
Graph
Kernel
1
Buffer
1
Kernel
2
…Buffer
2
Buffer
1
Kernel
3
![Page 87: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/87.jpg)
PUBLIC USE86 #NXPFTF
Graph
Kernel
1
Buffer
1
Kernel
2
![Page 88: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/88.jpg)
PUBLIC USE87 #NXPFTF
Graph
Kernel
1
Buffer
1
Kernel
3…
Kernel
2
![Page 89: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/89.jpg)
PUBLIC USE88 #NXPFTF
Graph
Kernel
1
Buffer
1
Kernel
3…
Kernel
2
Kernel
4…
![Page 90: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/90.jpg)
PUBLIC USE89 #NXPFTF
ISP Pipeline
Output buffer of the
kernel (function)
Function 0 requires one line in input
![Page 91: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/91.jpg)
PUBLIC USE90 #NXPFTF
ISP Pipeline
When enough lines are
available the next function
can be executed
Function 1 requires three lines in input
![Page 92: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/92.jpg)
PUBLIC USE91 #NXPFTF
ISP Pipeline
Pipelined processing
on multiple engines
![Page 93: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/93.jpg)
PUBLIC USE92 #NXPFTF
ISP Pipeline
The image can be
processed on the fly!
![Page 94: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/94.jpg)
PUBLIC USE93 #NXPFTF
GRAPH TOOL (DEMO)
![Page 95: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/95.jpg)
PUBLIC USE94 #NXPFTF
ISP KERNEL
![Page 96: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/96.jpg)
PUBLIC USE95 #NXPFTF
Example of 2:1 Scaling
Aliasing problemWhen simply copying
1 pixel over 2
feature of the StreamDMA
![Page 97: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/97.jpg)
PUBLIC USE96 #NXPFTF
Example of 2:1 Scaling
Aliased Anti-aliased
1 2 1
2 4 2
1 2 1
x 1/16
3x3 Gaussian filter
Low pass filter
![Page 98: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/98.jpg)
PUBLIC USE97 #NXPFTF
Example of 2:1 Scaling
1 2 1
2 4 2
1 2 1
x 1/16
1/16 1/8 1/16
1/8 1/4 1/8
1/16 1/8 1/16
1/16 equivalent to a right shift of 4
1/8 equivalent to a right shift of 3
1/4 equivalent to a right shift of 2
4 3 4
3 2 3
4 3 4
Shift Matrix:
![Page 99: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/99.jpg)
PUBLIC USE98 #NXPFTF
Example of 2:1 Scaling
start_scale:
mov confalu, (0 /*unsigned*/ |
(1<<1) /*saturate*/)
mov confaddt,(0 /*w*/ |
(0<<3) /*unsigned*/ |
(5<<5) /*shift*/ |
(0x40 << 9) /*
scale*/)
Code: IPU configuration
Confalu and Confaddt are core registers of the IPU
![Page 100: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/100.jpg)
PUBLIC USE99 #NXPFTF
Example of 2:1 Scaling
Pix
1,1
Pix
1,2
Pix
1,3
Pix
2,1
Pix
2,2
Pix
2,3
Pix
3,1
Pix
3,2
Pix
3,3
Pix
1,1
Pix
2,1
Pix
3,1
Pix
1,1
Pix
1,2
Pix
2,1
Pix
2,2
Pix
3,1
Pix
3,2
done d0,i
d0:
done d1,i
d1:
done d2,i
Code: Input initialisation Input matrix
InA
![Page 101: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/101.jpg)
PUBLIC USE100 #NXPFTF
Example of 2:1 Scaling1 0 1
0 0 0
1 0 1
Mask Matrix
d2:
mset 0b101000101
mov 4 // to w
mset 0b010101010
mov 3 // to w
mov w4,2
Code: Shifting matrix initialisation
![Page 102: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/102.jpg)
PUBLIC USE101 #NXPFTF
Example of 2:1 Scaling
Mask matrix
![Page 103: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/103.jpg)
PUBLIC USE102 #NXPFTF
Example of 2:1 Scaling1 0 1
0 0 0
1 0 1
Mask matrix
4 0 4
0 0 0
4 0 4
Working
matrix W
d2:
mset 0b101000101
mov 4
mset 0b010101010
mov 3
mov w4,2
Code: Shifting matrix initialisation
Matrix instruction
![Page 104: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/104.jpg)
PUBLIC USE103 #NXPFTF
Example of 2:1 Scaling
Working matrix W
![Page 105: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/105.jpg)
PUBLIC USE104 #NXPFTF
Example of 2:1 Scaling0 1 0
1 0 1
0 1 0
Mask matrix
4 3 4
3 0 3
4 3 4
d2:
mset 0b101000101
mov 4
mset 0b010101010
mov 3
mov w4,2
Code: Shifting matrix initialisation
Working
matrix W
![Page 106: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/106.jpg)
PUBLIC USE105 #NXPFTF
Example of 2:1 Scaling
4 3 4
3 2 3
4 3 4
d2:
mset 0b101000101
mov 4
mset 0b010101010
mov 3
mov w4,2
Code: Shifting matrix initialisation
Working
matrix WScalar instruction
(Mask matrix does not apply)
![Page 107: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/107.jpg)
PUBLIC USE106 #NXPFTF
Example of 2:1 Scaling
4 3 4
3 2 3
4 3 4
Shift matrix saved in
Working matrix WW
1 1 1
1 1 1
1 1 1
Mask matrix
swp // w -> ww
mset 0b111111111
Code: Shifting matrix initialisation
(important for future
instruction)
![Page 108: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/108.jpg)
PUBLIC USE107 #NXPFTF
Example of 2:1 Scaling
![Page 109: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/109.jpg)
PUBLIC USE108 #NXPFTF
Example of 2:1 Scaling
Load-right shift InA elements
according to the coefficient in WW
=> result saved in W
Add all the W elements together
and put it in output
+ Fetch new entry
loop:
lsr ina,ww
dout sum,odd,ixo
odd:
done loop,ix
end_scale::
Code: Main Loop
Fetch new entry without creating
any output: scaling of 2:1
Continue the loop
The kernel will restart from the beginning for each new line
![Page 110: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/110.jpg)
PUBLIC USE109 #NXPFTF
Kernel performances
0
50
100
150
200
250
300
350
400
450
500
available[MHz]
used[MHz]
2 MPixels @ 30
fps, dual exposure
![Page 111: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/111.jpg)
PUBLIC USE110 #NXPFTF
ISP Performances and Limitations
• ???
![Page 112: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/112.jpg)
PUBLIC USE111 #NXPFTF
SW
![Page 113: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/113.jpg)
PUBLIC USE112 #NXPFTF
ISP SW
• NXP provides:
−Sequencer code
−Graph tool + compiler
− ISP library
−Kernel examples
−Graph examples
• Programmer will create its own:
−Graph
−Kernels
![Page 114: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/114.jpg)
PUBLIC USE113 #NXPFTF
ISP Application
ISP
ISP Graph
IPU
Kernels
![Page 115: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/115.jpg)
PUBLIC USE114 #NXPFTF
ISP Application
ISP
Compiler
(AARCH64 –
Linux, SA)
ISP Graph
IPU
Kernels
IPU
Compiler
GDT
Compiler
![Page 116: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/116.jpg)
PUBLIC USE115 #NXPFTF
ISP Application
ARM Application for Linux or Standalone
ARM Code
main()
Device drivers
ISP librariesISP
Compiler
(AARCH64 –
Linux, SA)
ISP Graph
IPU
Kernels
IPU
Compiler
GDT
Compiler
![Page 117: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/117.jpg)
PUBLIC USE116 #NXPFTF
ISP Application
ARM Application for Linux or Standalone
ARM Code
main()
Device drivers
ISP librariesISP
Compiler
(AARCH64 –
Linux, SA)
ISP Graph
IPU
Kernels
IPU
Compiler
GDT
Compiler
Compiler
(AARCH64 –
Linux, SA)
Final APP
![Page 118: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/118.jpg)
PUBLIC USE117 #NXPFTF
CONCLUSION
![Page 119: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/119.jpg)
PUBLIC USE118 #NXPFTF
QUESTIONS
![Page 120: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/120.jpg)
PUBLIC USE119 #NXPFTF
DEMOS
![Page 121: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/121.jpg)
![Page 122: IMAGE SENSOR PROCESSOR (ISP)](https://reader031.fdocuments.us/reader031/viewer/2022013005/61cca892d628e77e267c21a7/html5/thumbnails/122.jpg)
PUBLIC USE121 #NXPFTF
ATTRIBUTION STATEMENT
NXP, the NXP logo, NXP SECURE CONNECTIONS FOR A SMARTER WORLD, CoolFlux, EMBRACE, GREENCHIP, HITAG, I2C BUS, ICODE, JCOP, LIFE VIBES, MIFARE, MIFARE Classic, MIFARE
DESFire, MIFARE Plus, MIFARE FleX, MANTIS, MIFARE ULTRALIGHT, MIFARE4MOBILE, MIGLO, NTAG, ROADLINK, SMARTLX, SMARTMX, STARPLUG, TOPFET, TrenchMOS, UCODE, Freescale,
the Freescale logo, AltiVec, C 5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C Ware, the Energy Efficient Solutions logo, Kinetis, Layerscape, MagniV, mobileGT, PEG, PowerQUICC, Processor Expert,
QorIQ, QorIQ Qonverge, Ready Play, SafeAssure, the SafeAssure logo, StarCore, Symphony, VortiQa, Vybrid, Airfast, BeeKit, BeeStack, CoreNet, Flexis, MXC, Platform in a Package, QUICC Engine,
SMARTMOS, Tower, TurboLink, and UMEMS are trademarks of NXP B.V. All other product or service names are the property of their respective owners. ARM, AMBA, ARM Powered, Artisan, Cortex,
Jazelle, Keil, SecurCore, Thumb, TrustZone, and μVision are registered trademarks of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. ARM7, ARM9, ARM11, big.LITTLE, CoreLink,
CoreSight, DesignStart, Mali, mbed, NEON, POP, Sensinode, Socrates, ULINK and Versatile are trademarks of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. All rights reserved. Oracle and
Java are registered trademarks of Oracle and/or its affiliates. The Power Architecture and Power.org word marks and the Power and Power.org logos and related marks are trademarks and service marks
licensed by Power.org. © 2015–2016 NXP B.V.