PUBLIC USE
MARIE-ANNE LE MENN
ADAS MICROPROCESSOR APPLICATION ENGINEER
FTF-AUT-N1807
MAY 19, 2016
FTF-AUT-N1807
IMAGE SENSOR PROCESSOR
(ISP)
PUBLIC USE1 #NXPFTF PUBLIC USE1 #NXPFTF
AGENDA
• The purpose of the ISP
−S32v234
−Camera interface
− Image sensor Processing
− Image pipeline
• Its architecture
• How to program it
PUBLIC USE2 #NXPFTF
THE PURPOSE OF
THE ISP
PUBLIC USE3 #NXPFTF
S32V234:
ADAS PROCESSOR
PUBLIC USE4 #NXPFTF
S32v234
• ADAS processor:
−Sensor fusion
−Front view camera
−Surround-view
PUBLIC USE5 #NXPFTF
PUBLIC USE6 #NXPFTF
PUBLIC USE7 #NXPFTF
PUBLIC USE8 #NXPFTF
PUBLIC USE9 #NXPFTF
S32v234: Block Diagram
PUBLIC USE10 #NXPFTF
S32v234: Block Diagram
PUBLIC USE11 #NXPFTF
S32v234: Block Diagram
PUBLIC USE12 #NXPFTF
S32v234: Block Diagram
PUBLIC USE13 #NXPFTF
S32v234: Block Diagram
PUBLIC USE14 #NXPFTF
S32v234: Vision Application
Image Pre-Processing
• ISP
• Multi-streams connectivity
Features Extraction
• Corners
• Edges
• Intensity gradients
• Shapes
Features Classification& Prediction
• SVM
• Adaboost
• K-nearest neighbor
Multi Image Processing
• Tracking, motion estimation
• Optical flow & disparity
• Stitching
Object Recognition & Fusion
• Object recognition/pedestrian
• Augmented reality
• Face detection
GFX Overlay & Video Distribution
• Safe fusion
• Graphic overlay & display
• 2D vs. 3D projection
CPU Platform
Cortex - A53
32kB I-cache
2 way
NEON
32kB D-cache
4 way
Cortex - A53
32kB I-cache
2 way
NEON
32kB D-cache
4 way
Cortex - A53
32kB I-cache
2 way
NEON
32kB D-cache
4 way
Cortex - A53
32kB I-cache
2 way
NEON
32kB D-cache
4 way
L2 Cache – 256kB 16 ways SCU
Dual Camera Interfaces
2 x MIPI CSI2
Image Signal Processing
HDR
Color Conversion
Tone Mapping
Parallel 20 bit
Image Cognition Proc.
L-mem
Sequencer
L-mem
32 CU 32 CU
DMA
APEX2 CL
Image Cognition Proc.
L-mem
Sequencer
L-mem
32 CU 32 CU
DMA
G2-APEX-642
Image Proc. PlatformGfx & Display
GPU OpenGL ES 3.0
DCU 18/24 bits RGB
Video Codec H.264
8-12 bits Encoder
8bit Decoder
Video & Display Platform
High bandwidth operations
Scalable MIMD local memory
Soft ISP
Scalable RISC – data fusion
SIMD co-processor - neon
Memory hierarchy and coherency
Graphic
Video codec
Smart display
PUBLIC USE15 #NXPFTF
CAMERA
INTERFACES
PUBLIC USE16 #NXPFTF
Camera Interface: MIPI-CSI
• MIPI-CSI2 standard
• 4 lanes up to 1.5Gbps each
• 4 virtual channels
• In theory: up to 6Gbps for 1 MIPI
interface
• In reality:
− No camera with such bandwidth
− LVDS limitation
S32v234 has 2 MIPI-CSI interfaces
PUBLIC USE17 #NXPFTF
Camera Interface: MIPI-CSI
• Typical sensor resolution:
−1280x800 @ 30/60 fps
−1920x1080 @ 30/60 fps
• Examples of camera:
−Sony: IMX224MQV
−Omnivision: OV10640
PUBLIC USE18 #NXPFTF
Camera Interface: Multiple Cameras
PUBLIC USE19 #NXPFTF
Camera Interface: Ethernet
• 1Gbps interface
• Easy synchronisation via AVB
• Embedded decoder for up to 4 video streams
PUBLIC USE20 #NXPFTF
Camera Interface: Ethernet
PUBLIC USE21 #NXPFTF
Camera Interface: Parallel
• Up to 100MHz pixel clock
• Pin out options:
−2 interfaces x16-bit
−1 interface x20-bit + 1 Interface x12-bit
Signals:
• Data [0:19]
• Vsync, Hsync, PixClk
• I2C
• Power-down
PUBLIC USE22 #NXPFTF
IMAGE SENSOR
PROCESSING
PUBLIC USE23 #NXPFTF
Raw Camera Image
PUBLIC USE24 #NXPFTF
Raw Camera Image
Why does it look greenish?
PUBLIC USE25 #NXPFTF
Bayern Pattern
PUBLIC USE26 #NXPFTF
White Balancing
Relative intensity has been normalized for each temperature (in Kelvins).
PUBLIC USE27 #NXPFTF
White Balancing
PUBLIC USE28 #NXPFTF
Raw Camera Image
Why is there 2 frames?
PUBLIC USE29 #NXPFTF
HDR: High Dynamic Range
PUBLIC USE30 #NXPFTF
HDR: High Dynamic Range
High
exposure
Low
exposure
Combination of
the 2 exposures
PUBLIC USE31 #NXPFTF
Image Sensor Processing
• White balancing
• De-bayering
• HDR
• Black level
• Noise filtering
• Vignetting
• Colour conversion: YUV, Gray scale
PUBLIC USE32 #NXPFTF
ISP Solutions
MCU
or
Bare Sensor
+
MCU
BOM Cost
ISP
Heat Problem
+Integrated ISP
extra: $3-4
Bare Sensor
+
Companion
+
MCU
ISP
Companion chip:
$3-4
S32v234: integrated solution
Sensor
PUBLIC USE33 #NXPFTF
Advantages
• Manage power on sensor
• Optimize BOM and board
• Smaller size through integration
Save Money
PUBLIC USE34 #NXPFTF
Image Pipeline
PUBLIC USE35 #NXPFTF
Image Pipeline
PUBLIC USE36 #NXPFTF
Image Pipeline
PUBLIC USE37 #NXPFTF
Image Pipeline
PUBLIC USE38 #NXPFTF
When to Use the ISP?
• Ethernet: ISP is already integrated with the camera
• MIPI-CSI & parallel interface:
−External ISP required for more than 2Mpix
− Internal ISP will save money for mono or stereo cameras
PUBLIC USE39 #NXPFTF
ITS ARCHITECTURE
PUBLIC USE40 #NXPFTF
WHAT IS A KERNEL?
PUBLIC USE41 #NXPFTF
Kernel
• Do not mistake:
−Linux kernel
PUBLIC USE42 #NXPFTF
Kernel
• Do not mistake:
−Linux kernel
− Image processing kernel
PUBLIC USE43 #NXPFTF
Kernel
• Do not mistake:
−Linux kernel
− Image processing kernel
−APEX kernel
− ISP kernel• Both can execute image processing kernel but not only
• Different:
• Programing method
• Capabilities
• Engines
PUBLIC USE44 #NXPFTF
ISP Kernel
• Assembly code
• Running on the Image Processing Units (IPU) of the ISP
• Each IPU can run a different kernel
• Examples:
−De-bayering
−Noise filtering
−Colour conversion
−…
PUBLIC USE45 #NXPFTF
ISP Kernel
• A lot of image processing steps (kernels) require multiple lines:
De-bayering Sobel filtering
IPUs can fetch multiple lines at the same time
PUBLIC USE46 #NXPFTF
ISP SUBSYSTEM
PUBLIC USE47 #NXPFTF
ISP Sub-system
IPUS
=
Scalar Image Processing Unit
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
PUBLIC USE48 #NXPFTF
ISP Sub-system
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…IPUV
=
Vector Image Processing Unit
PUBLIC USE49 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Each input/output
can fetch/output
multiple data at the
same time
PUBLIC USE50 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
PUBLIC USE51 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
M0
Core
PUBLIC USE52 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Data
RAM
M0
Core
Instr.
RAM
PUBLIC USE53 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
Crypto
protected
PUBLIC USE54 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Configuration and
synchronization signals
PUBLIC USE55 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Fast DMA
DDR
PUBLIC USE56 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Fast DMA
DDRConfiguration and
synchronization signals
done
start
PUBLIC USE57 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
Instr.
RAM
M0
Core
done
start
Programmable
Engines
Fast DMA
DDR
done
start
PUBLIC USE58 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Fast DMA
DDR
done
start
Frames are
processed on the
fly by the ISP
through the IPUs
MIPI-CSI
Camera
PUBLIC USE59 #NXPFTF
IPU
PUBLIC USE60 #NXPFTF
Scalar Engine (IPUS)
PUBLIC USE61 #NXPFTF
Scalar Engine (IPUS)
• Input window: 3x3 or 9x1
• Up to 3 Inputs:• Ex: R, G, B
PUBLIC USE62 #NXPFTF
Scalar Engine (IPUS)
• Instruction memory
• Decoding unit
• No data memory (no load/store), only registers
PUBLIC USE63 #NXPFTF
Scalar Engine (IPUS)
Working matrixMatrix ALU (3x3 -> 3x3)
PUBLIC USE64 #NXPFTF
Scalar Engine (IPUS)
Mask matrix: To select
on which elements of
the matrix the operation
or accelerator should be
applied to
3X3 Accelerators:
• Sum, scaled, clipped
• Max, median, min
Other accelerators:
• Histogram
• PRNG
PUBLIC USE65 #NXPFTF
Scalar Engine (IPUS)
Up to 2 output
PUBLIC USE66 #NXPFTF
Vector Engine (IPUV)
5x5 input
windows
4-way SIMD ALUHandles operation with:
• Horizontal vectors
• Vertical vectors
• Scalars
PUBLIC USE67 #NXPFTF
IPU Engines
• 8 x IPUS & 4 x IPUV
• 500MHz per Engine: 6000MHz total
• Pixels are 16-bit fixed point data in signed or unsigned mode
• Input and output managed by the StreamDMA engines
PUBLIC USE68 #NXPFTF
Input Matrix: IPUS
In2 In1 In0
In4 In2 In3
In7 In6 In5
In8 In7 In6 In5 In4 In3 In2 In1 In0
3x3
9x1
PUBLIC USE69 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
Input image
Input matrix
PUBLIC USE70 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
1st Input
PUBLIC USE71 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
2nd Input
PUBLIC USE72 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
… 5th input
PUBLIC USE73 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5 Pix 6 Pix 7 Pix 8
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14 Pix 15 Pix 16 Pix 17
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23 Pix 24 Pix 25 Pix 26
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32 Pix 33 Pix 34 Pix 35
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41 Pix 42 Pix 43 Pix 44
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50 Pix 51 Pix 52 Pix 53
1st Input of the 2nd line
PUBLIC USE74 #NXPFTF
Input Matrix: IPUS
Pix 0 Pix 1 Pix 2 Pix 3 Pix 4 Pix 5
Pix 9 Pix 10 Pix 11 Pix 12 Pix 13 Pix 14
Pix 18 Pix 19 Pix 20 Pix 21 Pix 22 Pix 23
Pix 27 Pix 28 Pix 29 Pix 30 Pix 31 Pix 32
Pix 36 Pix 37 Pix 38 Pix 39 Pix 40 Pix 41
Pix 45 Pix 46 Pix 47 Pix 48 Pix 49 Pix 50
3x3 mode 1st Input
PUBLIC USE75 #NXPFTF
HW Replacement Modes for StreamDMA Engines
StreamDMA has a lot of
additional features:
• Scaling
• Multiple data format
support
PUBLIC USE76 #NXPFTF
Input Matrix: IPUV
I
n
2
I
n
1
I
n
0
I
n
4
I
n
2
I
n
3
I
n
7
I
n
6
I
n
5
????????
PUBLIC USE77 #NXPFTF
SEQUENCER
PUBLIC USE78 #NXPFTF
Sequencer: Block Diagram
• Mapped into Host address space
• Peripherals can also be controlled by Host
CPU: IPUs and FastDMA
• Servicing the Debug Unit for the IPUs
PUBLIC USE79 #NXPFTF
HOW TO PROGRAM
IT
PUBLIC USE80 #NXPFTF
ISP GRAPH
PUBLIC USE81 #NXPFTF
ISP Sub-system
SRAM
SR
AM
contr
olle
r
IPU Modules
IPUS_0
IPUS_1
IPUS_7
…
IPUV_0
IPUV_3
…
Sequencer
Kernel
RAM
Data
RAM
M0
Core
Instr.
RAM
done
start
Fast DMA
DDR
done
start
Frames are
processed on the
fly by the ISP
through the IPUs
MIPI-CSI
Camera
PUBLIC USE82 #NXPFTF
Graph
Camera
Input
Buffer
1
Kernel
1
Buffer
2
Kernel
2…
PUBLIC USE83 #NXPFTF
Graph
Camera
Input
Buffer
1
Kernel
1
Final
Buffer… Final
Kernel
PUBLIC USE84 #NXPFTF
Graph
Kernel
1
Buffer
1
Buffer
2
PUBLIC USE85 #NXPFTF
Graph
Kernel
1
Buffer
1
Kernel
2
…Buffer
2
Buffer
1
Kernel
3
PUBLIC USE86 #NXPFTF
Graph
Kernel
1
Buffer
1
Kernel
2
PUBLIC USE87 #NXPFTF
Graph
Kernel
1
Buffer
1
Kernel
3…
Kernel
2
PUBLIC USE88 #NXPFTF
Graph
Kernel
1
Buffer
1
Kernel
3…
Kernel
2
Kernel
4…
PUBLIC USE89 #NXPFTF
ISP Pipeline
Output buffer of the
kernel (function)
Function 0 requires one line in input
PUBLIC USE90 #NXPFTF
ISP Pipeline
When enough lines are
available the next function
can be executed
Function 1 requires three lines in input
PUBLIC USE91 #NXPFTF
ISP Pipeline
Pipelined processing
on multiple engines
PUBLIC USE92 #NXPFTF
ISP Pipeline
The image can be
processed on the fly!
PUBLIC USE93 #NXPFTF
GRAPH TOOL (DEMO)
PUBLIC USE94 #NXPFTF
ISP KERNEL
PUBLIC USE95 #NXPFTF
Example of 2:1 Scaling
Aliasing problemWhen simply copying
1 pixel over 2
feature of the StreamDMA
PUBLIC USE96 #NXPFTF
Example of 2:1 Scaling
Aliased Anti-aliased
1 2 1
2 4 2
1 2 1
x 1/16
3x3 Gaussian filter
Low pass filter
PUBLIC USE97 #NXPFTF
Example of 2:1 Scaling
1 2 1
2 4 2
1 2 1
x 1/16
1/16 1/8 1/16
1/8 1/4 1/8
1/16 1/8 1/16
1/16 equivalent to a right shift of 4
1/8 equivalent to a right shift of 3
1/4 equivalent to a right shift of 2
4 3 4
3 2 3
4 3 4
Shift Matrix:
PUBLIC USE98 #NXPFTF
Example of 2:1 Scaling
start_scale:
mov confalu, (0 /*unsigned*/ |
(1<<1) /*saturate*/)
mov confaddt,(0 /*w*/ |
(0<<3) /*unsigned*/ |
(5<<5) /*shift*/ |
(0x40 << 9) /*
scale*/)
Code: IPU configuration
Confalu and Confaddt are core registers of the IPU
PUBLIC USE99 #NXPFTF
Example of 2:1 Scaling
Pix
1,1
Pix
1,2
Pix
1,3
Pix
2,1
Pix
2,2
Pix
2,3
Pix
3,1
Pix
3,2
Pix
3,3
Pix
1,1
Pix
2,1
Pix
3,1
Pix
1,1
Pix
1,2
Pix
2,1
Pix
2,2
Pix
3,1
Pix
3,2
done d0,i
d0:
done d1,i
d1:
done d2,i
Code: Input initialisation Input matrix
InA
PUBLIC USE100 #NXPFTF
Example of 2:1 Scaling1 0 1
0 0 0
1 0 1
Mask Matrix
d2:
mset 0b101000101
mov 4 // to w
mset 0b010101010
mov 3 // to w
mov w4,2
Code: Shifting matrix initialisation
PUBLIC USE101 #NXPFTF
Example of 2:1 Scaling
Mask matrix
PUBLIC USE102 #NXPFTF
Example of 2:1 Scaling1 0 1
0 0 0
1 0 1
Mask matrix
4 0 4
0 0 0
4 0 4
Working
matrix W
d2:
mset 0b101000101
mov 4
mset 0b010101010
mov 3
mov w4,2
Code: Shifting matrix initialisation
Matrix instruction
PUBLIC USE103 #NXPFTF
Example of 2:1 Scaling
Working matrix W
PUBLIC USE104 #NXPFTF
Example of 2:1 Scaling0 1 0
1 0 1
0 1 0
Mask matrix
4 3 4
3 0 3
4 3 4
d2:
mset 0b101000101
mov 4
mset 0b010101010
mov 3
mov w4,2
Code: Shifting matrix initialisation
Working
matrix W
PUBLIC USE105 #NXPFTF
Example of 2:1 Scaling
4 3 4
3 2 3
4 3 4
d2:
mset 0b101000101
mov 4
mset 0b010101010
mov 3
mov w4,2
Code: Shifting matrix initialisation
Working
matrix WScalar instruction
(Mask matrix does not apply)
PUBLIC USE106 #NXPFTF
Example of 2:1 Scaling
4 3 4
3 2 3
4 3 4
Shift matrix saved in
Working matrix WW
1 1 1
1 1 1
1 1 1
Mask matrix
swp // w -> ww
mset 0b111111111
Code: Shifting matrix initialisation
(important for future
instruction)
PUBLIC USE107 #NXPFTF
Example of 2:1 Scaling
PUBLIC USE108 #NXPFTF
Example of 2:1 Scaling
Load-right shift InA elements
according to the coefficient in WW
=> result saved in W
Add all the W elements together
and put it in output
+ Fetch new entry
loop:
lsr ina,ww
dout sum,odd,ixo
odd:
done loop,ix
end_scale::
Code: Main Loop
Fetch new entry without creating
any output: scaling of 2:1
Continue the loop
The kernel will restart from the beginning for each new line
PUBLIC USE109 #NXPFTF
Kernel performances
0
50
100
150
200
250
300
350
400
450
500
available[MHz]
used[MHz]
2 MPixels @ 30
fps, dual exposure
PUBLIC USE110 #NXPFTF
ISP Performances and Limitations
• ???
PUBLIC USE111 #NXPFTF
SW
PUBLIC USE112 #NXPFTF
ISP SW
• NXP provides:
−Sequencer code
−Graph tool + compiler
− ISP library
−Kernel examples
−Graph examples
• Programmer will create its own:
−Graph
−Kernels
PUBLIC USE113 #NXPFTF
ISP Application
ISP
ISP Graph
IPU
Kernels
PUBLIC USE114 #NXPFTF
ISP Application
ISP
Compiler
(AARCH64 –
Linux, SA)
ISP Graph
IPU
Kernels
IPU
Compiler
GDT
Compiler
PUBLIC USE115 #NXPFTF
ISP Application
ARM Application for Linux or Standalone
ARM Code
main()
Device drivers
ISP librariesISP
Compiler
(AARCH64 –
Linux, SA)
ISP Graph
IPU
Kernels
IPU
Compiler
GDT
Compiler
PUBLIC USE116 #NXPFTF
ISP Application
ARM Application for Linux or Standalone
ARM Code
main()
Device drivers
ISP librariesISP
Compiler
(AARCH64 –
Linux, SA)
ISP Graph
IPU
Kernels
IPU
Compiler
GDT
Compiler
Compiler
(AARCH64 –
Linux, SA)
Final APP
PUBLIC USE117 #NXPFTF
CONCLUSION
PUBLIC USE118 #NXPFTF
QUESTIONS
PUBLIC USE119 #NXPFTF
DEMOS
PUBLIC USE121 #NXPFTF
ATTRIBUTION STATEMENT
NXP, the NXP logo, NXP SECURE CONNECTIONS FOR A SMARTER WORLD, CoolFlux, EMBRACE, GREENCHIP, HITAG, I2C BUS, ICODE, JCOP, LIFE VIBES, MIFARE, MIFARE Classic, MIFARE
DESFire, MIFARE Plus, MIFARE FleX, MANTIS, MIFARE ULTRALIGHT, MIFARE4MOBILE, MIGLO, NTAG, ROADLINK, SMARTLX, SMARTMX, STARPLUG, TOPFET, TrenchMOS, UCODE, Freescale,
the Freescale logo, AltiVec, C 5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C Ware, the Energy Efficient Solutions logo, Kinetis, Layerscape, MagniV, mobileGT, PEG, PowerQUICC, Processor Expert,
QorIQ, QorIQ Qonverge, Ready Play, SafeAssure, the SafeAssure logo, StarCore, Symphony, VortiQa, Vybrid, Airfast, BeeKit, BeeStack, CoreNet, Flexis, MXC, Platform in a Package, QUICC Engine,
SMARTMOS, Tower, TurboLink, and UMEMS are trademarks of NXP B.V. All other product or service names are the property of their respective owners. ARM, AMBA, ARM Powered, Artisan, Cortex,
Jazelle, Keil, SecurCore, Thumb, TrustZone, and μVision are registered trademarks of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. ARM7, ARM9, ARM11, big.LITTLE, CoreLink,
CoreSight, DesignStart, Mali, mbed, NEON, POP, Sensinode, Socrates, ULINK and Versatile are trademarks of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. All rights reserved. Oracle and
Java are registered trademarks of Oracle and/or its affiliates. The Power Architecture and Power.org word marks and the Power and Power.org logos and related marks are trademarks and service marks
licensed by Power.org. © 2015–2016 NXP B.V.
Top Related