One dollar imaging with

FPGAONEONEONEONE

FPGAFPGAFPGAFPGA((((againagainagainagain))))

DOLLARDOLLARDOLLARDOLLARImaginImaginImaginImaginggggW i t hW i t hW i t hW i t h Mario Vigliar, PhD

DPControl Srl

SEDECO Imaging

GmbH

Chief Scientist

Dutch Machine Vision

Conference

May 17th, 2018

digital atelier

dpcontrol

is a company specialized in Video & Image

processing, able to develop IP, components and

high-performance processing systems with low-

power consumption.

dpcontroldpcontroldpcontroldpcontrol isisisis a TTM company.a TTM company.a TTM company.a TTM company.

mission

Machine vision specialist since 1994

About SEDECOStarted Imputer sales

Vision Components distributor

1994

1996

2000

2006

2008

Moved to own company building

Lensation distributor

Introduction MVNova MV script

2011Toshiba Teli and IMAC distributor

2017 Hikvision + Lucent distribution

2018DPControl joint R&D and distribution

Everything for successful machine vision implementation

SEDECO functions

• Supply quality products

• Application engineering

• Product development

• Introduce new technologies

• Knowledge distribution

Selection of A-type brands

Application Partner Program

Custom Optics, Electronics, Software

Mailings & Press releases

Techletter & Seminar/Training

Outline

Background

Algorithms

Architecture

Case study

Q&A

BackgroundWhy should we use FPGAs in our imaging projects?

Why not?

• Modern image processing techniques demand huge computational power

• Embedded CPUs are not able to achieve good performance on any task

• Low-power consumption is often a requirement

Some facts

• FPGAs are flexible• On-field reprogrammability• Many different I/O options, several package options

• Performance on FPGAs depends on algorithms’ inherent parallelism

• From multicore to manycore

Some facts

• Low-end FPGA can be extremely small

• BGA devices in 1.5 x 1.5 x 1 mm space• Down to chip-scale packages

• Low-end FPGA can be extremely cheap

• Down to 1 USD and below for large quantities

Small example

• H.264 integer inverse transform• In AVC video coding, for any 4 x 4 inverse quantized block X, the residual pixel block Y can be computed as in this formula

Small example

• The corresponding 2-D inverse integer transform can be computed as

by using Kronecker product

• Even if reasonably simple, CPUs are still not time and energy-efficient in doing that

Small example

• 2-D Inverse integer transform for H.264 can be as 850 LUTs small in FPGA, 27 clock turnaround

• Frequency is above 150MHz on small devices

• Enough for FullHD resolutions

• Mid-range LUT6 (or ALUT) devices can easily break the 350MHz barrier

• Ready for UltraHD-4K resolutions

ALGORITHMImage compression put on diet

Curse of resolution

• Different application domains all looking for same evolution

• Resolution increase is not equal to quality improvement…

• …but it is unavoidable

Curse of resolution

• Mega-pixel picture domain needs efficient compression techniques

• Efficiency is not only «high compression»

• Efficiency is not only «high PSNR ratio»

• Efficiency is not only «high encoding speed»

• Efficiency is ALL of these put together

Actual leading techniques• Derived from image-oriented compression

• JPEG / MJPEG• PNG

• Specifically designed for video compression

• ISO IEC H.264 / AVC / HEVC (H.265)• VP8/9• Microsoft VC1• Community driven Theora, Daala

Actual leading techniques

VC1

H.264

HEVC

MJPEG

VP8

VP9

Quality

Complexity

KG

ate

sM

Ga

tes

IoT Desiderata

• Minimum guaranteed compression ratio• Speed up to hundreds frames per second

• Low power consumption• Available now

No algorithm is available with these performance for IoT, if not based upon IP in a SoC

Medical imaging Desiderata• Minimum guaranteed compression ratio• Minimum image quality loss• Low power consumption• Available now

Same requirements of IoT, but more emphasis on quality (not a general rule, though)

Major issue

• Memory consumption• Memory bandwidth

• Up to hundreds MB/sec for transforms• Up to thousands MB/sec for motion estimation (ME)

• H.264 64x64 wide ME generated 9x109 memory access

Idea

• Focus on picking up a viable «reduction» of common techniques to derive a simple, quick and small image compressor

• Working on smaller macroblocks• Imply only integer arithmetic• Reduce memory to «zero» by avoiding ME

TLx

• Initially thought as texture compression algorithm

• Processing «x» rows of the image at a time

• Applying quantization procedures directly on the input pixels, not on transformed domain

• Smart interpolation function to reconstruct data after quantization

TLx

Raw input pixels

8/10/12 bits

Macroblockarrangement

In-MB scale/spaceconstruction

Quantization Delta coding

• Whole process can be done in pipeline

• Scale/space construction is builtupon variable function representation

• Adaptable to specific operative contexts

• Not fixed transform

TLx

Raw input pixels

8/10/12 bits

Macroblockarrangement

In-MB scale/spaceconstruction

Quantization Delta coding

See relevantpublishedinformation aboutscale/spacetransform

TLx

Raw input pixels

8/10/12 bits

Macroblockarrangement

In-MB scale/spaceconstruction

QuantizationDelta & entropycoding

• Quantization is upon kernel functionstoo

• Can be as simple as a linear model, or much more complicated

• The more complex the encoder, the more complex the decoder

IMPLEMENTATIONWhat we should really care about

DPC1-TL4/3/2

• What is DPC1-TLx?• Custom image compression coprocessor targeting small FPGAs and ASIC devices

• Less than 1.300 LUT4 in area

• 1 clock-per-pixel operation• After an initial delay of 24 clocks(less than 1.2 usec) 1 to 1 streaming is ensured

• Up to 90MPix/s operation on entry-level FPGAs

• 720p60 is 55 Mpix/s as comparison

DPC-TL2

• Only 2 rows of buffering

• Suitable for cable extenders

• Minimum ratio of 4:1

Uncompressed DPC1-TL2 4:1

DP

C-T

L2 C

lose

-up

(43

.4 d

B P

SN

R)

Image courtesy: Ph. G. Del Mastro – RAW 12MP Fuji X100 – 35mm F/2.0

DPC-TL2

• Bandwidth comparison between uncompressed and TL2-ized stream

Uncompressed

Mbit/s at

60 Hz

DPC1-TL2

Mbit/s at

60 Hz

VGA640 x 480

421.88 105.47

D1768 x 576

569.53 142.38

720P1280 x 720

1265.63 316.41

1080P1920 x 1080

2487.66 711.91

DPC-TL3

• TL3 uses 3 lines of buffers

• 4 by 3 sized macroblocks

• 5.14 : 1 minimum ratio

Uncompressed

Mbit/s at

60 Hz

DPC1-TL2

Mbit/s at

60 Hz

VGA640 x 480

421.88 82.03

D1768 x 576

569.53 110.74

720P1280 x 720

1265.63 246.09

1080P1920 x 1080

2487.66 553.71

DPC-TL2 vs. DPC-TL3• Same LUT footprint, 1.300 LUT4• Different buffer sizes

• Decoder is equally small (less than 800 LUT4)

• Fully pipelined, input from 24 bit RGB stream

720p60Compression

ratioArea (LUT4) Max freq. Line buffers Aux buffers

DPC1-TL2 4:1 1390 78.8 MHz 20736 bits15160 bits

(optional)

DPC1-TL3 5.14:1 1398 74.2 MHz 41472 bits 15160 bits

DPC-TL2 area report• Target device Lattice Lattice Lattice Lattice XP2XP2XP2XP2----5555

• Fully qualified AEC-Q100

• 166 Kbits internal EBR memories

• 12 DSP blocks (MUL18x18)

• 2 Onboard PLLs• 172 Maximum available I/O

• Channel encoder in extra 120 LUT4s

• 8b/10b encoder• Sync word management

DPC-TL2 area report

• XP2-5 floorplan for 720p60 encoder

Use caseWhere did you get the headline? Show me my dollar!

Endoscopy capsule

• Developed by Scuola SuperioreSant’Anna di Pisa

• Premium Premium Premium Premium technologiestechnologiestechnologiestechnologies for infor infor infor in----body body body body navigationnavigationnavigationnavigation

• RF RF RF RF communicationcommunicationcommunicationcommunication, , , , lesslesslessless thanthanthanthan 1Mbps 1Mbps 1Mbps 1Mbps speedspeedspeedspeed

• 640 x 640 10640 x 640 10640 x 640 10640 x 640 10----bit Bayer inputbit Bayer inputbit Bayer inputbit Bayer input

• DPCDPCDPCDPC----TL4 TL4 TL4 TL4 compressorcompressorcompressorcompressor and and and and sensorsensorsensorsensorcapturecapturecapturecapture logiclogiclogiclogic runningrunningrunningrunning on a single on a single on a single on a single Lattice iCE40Lattice iCE40Lattice iCE40Lattice iCE40----4K4K4K4K

Endoscopy capsuleRossella Fontana, Francesca Mulana, Carmen Cavallotti, Giuseppe Tortora, Mario Vigliar, Monica Vatteroni, Arianna Menciassi

An Innovative Wireless Endoscopic Capsule With An Innovative Wireless Endoscopic Capsule With An Innovative Wireless Endoscopic Capsule With An Innovative Wireless Endoscopic Capsule With Spherical Shape.Spherical Shape.Spherical Shape.Spherical Shape.IEEE Trans. Biomed. Circuits and Systems 11(1): 143-152 (2017)

Endoscopy capsule

• Extra small footprint

• Battery operated

• Onboard image processing and compression

• Sub-GHz RF link

Endoscopy capsule

• Lattice iCE40 MobileFPGA

• 40nm FPGA• Low power consumption

• No heating sinks, fans

• Multiple bitstreamsfeatures

• OTP feature

Endoscopy capsule• iCE40-4K LP floorplanand power report

TL4 on endoscopy images

--- Removed due to privacy issues ---

Q&ADo not shoot the FPGA developer

iCeVisionThe new standard in small FPGA development

iCEVision Board

iCEVision Board• Enables designers to evaluate

key connectivity features of

the iCE40 UltraPlus FPGA.

• Rapid prototyping of user

functions

• The exposed I/Os allow users

to quickly implement solutions

• Compatible with the most

common camera interfaces,

ArduCam CSI and PMOD.

• Effortless connectivity and

simple programming

iCEVision BoardFeaturesFeaturesFeaturesFeatures

• iCE40 UltraPlus (iCE40UP5K) device in SG48 package

• 8Mb Spi programmable Flash

• 1Mb SRAM 20MHz

• 3 LED users for RGB LED applications

• PMOD connector

• 2 ArduCam m/s connectors allows multiple boards connection

• CSI 24pins Connector

• 50mmx50mm size Board

iCEVision Board

ToolchainsToolchainsToolchainsToolchains

• LLLLatticeatticeatticeattice RadiantRadiantRadiantRadiant

• LLLLatticeatticeatticeatticeiCECube2iCECube2iCECube2iCECube2

• iCEStormiCEStormiCEStormiCEStorm OSSOSSOSSOSS

OSS softOSS softOSS softOSS soft----core core core core availableavailableavailableavailable

• RiscRiscRiscRisc----V V V V (picorv32, (picorv32, (picorv32, (picorv32, VexRiscVVexRiscVVexRiscVVexRiscV))))

iCEVision Board

SupportSupportSupportSupport

www.dpcontrol.comwww.dpcontrol.comwww.dpcontrol.comwww.dpcontrol.com

gitlab.dpcontrol.com/gitlab.dpcontrol.com/gitlab.dpcontrol.com/gitlab.dpcontrol.com/icicicicevisionevisionevisionevision

BuyBuyBuyBuy

Now available at Now available at Now available at Now available at Amazon.(Amazon.(Amazon.(Amazon.(it,de,frit,de,frit,de,frit,de,fr,uk,uk,uk,uk))))

DAXPYVPU for one-Watt class of machine vision systems

DAXPYA new deal in A new deal in A new deal in A new deal in smart camera smart camera smart camera smart camera systems for systems for systems for systems for industrial industrial industrial industrial usageusageusageusage

• SmallSmallSmallSmall

• 100 100 100 100 GflopGflopGflopGflop/s /s /s /s range of range of range of range of operation operation operation operation

• EmbeddableEmbeddableEmbeddableEmbeddable

• LowLowLowLow----powerpowerpowerpower

• Ready for deep learning inference

DAXPYFeaturesFeaturesFeaturesFeatures

• MA2450 VPU• 2 x 2 MIPI lanes input

• 12 SHAVE VLIW @600MHz

• USB3, GPIO, uSDonboard

• 50mm x 35mm size Board

DAXPY• Supportedsensors

• From 1.3 to 10MP• Global or rollingshutter

• LVDS, SubLVDS, DVP converted to MIPI 2/4L

• Standard industrial I/F

• USB3 w/ USB3 Vision

• Supporting TLx-TLB for RAW-likeoutput up to 3.2 Gbps

DAXPY• Programmable

• daxpy block tool– web based GUI to build up yourown image processing pipeline, eventbases

• daxpy cloud –server side vision tools, deploy on Amazon F1 or Google Cloud Services

• daxpy AI – DNN engine on-board

Thanks !For any further information, please contact us

mvigliar@dpcontrol.com

www.dpcontrol.com