Feature Insect-Vision Inspired Collision Warning Vision...

6 IEEE CIRCUITS AND SYSTEMS MAGAZINE 1531-636X/08/$25.00©2008 IEEE SECOND QUARTER 2008

1. Introduction

Making cars safer is one of the major challenges ofthe automotive industry. Future cars might beideally expected to “behave” following laws sim-

ilar to those formulated by Isaac Assimov for robots.Namely: (1) a car should not injure a human being, or,through inaction, allow a human being damage; (2) a carmust obey orders given by human beings, except thatthese orders are in conflict with the first law, and; (3) acar must protect its own existence as long as such pro-tection does not conflict with neither the first nor the

Gustavo Liñán-Cembrano, Luis Carranza,Claire Rind, Akos Zarandy, Martti Soininen,and Angel Rodríguez-Vázquez

Insect-Vision Inspired Collision WarningVision Processor for Automobiles

Abstract

Feature

Digital Object Identifier 10.1109/MCAS.2008.916097

Vision is expected to play important roles for car safety enhancement. Imaging systems can beused to enlarging the vision field of the driver. For instance capturing and displaying views of hid-den areas around the car which the driver can analyze for safer decision-making. Vision systemsgo a step further. They can autonomously analyze the visual information, identify dangerous sit-uations and prompt the delivery of warning signals. For instance in case of road lane departure,if an overtaking car is in the blind spot, if an object is approaching within collision course, etc.Processing capabilities are also needed for applications viewing the car interior such as “intelli-gent airbag systems” that base deployment decisions on passenger features.

On-line processing of visual information for car safety involves multiple sensors and views,huge amount of data per view and large frame rates. The associated computational load may beprohibitive for conventional processing architectures. Dedicated systems with embedded local pro-cessing capabilities may be needed to confront the challenges. This paper describes a dedicatedsensory-processing architecture for collision warning which is inspired by insect vision. Particu-larly, the paper relies on the exploitation of the knowledge about the behavior of Locusta Migra-toria to develop dedicated chips and systems which are integrated into model cars as well as intoa commercial car (Volvo XC90) and tested to deliver collision warnings in real traffic scenarios.

Authorized licensed use limited to: Newcastle University. Downloaded on March 16,2010 at 10:30:53 EDT from IEEE Xplore. Restrictions apply.

second law. The challenge of increased safety requires dif-ferent innovations among which the enhancement of carsmartness is of primary importance. A smart car shouldprimarily be able of sensing its state: engine parameters,cinematic parameters, passengers typology and move-ments, position relative to the road, obstacles, … But justsensing is not enough; smart cars must also be capable ofprocessing the acquired data to either giving informationto the driver or eventually taking control [1], [2].

Out of the many sensor types, it is generally understoodthat optical sensors (devices to acquire images) and visionsystems (systems to acquire, process and extract the infor-mation from image flows) will be key ingredients in passiveand active safety systems of future cars [3], [4]. Safety-sys-tem roadmaps envisage the use of many vision devices percar; some of them looking at outside the vehicle and otherslooking at the inside. Inside vision sensors will be used tomonitor sizes, positions and movementsof passengers and hence to control airbagdeployment in case of crash [5]. They willalso be employed to analyze the driverstatus; for instance, to detect drowsiness[6]. Furthermore, car architects may takeadvantage of the availability of visiondevices to realize not-safety-related taskssuch as human-computer-interface. Re-garding outside sensors they will serve,among other tasks, to keep the car withinroad lanes, to calculate proper speed anddistance of the car relative to others andto detect either objects or cars approach-ing on a collision course.

Figure 1 compiles potential applica-tions of vision for cars. Vision will be usedin combination with other sensor sys-tems (radar [7], lidar, etc.) to support thedevelopment of future advanced driverassistance systems (ADAS) capable of tak-ing full control of the car engine, brakeand steering in order to avoid accidents[8]. However the endeavor to incorporatevision to cars is demanding. Apart fromthe typically long automotive designcycles (several years) the development ofvision for automobiles is slowed down bythe challenges which confront the analy-sis of rapidly changing image flows in real-time. On the one hand these challengesare due to the huge amount of data which

images contain. On the other hand, they are motivated bythe complex ways in which the safety-relevant informationis encoded into spatial (intra-frame variations) and tempo-ral (inter-frame variations) of the image sequences. Tobegin with, large computational and memory resourcesmay be needed for real-time completion of even concep-tually simple tasks. To overcome these problems newarchitectural solutions are advisable.

This paper presents a bio-inspired vision system fordetection of collisions threats in automotive applications[10]. The rationale for adopting a bio-inspired architec-tural approach relies on the observation that the visualdetection of motion is essential to survival in many ani-mal species. Motion tells animals about predators andpreys, about its own movements and that of othersaround it. Particularly, Locusta Migratoria (from now onLocust) [11] is exceptionally good at detecting and

7SECOND QUARTER 2008 IEEE CIRCUITS AND SYSTEMS MAGAZINE

Rear Cameras Side Cameras

Fron

t Cam

eras

Side Cameras

Blin

d-S

pot

Det

ectio

n

Col

lisio

nM

itiga

tion

Driv

erA

ssis

tanc

ePe

dest

rian

Prot

ectio

n

Traf

fic-S

ign

Assi

stan

ce

Lane

-Cha

nge

Assi

stan

ce

Col

lisio

n Av

oida

nce

Col

lisio

n Av

oida

nce

Lane

Dep

artu

re

Park

ing

Aid

sH

uman

-Com

pute

r

Inte

rface

Occ

upan

tM

onito

ring

Driv

er S

tatu

s

Sm

art

Sea

t Bel

ts

Smar

t Air

Bag

Outside Vision

Inside Vision

Figure 1. Some envisaged applications of vision for cars.

Gustavo Liñán-Cembrano and Luis Carranza are with the Instituto de Microelectrónica, Seville, Spain. Claire Rind is with the University of New-castle Upon Tyne, U.K. Akos Zarandy is with the Computer and Automation Research Institute, Budapest, Hungary. Martti Soinininen is with theVolvo Car Corp., Göteborg, Sweden. Angel Rodríguez-Våzquez ([email protected]) is with Innovaciones Microelectrónicas, Seville, Spain.


reacting to the visual motion of an object approaching ona collision course. Indeed, some of the largest neurons inthe Locust brain are dedicated to this task. We exploitthe knowledge of the neural circuitry underlying thisability to construct artificial vision systems capable todeliver collision-warning alarms for cars.

Besides new paradigms, building efficient vision sys-tems for automotive applications requires careful choiceof the hardware architecture. Two basic possibilities arise:

■ Using separate sensors and processors, relyingeither on CCD or on CMOS for the sensors [4].

■ Building dedicated CMOS solutions which mergethe sensing and the processing parts [9].

The main advantage of the second alternative is thatdedicated readout circuitry, dedicated error correction cir-cuitry and dedicated processing circuitry can be embed-ded on-chip together with the sensing circuitry. Suchembedding can happen either pixel-by-pixel (in-pixel cir-cuitry), or at chip level (off-pixel circuitry), or as a com-bination of both. Thus, for instance, in-pixel processingcircuitry can be used to obtain high-speed through paral-lel processing in tasks where many data are involved such

as spatial filtering, image feature extraction, motion esti-mation, … [12]. Then, off-pixel embedded digital proces-sors can be used for control and high-level processingtasks involving a reduced data set [13].

Sensory-processing embedding is crucial for some cus-tomized processing architectures [14]. Furthermore, it canbe used to speed up the computations needed to adapt thesensor response to changing conditions in the environ-ment; i.e., to make the sensor section capable of acquiringimages with High Dynamic Range (HDR). This latter featureis crucial since car vision sensors must be able to “see”under a very wide range of lighting and weather conditions.

Figure 2 shows a conceptual block diagram of thetasks involved in the development of a bio-inspired,custom CMOS vision system to be mounted at cars forcollision-warning. The final goal encompasses the coor-dinated completion of multi-disciplinary activities ofquite diverse nature, namely:

■ To model the underlying biological behaviors bymathematical descriptions.

■ To refine the bio-models upon a representationrealizable through electronic circuits.

8 IEEE CIRCUITS AND SYSTEMS MAGAZINE SECOND QUARTER 2008

Bio-ModellingHeuristic Considerations

Computational Video Analysis

ChipDesign

Hardware-SoftwareProcessor

Design

System Specificationand Test

MathematicalVision Models

HardwareCompatible M.

OtherModel Features

OptimizedHardware-C M.

Sensor/ProcessorIntegration

Hard-SoftPlatforms

LocustProcessor SoCs

LocustSensor Chips

BenchmarkVideo Sequences

Requirementsand Specs

HDR CMOSPixels

Model CarTests

Actual CarTests

Interpretation

Figure 2. Conceptual flow of activities and tasks from the understanding of the neuro-biological principles and behaviors ofinsects to the integration of a prototype CMOS vision system in a real car and the test under real traffic conditions.


■ To optimize the structure and parameters of suchrepresentation using benchmark real trafficsequences specified by car builders.

■ To conceive and design dedicated CMOS chipscapable of: 1) acquiring HDR images; 2) imple-menting the bio-inspired processing algorithm;3) fulfilling the robustness and reliability require-ments of automotive.

■ To conceive and design hardware/software plat-form to: 1) host the dedicated chips; 2) interactwith the car electronics through the availablebuses; 3) fulfilling automotive requirements.

■ To integrate the hardware/software system intocars and to perform testing in benchmark colli-sion scenarios.

As Figure 2 shows, the abilities and features of Locustmust be supplemented with other features for properoperation. Also, practical considerations and experiencedictate the convenience of using model cars as well ascommercial cars for testing purposes.

2. The Underlying Biological Models

Natural vision systems have been improved through mil-lennia of evolution and are more robust, compact and effi-cient that artificial counterparts. Many insects rely onvision for cruise control, navigation and collision avoid-ance [11], [15]–[21]. They are also able to perform thesetasks within wide range of lighting condition. Why not totake advantage of the knowledge about these systems?

Figure 3 shows the concept of a system based on the“Lobula Giant Movement Detector” (LGMD) neural struc-ture which is found in the visual system of the Locust[11], [21]. This structure fires an alarm if some collisionthreat is detected, thus giving rise to an evasive maneu-ver. An emulated-LGMD module is at the core of the sys-tem in Figure 3 and has been complemented for enlargedfunctionality with two other modules [22], namely:

■ A Topological Feature Estimator (TpFE), whosepurpose is to make an early classification of theapproaching object that generates the alarm.

■ An Attention Focusing Algorithm (AFA), aimed tooptimize the use of the computing resources byrestricting the processing to the zones of the framethat present more activity at a given instant.

Deployment of warning signals in the presence oflooming objects on a collision course is the basic role ofthe LGMD module. The TpFE and AFA modules are meantto provide further information about the environmentand hence to: optimize the use of computer resources, todiscriminate between real and spurious alarms, and toallow prompt appropriate evasive maneuvers.

Several works have addressed the use of insect visionsfor vehicles. Based on Elementary Motion Detector (EMD)

units, the work in [23] presents a bio-inspired analog VLSIchip that detects collisions up to 500msec before theyoccur. EMD units provide input to several directionally-sensitive neural structures which are found in insects likethe flies. The idea is to design an array of EMD disposedin a radial way, from the centre of the array to the bor-ders. This is the neural structure thought to be used inthe landing mechanism of the fly. On the one hand, anapproaching object would generate response in almost allthe EMD’s, given that its edges would all move outwardsin the radial sense. On the other hand, an object thatapproaches in non-collision course, or a translatingobject, would generate response in only a part of them.Simple steering and collision avoidance mechanismsbased on fly EMDs have also been investigated in [24]using robot platforms. However system integration ofthese approaches into real cars is still in its earlier stages.

A. LGMD Behaviour and EmulationThe LGMD neural structure, devised upon experimenta-tion with Locusts [21], consists of 4 retino-topical layersof mutually interacting neurons, see Figure 4. Layer 1 rep-resents the input to the neuronal circuit where P-unitsperform a high-pass temporal filter of the scene by sub-tracting the value of the luminance in a previous instant,�t, from the current local value. Outputs from the P-unitsare simultaneously fed into an excitatory neuron E andan inhibitory neuron I. These two neurons constitute thesecond Layer. Layer 3 contains an excitatory neuron S,which collects excitatory inputs from E neurons withinits retino-topic unit and inhibitory activity from I neurons


LocationAttention Focusing Module

GMD Module

CollisionAlarms

ClassificationTopological Classification

Module

Figure 3. Basic components of the proposed bio-inspired sys-tem. Input visual information is passed to the LGDM modulefor evaluation. Spatial attention of that LGMD is driven by theAFA module. Finally, information regarding the nature of theinput object is extracted by the TpFE module.


located within a finite neighborhood—called area ofinfluence. Inhibitory activity spreads laterally withdelays �t and 2�t before entering the S units, thus pro-viding a mechanism for spatio-temporal computation.Finally, excitatory activity of all S neurons convergesinto a single neuron, the already mentioned LGMD, inLayer 4, where it is accumulated. This biological modelalso suggests the existence of a global inhibition mech-anism, F neuron, which is fired whenever the number ofactivated photoreceptors exceeds a given thresholdwithin a time interval.

Mathematically, the behavior of each neuron can bedefined by a spatio-temporal differential equation whichrules the evolution of a state variable—called membranepotential in a biological language. Neuron activity andhence the information that is transmitted to otherneurons in the structure occurs in different forms butalways as a function of such state variable. In the originalformulation, neurons belong to one of the following class-es, namely: Linear Threshold Cells (LT), or Integrate andFire Cells (IF). In both cases, the membrane potential v(t)is defined by a discrete time equation,

vi (t + 1) = Pi · vi(t ) + g Exci ·

N Esc∑j=1

wij · aj

(t − δ ij

)

− g Inhi ·

Ninh∑k=1

wik · ak

(t − δik

)(1)

where pi is the persistence of the membrane potential,g Exc

i and g Inhi are the gains of the excitatory and inhibito-

ry channels respectively, wik is the strength of the synap-tic connection from neuron k to neuron i, and δik is thedelay associated to this connection.

Neuron activity ai for Linear Threshold Cells is defined as:

ai (t ) ={

vi (t ) vi (t ) ≥ θ

0 otherwise(2)

Regarding Integrate and Fire neurons, the activity is thegeneration of a discrete-amplitude spike whenever themembrane potential exceeds the threshold.

ai (t ) ={

β vi (t ) ≥ θ

0 otherwise(3)

Besides, when a spike is produced, the membrane poten-tial is hyper-polarized such that vi(t + 1) → vi (t + 1) − α,being α the amplitude of the hyper-polarization.

Table 1 shows the equations governing the originalbio-model.

B. Computational LGMD Model The bio-model of Figure 4 captures correctly the reactionsof the Locust to looming objects in collision course and isa starting point to build a car collision warning system.However, towards this end, its structure and its parametersmust be modified to reach the following targets:

■ Fit the frame rate to the distance resolutionrequired by safety regulations. For instance, for 25Fps the distance resolution will be 1 meter assum-ing 90 km/h speed.

■ Make the model realizable by electronic circuits.Particularly, implementation becomes largely sim-plified if all model operations are realized througharithmetic operations between matrices.

■ Select the model structure and parameters to guar-antee robust operation within the range of illumina-tion conditions, temperature and spreading of theunderlying circuit parameters. Illumination rangegoes well from 0.3 cd/m2 to 30000 cd/m2; tempera-ture range goes from 40◦C to 110◦C; and for circuitparameters spreading typically 6σ design [25].

■ Guarantee correct model operation for a completeset of benchmark collision scenarios and trafficvideo sequences. These sequences were deliveredby Volvo Car Corporation [26]. Figure 5 shows exem-plary sample frames for some of them. The accuracy

IEEE CIRCUITS AND SYSTEMS MAGAZINE SECOND QUARTER 200810

P

E

E

I

I

I

I

I

F

E

E

S

S

S

S

S

S

LGMD

E

P

P

P

P

Δt

Δt

Δt

Δt

Δt Δt

Δt Δt

Δt Δt

Figure 4. Conceptual model for the Lobula Giant MovementDetector (LGMD) [11].


of the model response for each scenario/benchmarkwas qualified by expert human observers.

■ Suppress spurious threats by the incorporation offeatures different from those observed in the bio-logical beings.

The modification procedure follows a heuristicsearch where hypothesis are validated by simulationswith detailed electrical macro-models. To build suchmodels, non-ideal electronic effects such as memoryleakage, mismatching, non-linearity in the operators, off-sets, electronic noise, etc. must be analyzed anddescribed by equations. The result of this heuristic

model tuning is the so-called Simplified NoM (NoiseOptimized Model) shown in Figure 6. Main differenceswith the original model are:

■ The model parameters and state variables are tunedsuch that everything can be satisfactorily executedwith 6-bits equivalent accuracy. Moreover, inputimage was decided to be 100 × 150 pixels.

■ In the competition between inhibition and excita-tion, the inhibitory pathway contains informationabout the current spiking rate.

■ The LGMD threshold is made adaptive instead ofconstant.

(a) (b) (c) (d)

Figure 5. Exemplary sample frames from selected test sequences. (a) Highway driving. (b) Roundabout. (c) Collision with testballoon car. (d) Pedestrian crossing in front of the car.

Cell State equation and output equation.

P Pij(t + 1) = 0.4 · Pij(t) + Lij(t) − Lij(t − 1)

P∗ij (t + 1); Integrate and Fire with β = 1 α = 0.5 θ = 0.3

E Eij(t + 1) = 0.1 · Eij(t) + 0.6 · P∗ij (t)

E∗ij (t + 1); Linear Threshold with θ = 0

I Iij(t + 1) = 0.8 · Iij(t) + 0.2 · P∗ij (t)

I∗ij (t + 1); Linear Threshold with θ = 0

S Sij(t + 1) = 0.4 · Sij(t) + E∗ij (t + 1) − 0.4

[ ∑Nr1

I∗kl(t − 1) + 0.8 · ∑Nr2

I∗kl(t − 1) + 0.5 · ∑Nr3

I∗kl(t − 2)]

S∗ij (t + 1); Integrate and Fire with β = 1 α = 0.5 θ = 0.5

where:

Nr1 are the four nearest neighbors (top, bottom, right, left)

Nr2 are the four nearest neighbors in 45◦ (top-right, top-left, . . . )

Nr3 are the four second-level nearest neighbors (top, bottom, right, left)

F F(t + 1) = 0.1 · F(t) + 0.008 ·N∑

i=1

M∑j=1

P∗ij (t)

F∗(t + 1); Linear Threshold with θ = 0.15

LGMD y(t + 1) = 0.4 · y(t) − 5 · F∗(t − 1) + 0.08 ·N∑

i=1

M∑j=1

S∗ij (t)

y(t + 1); Integrate and Fire with β = 1 α = 0.25 θ = 0.5

Table 1. Original model.



■ Threat scenarios are classified according to the fir-ing rate of a neuron which receives informationfrom the LGMD and the Threshold cell.

Mathematically the model defines an ON-OFF cell whichcomputes the local contrast C (i, j, t) as:

C (i, j, t ) = |L(i, j, t ) − L(i, j, t − 1)| (4)

with luminance values in the range [0:63]. The inhibition is produced as a spatial low-pass fil-

tered version of the contrast, then:

I(i, j, t ) = K ⊗ C (i, j, t ) (5)

where K is a typical 3 × 3 low-pass kernel and ⊗ denotesconvolution.

Excitation from the ON-OFF cells C (i, j, t), and inhibi-tion I(i, j, t ) are combined into the Summing Units to pro-duce a net excitation S(i, j, t ):

S(i, j, t) = max {C (i, j, t) − w(t) · I(i, j, t − 1), 0} (6)

where w (t ) are empirical, firing rate dependent,weights which are defined below.

The net excitation enters the LGMD neuron to gener-ate a membrane potential e (t ):

e(t) =100∑i=1

150∑j

S(i, j, t)150

(7)

whereas the LGMD output E (t) is computed as:

E(t) = −→u ◦ [e (t ), E (t − 1), E(t − 2)]t (8)

where ◦ denotes the dot product, and −→u = (1/24) ·[20, 3, 1], which was found by using genetic algorithmsand different test videos provided by Volvo Car Corpora-tion. By definition E(t) is zero for the first 15 frames(t = 1, 2, …15).

The threshold computation involves a second LGMDvariable v(t) which results from:

v(t) = −→u ◦ [e(t ), v (t − 1), v (t − 2)]t (9)

PΔt

Δt

Δt

Δt

Δt

Δt

Δt

Δt

Δt

Δt

Δt

Δt

P

P

P

SPKTHRLGMD

ON

-OF

FO

N-O

FF

ON

-OF

FO

N-O

FF

Inhi

bitio

nIn

hibi

tion

Inhi

bitio

nIn

hibi

tion

S-Units

100x150

S-Units

S-Units

S-Units

Figure 6. Computing model selected for electronic implementation.



which is also zero for frames 1 to 15, and that is employedto calculate a raw threshold V (t ) as:

V(t) = −→V ◦ [v(t − 5), v(t − 10), v(t − 14)]t (10)

where the parameters in −→V are

−→V = (1/7) · [1, 3, 3], and

were found in the same genetic algorithm used for −→u .Finally, the threshold cell output is the effective thresh-

old W(t) which is computed as:

W (t ) = max{V (t ), 50} + 15 (11)

Threshold and LGDM output values are passed to the fir-ing neuron (SPK in Figure 6) which fires an spike in frame tj if

E(tj) ≥ W(tj) (12)

Besides, if a spike has been produced, the LGMD suffers ahyper-polarization which makes [E(tj − 1), E(tj − 2),

E(tj − 3)] = 0. Observe that while the LGMD states arereset after firing spikes, the threshold cell, which is gov-erned by a similar state equation, does not. Note also thatin cases where activity variation is low, LGMD and thresh-old cell outputs tend to the same value and therefore nospikes will be produced.

Finally, the model incorporates an accumulator whichcontains the number of spikes produced in the last fiveframes. The state of this accumulator, n (t), is used to definethe response of the model (State) and to modify the strengthof the inhibitory pathway w (t ) in (6) as indicated in Table 2.

C. Attention Focusing Algorithm (AFE)Heuristics search shows that, typically, all relevant infor-mation is contained within reduced regions of the lastmodel layers. This happens for instance when an object isapproaching and the excitation is not generated by a turn-ing movement or camera vibration—see Figure 7. In thesesituations focusing on the regionof interest reduces the amount ofinformation to be processed andimproves the model efficiency.

Inputs to the AFE module aretaken from the S-layer becausethere the background and othernon significant pieces of informa-tion have been already removed.The AFE divides the input frameby defining an attention gridformed by groups of cells, some ofwhich are enabled and other dis-abled. Each group defines an atten-tion-cell. Cells within an enabledattention-cell are active and con-

tribute to the model computation. On the contrary, cellswithin a disabled attention-cell remain silent and their out-puts do not contribute to the LGMD potential.

At the beginning, the attention grid is, by default, achess board pattern and the following modules are usedto control the activation of attention cells:

■ Attention: this module decides which attentioncells exhibit enough activity to activate theirneighbors (activating cell). It also disables theones in which there was not enough activity in theprevious frame, transferring their activation tothe nearest disabled attention cell that belongs tothe default activation grid.

■ Activate: this module is called by the Attention one,and activates the attention cells that surround thecurrent activating cell. It returns the number ofcells it has actually activated, since it depends onthe location of the central cell, and the previousstate of activation of its surrounding attention cells.

■ Disable: this module is also called by the Attentionone, and its duty is to switch off the same numberof attention cells that Activate has activated. Thegoal is to maintain constant the amount of informa-tion to be extracted from the array of the S-cells.The rule is to switch off the attention cells that arefurthest to the current activating cell.

Experience reveals that the best default attention gridsare the ones that uniformly distribute the attention overthe frame when there is no particular source of excitation.


Figure 7. Examples of distribution of the excitation in the S layer. Observe that it main-ly concentrates in the surroundings of the approaching object (inside the red border),whereas the rest of the cells in the image remain silent.

n (t ) 1 2 3 4 5w (t ) 0.75 1.0 1.25 1.5 1.5State No danger Attention Danger Extreme

Danger

Table 2. w (t ) and model output as a function of the spiking rate.


Examples of the concentration of the attention can beobserved in Figure 8. The circle pointer indicates the cen-tre of mass of the excitation in the S-layer. Green colormeans neither danger nor activity; yellow means activitybut no danger; and red activity an alarm.

D. Topological Feature Estimator (TpFE)The way how the LGMD potential is calculated involves los-ing information about the shape of the object originatingthe alarm. However, this information is pertinent in manycases. For instance, a sharp turning of the car makes thebackground to move rapidly, spreads the activity out acrossthe entire S-layer and may produce false alarms. Similar sit-uations happen when the car is approaching a horizontalroad line, if a bulge of the road shakes the camera, etc. TheTpFE module overcomes these problems by making anearly classification of the approaching objects. In additionto making the system more accurate, such classificationhelps when taking countermeasures with better fitting inthe type of the approaching object (whether it is a pedes-trian, a car, etc.).

Using the information contained in the S-layer, shapesare classified into four different classes correspondingrespectively to:

■ vertical-shaped object such as traffic lights, per-sons … —vertical-shaped class

■ road line or road stripe—horizontal-shaped class■ turning movements—spatially global class■ an approaching car—squared-shaped class.

Classification requires, first, calculating a vertical vector(obtained by adding up column entries, row-by-row) and ahorizontal vector (obtained by adding up row entries, col-umn-by-column), respectively. Usually, the first entries ofthe vertical vector are discarded to filter out the spuriousactivity caused by the movement of the horizon. Then twostatistical descriptors δV (t ) and δH (t ) are built using themean value μ and the variance σ of these vectors, namely:

δV (t ) = μV (t ) − σV (t )

δH (t ) = μH (t ) − σH (t ) (13)

The dominant components of the shape are evaluated bycomparing the values of these descriptors with a prioridefined values for each of the classes being considered.Figure 9 shows examples of object classification by theTpFE module.

3. From the Locust to the Car: LOCUST CMOS Chips

The correct operation of the collision-warning systemwhen mounted on real cars requires proper image acqui-sition at the photosensor layer. This is challenging due to:

■ The necessity to handle wide illumination ranges.■ The stringent automotive requirements regarding

temperature range, vibrations, EMI, …■ The necessity to guarantee correct operation

under wide tolerance margins of the underlyingtechnological parameters.

While the last two challenges apply to both the sensorand the processor, the first is spe-cific for the front-end [29].

Different approaches for HighDynamic Range (HDR) image sen-sors have been reported such aslogarithmic compression, dual andmultiple sampling, programmablegain, intelligent control of expo-sure, …, etc. Some of the most rel-evant early ideas are found in[30]–[37]. Overviews are present-ed in [38] and then in [39], [40].Practical circuit implementationscan be found at IEEE Solid-State Cir-cuits archive. One may rely onavailable HDR sensors and com-bine them with standard digitalprocessors to implement the colli-sion warning models.

Alternatively, the design of thefront-end sensory part and thatof the processing part can beaffronted as linked componentsof a single sensory-processing


Figure 8. Examples of the attention focussing depending on the activity detected ineach part of the image. The attention pattern in zones where no activity is detectedcorresponds either to the default attention grid or to disabled attention cells (black).


design problem. Still different architectural solutionsapply, namely:

■ Using a fully retino-topic architecture where a sig-nificant part of the processing is completed in fullyparallel manner through dedicated in-pixel analogand mixed-signal circuitry

■ Keeping only a few mixed-signal functions in-pixeland shifting all the remaining to be handled by ded-icated digital processors.

To untie the gordian knot of this architectural choicethe following question should be answered: where toplace the border between the parallel mixel-signal circuit-ry and the serial digital processing circuitry? This ques-tion arises when one confronts the design of any on-chipsensory-processing system [13]. Its answer involvesissues regarding the application (for instance whethercomplete images or only image features must be down-loaded from the sensor); considerations on the computingparadigm to adopt [14]; and issues regarding feasibility ofthe available technologies and circuit primitives [41].

In the case of automotive applications the decision aboutwhere to place the border between the analog and the digi-tal domain is largely conditioned by data storage issues. Par-ticularly by those related to short-term storage ofintermediate data. Analog storage provides compact solu-tions for neural and bio-inspired VLSI systems [42] and hasbeen demonstrated through both academic [12], [43], [44]and industrial [13] CMOS VLSI chips. However, in automo-tive applications leakage currents and reliability considera-tions pose difficult challenges to the practical usage ofanalog storage. They can be addressed by using a refreshingscheme consisting of AD and DA converters in a loop [42],buts this increases the pixel size and severely penalizes thepixel count and hence the spatial resolution of the sensor.

A. HDR CMOS Pixel

Figure 10 shows a concept of a HDR pixel which com-bines the principles of companding pixels with those ofpixels based on the control of the integration time. Com-panding in this pixel is not determined by inaccuratedevice features as it happens with logarithmic compres-sion based on the I-V characteristics of MOS transistors inweak inversion. Instead, companding is controlled by tim-ing. The principle is similar to that of integration pixels.The photogenerated current is integrated in a capacitorproducing a decay of the voltage. Assuming that both thephotogenerated current and the pixel capacitance remainconstant during the operation, the decay happens to belinear and determined by the result of the multiplication ofthis current and the integration time,

Vph(t) = VRST − Iph

Cpix· t (14)

In integration pixels this voltage defines the pixel output.In the pixel of Figure 10 this voltage signal is employed toset the time interval during which a reference signal isfirst tracked and finally sampled. This latter signal iseither stored on a capacitor, Figure 10(a), or on a digitalregister, Figure 10(b).


Figure 9. Examples of object classification using the TpFEmodule and the corresponding alarm level.


The mixed-signal choice of Figure 10(b) has the mainadvantages of precluding the analog memorization errorsand the Fixed Pattern Noise artifact (FPN [39]) caused bythe mismatch of the analog buffer which delivers Vpix acrossthe photosensor array. In this pixel, an analog signal VRE F (t)is compared with the photo-generated voltage Vph(t ), and aN-bit digital signal VIO[N–1:0] is continuously sampled bythe N-bit register on the pixel. While Vph(t ) ≤ VRE F (t ), thecomparator disables the write-enable signal in the memoryand the pixel output tracks the time evolution of the digitalsignal. When Vph(t ) = VRE F (t ) the current value of the digi-tal signal is stored as the pixel output.

In the case of linearly increasing reference signals(as shown in Figure 10(c)), the I-V compression curveis given by:

Vpix( Iph) = VRST

1 + IphISat

(15)

Which is obtained by assuming that VRE F (t) =VRST · t/TEX P , and where ISat denotes the value of thephotogenerated current which produces a voltage drop

VRST within a time interval TEX P, or, in other words, thephotogenerated current wich produces the saturation ofthe pixel in integration mode for this exposure.

This companding approach enhances the dynamicrange by precluding brightest pixels to saturate. Conven-tional integration pixels exhibit a dynamic range given by

DRL IN = 20Log(

F SVN

)(16)

where F S = VRST − VN is the maximum attainable(detectable) voltage drop across the integration capaci-tance, and VN is the minimum voltage drop that can bedetected (usually this is referred to the noise value, sinceit is assumed that a signal is detectable if its SNR is one).This is also the value of the dynamic range at the integra-tion capacitor of the HDR pixel. However, the dynamicrange of the photogenerated current is larger due to theaction of companding. Maximum photogenerated currentproduces a voltage drop VRST − VN in a short time,whereas the minimum photogenerated current producesa voltage drop VN in a time interval TEX P. After some cal-culations one obtains:


RSTRSTVph

VREF

VRST

+

−

VIO

VIO

Vdd

IB

M1

Vpix

Vph+

−

VIO[5:0]

WEDRAM#1

VREF

(a) (b)

(c)

0 1 2

0 1 2

3 4 5 6 7

DRAM#1 Update Blocked

Time

Time

VIO[5:0]

DRAM#1

Vmax(t)

Vph(t )

Vmin(t )

VREF(t)Darkness

Tfirst Texp

Tlast

*

Figure 10. HDR mixed-signal active pixel concept with analog sampling (a), and digital sampling (b), typical waveforms for a 3-bitstaircase.


DRHDR = 20Log

(max

(Iph

)min

(Iph

))

∼= 40Log(

F SVN

)(17)

It is seen that the DR of the photocurrent is two-orders-of-magnitude larger than that of the integrated voltage.

Seeking for simpler design, the reference signal isreplaced by a digitally-generated staircase followed by aDigital to Analog Converter (DAC). If the converter satis-fies VLS B > VN , and it is properly programmed to coverthe maximum attainable voltage drop at the photodiodecapacitance, then the dynamic range results:

DRHDR = 20Log

[1 − 2−(N+1)

2−(2N+1)

(1 − 2−N − 2−(N+1)

)]

(18)

The global operation is illustrated by the transfer curveof Figure 11(a), where the code 2N − 1 corresponds to a

photogenerated current of 18pA. The figure also showsthe curve corresponding to linear integration. It is seenthat the HDR pixel produces non-saturated outputs athigher luminances. Besides, different transfer curves canbe obtained by properly programming the reference sig-nal VRE F (t) and VIO[N–1:0] as it is illustrated by the 3-Dfamily of measured curves shown at the figure inset.Hence a feedback signal can be applied to adapt the pixelresponse to the global (inter-frame) illumination condi-tions. Under the assumption that VLS B > VN , equation(18) shows that the DR can be controlled as a function ofthe number of bits in the DAC. It provides some degree offreedom to cover different specifications, but has coun-terparts. Particularly it has a direct impact on the numberof memories on the pixel which are devoted to VIO[N–1:0]storage, and hence on the spatial resolution of the sensor.

Figure 11(b) illustrates the operation of a CMOSimager prototype with a 6-bit DAC (DRHDR = 78 dB


1Vout(V)

001020

304050607080 102 101 100 10−1 10−2

Light Power (W/m2)

0.2

0.40.60.8

010−13 10−12 10−11

Photogenerated Current (A) Log. Scale

10−10 10−9

10

20

30

40

50

60

63

Output Code (Dec) Transfer Curve for Staircase Reference Signals

HDR

Linear

Time (ms)

(a)

(b)

Figure 11. HDR pixel response: (a) Transfer curve; (b) System mounted in the roof of a XC90 and exemplary HDR image acquisition.



and). This picture shows an exemplary HDR imagecaptured when the car is exiting a tunnel. Note thatboth the darker and brighter parts of the scene arecaptured with good level of details.

B. Semi-Retinotopic Sensor ArchitectureThe pixel architecture of Figure 12(a) takes into accountconsiderations regarding data storage reliability and thecomputational demands of the collision warning algo-rithm. Besides the photodiode and the comparator need-ed for HDR acquisition, the pixel contains three DRAMblocks, a DRAM refresh buffer, and a bidirectional I/O

buffer which transmits VIO to the pixel during opticalsensing and outputs the pixel values during readouts.DRAM#1 block is employed as VIO memory during opti-cal acquisition. Thus, at the end of a given image captur-ing process, the contents in the three pixel memoriescorrespond to three consecutive frames. This informa-tion is downloaded into the digital processor described inthe next section which executes the collision alarm algo-rithm. Afterwards, the chip moves DRAM#2 to DRAM#3,and DRAM#1 to DRAM#2, leaving DRAM#1 available forthe next optical acquisition.

Figure 12(b) shows the architecture of the completesensor which, besides the sensor array, contains the fol-lowing blocks:

■ A 16 kBytes SRAM block which is used by theprocessor in calculations

■ An input/output digital buffer which stores 150 pix-els (a whole row) and allows data transfer rates athigh-speed (128 Mbytes/s)

■ A HDR profile block which stores, in digital format,the waveforms to be applied at VIO[5:0] duringoptical acquisition, and the digital codes to be pro-vided to the Digital-to-Analog converter which gen-erates VRE F (t )

■ A biasing generator, which creates the requiredanalog signals for the chip

■ A PTAP temperature sensor which is used to adaptthe DRAM refreshing period, and

■ An auto-exposure block. This auto-exposure blockallows the definition of a region of interest and atarget value for the average value for the pixelswithin that region. It computes the mean value ofthe pixels in the defined region of interest duringimage downloading and generates a feedback sig-nal to adapt the sensor response to the global illu-mination conditions.

C. Dedicated Digital ProcessorFigure 13(a) shows the architecture of a System-on-Chipconsisting of the semi-retinotopic sensor and a dedicateddigital processor. This latter processor embeds four coresdedicated respectively to: a) control the sensor (LμCmicro-controller); b) perform the arithmetic operationsand memory access tasks involved in the emulation of theLGMD model (Full Custom Locust Model Computing Unit);c) perform the corresponding operations for the Topolog-ical Feature Extractor (Full Custom TpFE Computing Unit);d) control and coordinate the overall operation. The firstone is a full custom RISC micro-controller; the second andthe third are identical full-custom arithmetic processingcores; and the fourth is a general purpose processor.

The first three cores above operate in parallel, syn-chronously, in order to guarantee real-time operation

InternalControlLogic

HDRSensor

In DRAM#1 Out

In DRAM#2 Out

In DRAM#3

DRAM Refresh

Out

IO

6-b Column Bus

Con

trol

s

BiasingSignals

VREF

Biasing Signals

Zcomp

N × MPixels

EAB

EDB ECB

IO16 kBSRAM

AutoExpos.

Temp.Sensor

IDBICB

HDRProfile

BiasingSignals

Analog

Analog

Digital

HPB

(a)

(b)

Figure 12. Semi-retinotopic pixel (a) and sensor chip (b) forcollision warning based on the vision system of the Locust.


with a fixed frame rate. As compared to a single corechoice, the multi-core solution yields reduced hardwarecomplexity. Hence, its corresponding hardware designcycles are shorter. Besides, the multi-core architecture is

easier to re-program when fine tuning is required to fitthe model response to real-time automotive scenarios.Conversely, the single-core processor is prone to exhibitnon-deterministic behaviors that impact on the frame


CMOS Sensor

LensSRAM

LμCMicrocontroller

Full CustomLocust Model

Computing Unit

Full CustomTpFE

Computing Unit

Processor

LVCNeuronLayerSRAM

Multichannel DMAController

Res

gist

er S

et

VectorialALU

Multiplier

BarrelShifter

Reg

iste

r S

et

DATASRAM

Data Path and Controls

SequencerInstruction

SRAM

LVC

Sensor Array

Analog Refs

HDR Profile

Autoexposure Module

4096 x 32 bSRAM

SFR

Data

Hndshk

IO

IOGPR

Timing

Timers

PrescalersALU

D RAM

Data PathDPR

SFI Sequencer Sequencer

Control Unit II Code ROM

Control Unit I

I RAM

LμC

(a) (b)

(c)

Figure 13. Collision-warning System-on-Chip (SoC) architecture based on the vision system of the Locust: (a) Chip architecture;(b) Full custom computing unit block diagram; (c) Sensor microcontroller.


rate. As the matter of fact, successful automotive visionprocessors use multi-core solutions for computationallyintensive applications, real-time visual recognition andscene interpretation [45]–[47].

The core labelled LμC micro-controller in Figure 13(a)is devoted to low-level control functions, while the so-called full custom computing units are devoted to inten-sive arithmetic calculus. The corresponding tasks areenumerated below:

■ Low-level control of the HDR sensor. These taskscomprise the downloading of images and the con-trol of the surrounding circuitry of the sensorarray: control of the HDR profile block, the inter-nal analog references, the auto-exposure blockand the internal DRAM memories refresh andmoving data processes.

■ Emulation of the LGMD model. These tasks includememory transfers, matrix calculus and convolutionsrelated with the LGMD neural structure model: theON-OFF cells movement map, the inhibition layerlow pass filter and the S-units layer calculations.

■ Topological Feature Estimator algorithm. Thesetasks include matrix calculus, means, variancesand arithmetic operations related with the geo-metrical descriptors of the objects that are inthe field of view.

The digital processors and the HDR CMOS image sen-sor are arranged into three levels of control hierarchy:the general-purpose processor at the top-level; the arith-metic computing units together with the RISC micro-con-troller at the second; and the HDR CMOS image sensor atthe third. This multi-core design is flexible and powerfulenough to accept changes and improvements of the rou-tines and models without affecting the real-time perform-ance of the collision-warning.

The control-commanding requirements of the HDRsensor include asynchronous and real-time low-leveltasks. The custom micro-controller included in thesystem, called LμC, frees the general purpose proces-sor from these tasks, increasing the overall systemefficiency. This controller receives high level com-mands from the host processor and translates theminto a series of function sequences to control the HDRsensor circuitry.

The LμC is a RISC computer with fixed encoding reg-ister-register Harvard architecture whose programmingis realized in assembler. Its architecture (see Figure13(c)) comprises General Purpose Registers (GPR), Spe-cial Function Registers (SFR), an ALU, a bidirectionalshifter, timers and two control units with sequencers.The device accepts up to 5 interrupts sources.

The LμC architecture is designed to receive new highlevel commands from the host CPU and to send data,coming from the micro-controller control routines, even ifthe LμC is busy. Its data path includes special features tospeed-up the interchange of information between regis-ters and between registers and memory. The controller’s


Figure 15. Installation of the collision-warning system in therear mirror area of the XC90.

(a)

(b)

Transceiver

CameraVideo Sender

80c51 μP RAM

Flash

xc2s200EFPGA

SpeedSensor

DirectionServo

MotorController

Figure 14. Model car: (a) High-Level architectural conceptand (b) photo.


instruction-set has a subsetof instructions designed toaccelerate image uploadingand downloading processeswhile saving CPU load.

The full custom arithmeticcomputing units are con-ceived to deal with memorytransfers and matrix calculusrelated with neurons modelsand statistical descriptors.Besides coordinating theoperation of the whole sys-tem, the general purposeprocessor deals with low pro-cessing demanding scalar cal-culations of the LGMD modeland the object classification.

Both units, Figure 13(b),include a vectorial comput-ing core, with functionalityresembling that of standardDSP execution units. It con-tains a vectorial ALU, a multi-plier and a barrel shifter, plustwo sets of registers for tem-poral data storage and a pro-grammable sequencer. Inaddition, a multi-channelDirect Memory Access de-vice (DMA), which operatesas a stand-alone sub-system,is devoted to image andintermediate data transfers,servicing interrupts whendata transactions are com-plete. The computing cores,once programmed, work asperipheral units, processingraw data corresponding tothe sensed images, and deliv-ering processed informationaccording to the programmedsequences of instructions.

The computing capabili-ties of the processing unitsare oversized to allow newfunctions and enhancements,such as including the speedof the vehicle together withthe steering angle within theLocust model and improvingthe TpFE capabilities.


1 2

43

20

40

60

80

50 100 150

30

20

10

0

−10

−20

−30500 100 150

30

20

10

0

−10

−20

−30500 100 150

30

20

10

0

−10

−20

−30500 100 150

The car is overtaken by the other car.In this scenario, a car passes the camera. It comes from back in the left lane. Observethe signal in the red rectangle. The threat signal also changes, it goes into negativeshowing that moving object (the car) falls into the central region asymmetrically.

20

40

60

80

50 100 150

From Left Object30

20

10

0

−10

−20

−300 50 100 150

From Right Object

From Left Object From Right Object

30

20

10

0

0

−10

−20

−300 50 100 150

30

20

10

−10

−20

−300 50 100 150

Balloon Car TestIn this scenario, the car hits a balloon car. In all experiments the systems respondedapproximately 1 s in advance to the collision. Activity above ten means potential danger.

Figure 16. Real car in-field test examples.


4. System Integration into Cars and Test

Integration into a car and testing are the mandatory cul-mination of all tasks and the gateway for eventual practi-cal usage. In addition, they are necessary for refinementof the models and the hardware/software processors.This latter task requires test platforms which are handy,manageable, open and flexible. Since these features aredifficult to achieve with real cars, resorting to model carsis absolutely necessary during the first stages of modeldevelopment and tunning.

Figure 14(a) shows the architecture of a model car(photo in Figure 14(b)) which is controlled via radio byinterchanging control commands and state informationwith an external computer. The electronic system of thecar comprises: a radio transceiver with a FSK modem; anexpandable embedded single board computer for radiopacket transmission management tasks; a full customdata IO handling control unit; peripheral control unitsdevoted to control the electromechanical subsystems ofthe car; and a camera connected to a SHF emitter forimage capture and transmission. The mechanical subsys-tem contains a servo for steering angle control, a highspeed electric motor and differential drive.

For practical use, the model car must be complementedwith scaled-down reproductions of the benchmark trafficscenarios considered. Realistic situations are replicated inthese scenarios and the feedback from qualitative andquantitative tests employed to refine the models. Particu-larly, the model car is crucial for experimenting with dan-gerous traffic situations which are not feasible in real cars.

The most significant milestone of the model car tests isa stable hardware platform, ready to integrate into anactual car. A Volvo XC90 was prepared, mechanically andelectrically, for that purpose. The sensor system wasmounted into the specially made bracket in the interiorrear mirror area as shown in Figure 15. A Head Up Displayand an audible warning device were installed. The Head UpDisplay is the black plastic bar on the dashboard, in thefront side of the instrument cluster cover. Inside the blackplastic cover there is a row of red LEDs that are pointing atthe windscreen (approximately 15cm up from the loweredge of the windscreen). When there is a collision alarm, avirtual brake light is projected on the windscreen glass asa red horizontal line. An audible warning is also given byactivating the buzzer that is mounted in the environmentalnode which is connected to the CAN bus in the vehicle.

In-field tests covered many different scenariosinvolving pedestrians and other cars. To enhance theselectivity for objects moving in different directions,the image is divided into four regions (see picture at thetop in Figure 16). The regions are defined by their sepa-rating sections and sections are defined by endpointsthat can be dynamically adjusted to the geometry of the

road. Figure 16 shows some representative examples ofthe way this segmentation works. Problems are basical-ly encountered with the handling of some shadows dueto the monochrome and monocular nature of the sen-sor. In some cases, shadows produce false alarms. Waysto overcome this drawback requires either incorporat-ing color information, or 3D vision or advanced adap-tive brightness control algorithms.

Conclusions

Conventional engineering methodologies follow a top-down approach (system specification, behavioralmodeling, architecture and block selection, block imple-mentation, block assembling, system simulation and tun-ing, verification and refinement). Alternatively, biologyhas evolved to solve complex sensory-perception-actua-tion problems by following what can be considered as abottom-up approach (systems evolve through the organi-zation of elementary units which improve by following akind of trial-and-error sequence of events). This paperoutlines the main facets and results of a hybrid strategythat combines a bottom-up route from biological behav-iors up to circuit implementations; and a top-down routefrom system specifications to processor and chip archi-tectures and then finally to system integration and test.

Such hybrid methodological flow is needed to go fromthe mathematical modeling of the behavior of biologicalstructures to the in-field test of collision threats in real traf-fic scenarios. The outcome of such a combination of multi-disciplinary activities is the practical demonstration of thepossibilities of biological systems when supplemented withproper circuit architecture design. It motivates furtherdevelopments looking at the design of automotive-qualifiedvision processors for dealing with collision threats.

Acknowledgment

Authors would like to acknowledge the contributions ofDr. R. Stattford, Dr. S. Yue, Mr. J. Cuadri and Dr. E. Rocaregarding the biological model, and of Dr. I. Petras and Dr.V. Gal regarding system integration and test.

Research in this work has been partially supported bythe spanish DIVISA Project TEC2006-15722 and ECLOCUST Project IST:2001-38097.

References[1] R. Bishop, “A survey of intelligent vehicle applications worldwide”.Proc. of the IEEE 2000 Intelligent Vehicles Symposium, 2000, pp. 25–30.[2] B.W.D. Jones, “Building safer cars,” IEEE Spectrum, pp. 82–85, Jan. 2002.[3] M. Bertozzi, et al., “Artificial vision in road vehicles,” Proceegings ofthe IEEE, vol. 90, July 2002, pp. 1258–1271.[4] B. Lakshmanan, “Vision on wheels,” SPIE’s OEMagazine, pp. 13–15, June 2003.[5] C.Y. Chan, “Trends in crash detection and occupant restraint tech-nology”. Proceedings of the IEEE, vol. 95, Feb. 2007, pp. 388–396.[6] J.C. McCall and M.M. Trivedi, “Driver behavior and situation awarebrake assistance for intelligent vehicles,” Proceedings of the IEEE, vol. 95,Feb. 2007, pp. 374–387.



[7] G. Wanielik, et al., “Polarimetric millimeter wave imaging radar andtraffic scene interpretation,” IEEE Colloquium on Automotive Radar andNavigation Techniques, vol. 4, pp. 1–7, 1998.[8] Ü. Özgüner, C. Stiller and K. Redmill, “Systems for safety and auto-mous behavior in cars: the DARPA grand challenge experience,” Pro-ceedings of the IEEE, vol. 95, Feb. 2007, pp. 397–412.[9] Automotive cameras for safety and convenience applications. Whitepaper: SMaL Camera Technologies Inc, 2004 Version2.[10] The LOCUST Project: IST 2001-38097 at www.imse.cnm.es/~locust[11] C. Rind and P.J. Simmons, “Seeing what is coming: building collision-sensitive neurons,” Trends in Neuroscience, vol. 22, pp. 215–220, 1999.[12] A. Rodríguez-Vázquez, et al., “ACE16k: The Third Generation ofMixed-Signal SIMD-CNN ACE Chips Toward VSoCs,” IEEE Transactions onCircuits and Systems-I, vol. 51, pp. 851–863, IEEE May 2004. [13] A. Rodríguez-Vázquez, et al., “The Eye-RIS CMOS Vision System,”Analog Circuit Design, Springer 2008.[14] T. Roska, “Cellular wave computers for nano-tera scale technology—beyond Boolean spatial temporal logic in million processor devices,” IEEElectronics Letters, vol. 43, April 2007.[15] C. Cliiford and K. Langley, “A model of temporal adaptation in flymotion vision,” Vision Research, vol. 36, pp. 2595–2608, 1996.[16] D. Edwards, “The cockroach DCMD neuron. II, Dynamics of responsehabituation and convergence of spectral inputs,” Journal of ExperimentalBiology, vol. 99, pp. 91–107, 1982.[17] N. Franceschini, et al., “From insect vision to robot vision,” Philosoph-ical Transactions of the Royal Society of London, vol. 339, pp. 283–294, 1992.[18] R. Harris, et al., “Adaptation and the temporal delay filter of flymotion detectors,” Vision Research, vol. 39, pp. 2603–2613, 1999.[19] R. Harrison, An analog VLSI motion sensor based on the fly visual sys-tem. Ph.D. thesis, California Institute of Technology, California, USA, 2000.[20] S. Judge and C. Rind, “The locust DCMD, a movement detecting neu-ron tightly tuned to collision trajectories,” Journal of Experimental Biol-ogy, vol. 200, pp. 2209–2216, 1997.[21] F.C. Rind and D.I. Bramwell, “Neural network based on the inputorganization of an identified neuron signaling impending collision,” J.Neurophysiology, vol. 75, pp. 967–984, 1996.[22] J. Cuadri et al., “A bioinspired collision detection algorithm for VLSIimplementation,” Proc. of SPIE Conference on Bioengineered and Bioin-spired Systems, vol. 5839, May 2005, pp. 238–248.[23] R.R. Harrison, “A Low-Power Analog VLSI Visual Collision Detec-tor”, Advances in Neural Information Processing Systems (NIPS 2003),Vancouver, Canada, 2004. [24] D. Floreano, et al., “From wheels to wings with evolutionary spikingneurons,” Artificial Life, 2004.[25] J.C. Zhang and M.A. Styblinski, Yield and Variability Optimization ofIntegrated Circuits. Kluwer Academic Publishers, 1995.[26] EC-IST LOCUST Project web page: www.ims.cnm.es/locust[27] S. Yue, et al., “A bio-inspired collision mechanism for cars: Opti-mization of a model of a locust neuron to a novel environment,” J. Neu-rocomputing, vol. 69, pp. 1591–1598, 2006.[28] F. Medeiro, et al, Top-Down Design of High-Performance Modulators.Kluwer Academic Publishers, 1999.[29] B.J. Hosticka, et al., “CMOS imaging for automotive applications,”IEEE Transactions on Electron Devices, vol. 50, pp. 173–183, 2003.[30] V. Ward et al., “CMOS photo-detector with built-in light adaptationmechanism,” Microelectron. J., vol. 24, no. 5, pp. 547–553, 1993.[31] A. Deval, et al., “Locally Adaptative Active Pixel Sensors,” Proc. ofthe IEEE Int. Symp. on Circuit and Systems, vol. 1, 1997, pp. 685–688.[32] O. Yadid-Pecht and E. Fossum, “Image sensor with ultra wide lineardynamic range utilizing dual output CMOS active pixel sensors,” IEEETrans. Electron Devices, vol. 44, pp. 1721–1724, 1997.[33] S. Decker, et al., “a 256 × 256 CMOS imaging array with wide dynam-ic range and column parallel digital output,” IEEE Journal of Solid-StateCircuits, pp. 2081–2091, vol. 33, no. 12, Dec. 1998.[34] D.X.D. Yang, et al, “A 640 × 512 Image sensor with ultrawidedynamic range floating point pixel level ADC,” IEEE J. of Solid StateCircuits, pp. 1821–1834, vol. 34, no. 12, Dec. 1999.[35] M. Schanz et al., “A high dynamic range CMOS image sensor forautomotive applications,” Proc. of the 199 European Solid-State CircuitsConference, 1999, pp. 246–249.[36] S. Kavadias, et al., “A logarithmic response CMOS image sensor withon-chip calibration,” IEEE J. of Solid-State Circuits, vol. 35, pp. 1146–1152,August 2000.

[37] T. Hamamoto and K. Aizawa, “A computational image sensor withadaptive pixel-based integration time,” IEEE J Solid-State Circuits, vol. 36,pp. 580–585, April 2001.[38] O. Yadid-Pecht, “Wide-dynamic-range sensors,” Optical Engineering,vol. 38, pp. 1650–1660, October 1999.[39] A. El Gamal and H. Eltoukhy, “CMOS image sensors,” IEEE Circuits &Devices Magazine, vol. 21, pp. 6–20, May/June 2005.[40] M. Bigas, et al., “Review of CMOS image sensors,” MicroelectronicsJournal, vol. 37, pp. 433–451, 2006.[41] A. Rodríguez-Vázquez, et al., “Mismatch-induced trade-offs and scal-ability of analog preprocessing visual microprocessor,” Analog IntegratedCircuits and Signal Processing, vol. 37, pp. 73–83, Kluwer Academics 2003.[42] Y. Horio and S. Nakamura, “Analog memories for VLSI neurocom-puting,” in Artificial Neural Networks: Paradigms, Applications and Hard-ware Implementations, 1992, pp. 344–363, IEEE Press.[43] G. Liñán-Cembrano, et al., “A 1000 FPS at 128 × 128 Vision processorwith 8-bit digitized I/O,” IEEE Journal of Solid-State Circuits, vol. 39, pp.1044–1055, IEEE July 2004.[44] R. Carmona, et al., “A 0.5 μm CMOS random access analog memorychip for TeraOPS speed multimedia video processing,” IEEE Transactionson Multimedia, vol. 1, pp. 121–136, IEEE June 1999.[45] Mobileye home page: www.mobileye.com[46] W. Raab et al., “A 100-GOPS Programmable Processor for VehicleVision Systems,” IEEE Design & Test of Computers, vol. 20, pp. 8–16, 2003.[47] U. Ramacher et al., “A 53 GOPS Programmable Vision Processor forProcessing, Coding-Decoding and Synthesizing of Images,” Proc. 27thEuropean Solid-State Circuits Conf., 2001, pp. 133–136.

Gustavo Liñán-Cembrano received thedegrees of Licenciado and Doctor (PhD) inPhysics, in the speciality of Electronics,both from the University of Seville, Spain,in June 1996 and September 2002, respec-tively. In 1995 he was a granted student ofthe Spanish Ministry of Education at the

Institute of Microelectronics of Seville CNM-CSIC. From 1997to 1999 he got a doctoral Grant at the Institute of Micro-electronics of Seville CNM-CSIC funded by the AndalusianGovernment. From February 2000 to June 2004 he was anAssistant Professor of the Department of Electronics andElectromagnetism at the School of Engineering of the Uni-versity of Seville and at the Faculty of Physics. Since June2004 he is a Tenured Scientist of the Spanish Council ofResearch. His main areas of interest are the design and VLSIimplementation of massively parallel analog/mixed-signalimage processors. He has received the “Best Paper Award1999” and “Best Paper Award 2002” from the InternationalJournal of Circuit Theory and Applications. He is also co-recipient of the “Most Original Project Award”, of the “Salvài Campillo Awards 2002”, conceded by the Catalonian Asso-ciation of Telecommunication Engineers.

Luis Carranza received the B.S. degreein physics from the University of Seville,Seville, Spain, in 2000, and is workingtowards the Ph.D. degree in the Depart-ment of Electronics and Electromagnet-ism of the same university. Since 2001,he has been with the Department of

Analog Circuit Design, Institute of Microelectronics of



Seville, Centro Nacional de Microelectr—nica (IMSE-CNM), Seville, Spain. His research interests are in designof processor-based architectures and customized digitalsignal processors for systems on chip.

Claire Rind received her BSc at the Uni-versity of Canterbury, New Zealand in1976. She completed a doctorate at theZoology Department, Cambridge Universi-ty (UK) in 1981, and a post-doc in the Zool-ogy Department at the University ofNewcastle, where she worked on the

synaptic mechanisms underlying feature detection in thelocust. She then received two prestigious fellowships: a fiveyear BBSRC Advanced Research Fellowship and a ten yearRoyal Society University Research Fellowship (1989–1999).During this period, she supervised four Ph.D. students fromboth biological and engineering backgrounds and devel-oped an interest in the biological basis of collision avoid-ance in the locust. She began collaboration with roboticsengineers at the Institute of Neuroinformatics, ZurichSwitzerland. This continued with the collaborative EU fund-ed project described in this paper. Claire Rind is currently areader in the School of Biology and a member of the Insti-tute of Neuroscience at Newcastle University. She has pub-lished over 50 research papers, written five book chapters,and four review articles, as well as many abstracts in theproceedings of international conferences.

Akos Zarandy received his master degreeof Electrical Engineering in 1992, and hisPhD in 1997. During his PhD studies, hespent more than two years in various lab-oratories of the University of California atBerkeley, USA. During some years he wasC.E.O. of AnaLogic Computers Ltd, which

is a small enterprise founded to develop commercial appli-cations of the analogic cellular active wave sensing-com-puting technology.

His areas of interest are mainly focused to the applica-tion of analogic cellular computing technology, includingdevelopment of high-performance vision systems based onfocal plane analogic cellular computing chips, and applica-tion development on these devices.

Martti Soininen received his Master ofScience in Electrical Engineering in 1984 atthe Chalmers University of Technology.After his graduation, he worked at VolvoTruck Corporation designing specialadvanced testing equipments.

In 1993, Mr. Soininen joined Volvo CarCorporation (VCC) to work with sensors mainly for new

safety functions. He has a wide experience of laser andradar sensors in cars for Adaptive Cruise Control and Col-lision Warning and other safety systems. Mr. Soininen rep-resented VCC in the Adaptive Cruise Control Sensor Groupin the Prometheus program. He also participated in the EUSI_GYRO project together with IMSE-CNM, Bosch, and STS.He is a member of the Ford Motor Company common PreCrash Sensing group.

Angel Rodríguez-Vázquez received aPh.D. degree in Physics-Electronics in1983. He is a Full Professor of Electronicsat the University of Seville and head of Innovaciones Microelectrónicas S.L.(AnaFocus).

Prof. Rodríguez-Vázquez founded andheaded a research unit on High-Performance Analog andMixed-Signal VLSI Circuits of the Institute of Microelectron-ics of Seville (IMSE-CNM). His team made pioneering R&Dactivities on bio-inspired microelectronics, including visionchips and neuro-fuzzy interpolators and controllers. Histeam also made significant contributions to the applicationof chaotic dynamics to communications, including thedesign and production of the first world-wide chaos-basedcommunication MoDem chips. Some 30 high-performancemixed-signal chips were successfully designed by his teamduring the last 15 years in the framework of different R&Dprograms and contracts. These include three generation ofvision chips for high-speed applications, analog front-endsfor XDSL MoDems, ADCs for wireless communications,ADCs for automotive sensors, chaotic signals generators,complete MoDems for power-line communications, etc.Many of these chips are state-of-the-art in their respectivefields. Some of them entered in massive production.

In 2001, he was one of the co-founders of AnaFocus, ahigh-technology company focused on the design of mixed-signal circuits with emphasis on CMOS smart imagers andbio-inspired vision systems on chip. He has been headingAnafocus since 2004.

Prof. Rodríguez-Vázquez has authored/edited: eightbooks; around 40 chapters in contributed books, includingoriginal tutorials on chaotic integrated circuits, design ofdata converters and design of chips for vision; and morethan 400 articles in peer-review specialized publications.He has served as Editor, Associate Editor and Guest Editorfor different IEEE and non-IEEE journals, as chairman fordifferent international conferences and is in the committeeof some international journals and conferences. He hasreceived a number of international awards for his researchwork (including the IEEE Guillemin-Cauer best paperaward) and was elected Fellow of the IEEE for his contri-butions to the design of chaos-based communication chipsand neuro-fuzzy chips. He is an IEEE Fellow.



Feature Insect-Vision Inspired Collision Warning Vision...

Documents

Transcript of Feature Insect-Vision Inspired Collision Warning Vision...