Bala Lakshminarayanan, Mike McCullough

NVL Sensor Fusion Test Bed

March 18, 2004

Bala Lakshminarayanan, Mike McCullough

Introduction

• US Army Night Vision & Electronic Sensors Directorate (NVESD)

• Network of acoustic and image sensors– Visible and IR

• Classification of civilian targets

Bala Lakshminarayanan, Mike McCullough

Motivation & Background

• Military targets ~98%– 3 levels of fusion process, acoustic and

seismic data

• Current civilian classification ~80%– Improve accuracy rating– Both civilian targets and personnel

classification– Image data

Bala Lakshminarayanan, Mike McCullough

Objectives

• Generation of data set of image and acoustic data– Development and fusion of moving target ATR

algorithms

• Establish methods to collect data and its “ground truth”

Bala Lakshminarayanan, Mike McCullough

SFTB Setup

SFTB Base Station

Node 3 Acoustic sensor

Node 1 Acoustic sensor Node 2 Acoustic sensor

Node 3 IR sensor

Node 3 Acoustic sensor

Node 1 IR sensor

Node 1 Acoustic sensor

Node 2 Visual sensor

Node 2 Acoustic sensor

Met station & GPS

Commands through DOS Scripts

GPS

GPS

GPS

Wireless Ethernet connection

Bala Lakshminarayanan, Mike McCullough

Sensors

• Indigo Alpha Thermal camera

• Pulnix TMC-7DSP Color camera

• Knowles BL-1994 Microphone

Bala Lakshminarayanan, Mike McCullough

General Test Conditions (1)

• 3 nodes each with hexagonal acoustical array of 7 microphones and imaging sensor– Nodes 1 & 3 have uncooled IR camera– Node 2 has visible color camera

• Nodes gather information simultaneously for 3 minutes

• Acoustic sensor turns on imaging sensors

• MUSIC algorithm for DOA estimation

Bala Lakshminarayanan, Mike McCullough

General Test Conditions (2)

• Targets moving on gravel & asphalt roads

• Fully exposed– Trees or other vehicles occasionally in the

way

• License plates on the targets are not readable

• Stationary sensors

• Daylight operation (9:30am to 3:30pm)

Bala Lakshminarayanan, Mike McCullough

General Test Conditions (3)

• Target motion– Constant speed– Stops midway

• Constant acceleration, deceleration• Stops for count of ten

• Each target traverses at 5, 10, 15, 20 mph• Start and stop outside FoV of nodes• Creation of different scenarios

Bala Lakshminarayanan, Mike McCullough

SFTB Operation (1)

• Attended mode– Short term data collection / Demo mode

– Collects 4 types of data

– Surveillance, directed, pan scanning

– SFTB_Base.exe, FullSim.exe

• Data collection mode– Pure data collection

– Collects 4 types of data

– Acquire.exe (Video and acoustic), MetEffects.exe

Bala Lakshminarayanan, Mike McCullough

SFTB Operation (2)

• Collected data– Acoustic .dat – Image .arf– Ground Truth .agt

• Filenames depends on sensor, node, scenario and targets

Bala Lakshminarayanan, Mike McCullough

Numbering System

• SSSN00000_0000

• SSS = camera name– IN1 = Indigo IR camera 1– IN2 = Indigo IR camera 2– PX1 = Polinex Visible camera 1– AC1 = Acoustic number 1

• N = node number (1-3)• 00000 = scenario number• _0000 = number indicating vehicle number (1-7)

– Can have multiple numbers multiple targets

Bala Lakshminarayanan, Mike McCullough

Number System Example

• in1200004_0003– In1 indigo thermal camera #1– 2 node 2– 00004 scenario number 4– _0003 target 3

• ac1300001_0056– ac1 acoustic array #1– 3 node 3– 00001 scenario number 1– _0056 both targets 5 and 6

Bala Lakshminarayanan, Mike McCullough

AGT Format

• Very similar to a class in a high level programming language

• Agt{

PrjSect {…}SenSect {

SenUpd {…}}TgtSec{

TgtUpd {…}}

}

Bala Lakshminarayanan, Mike McCullough

AGT format

Bala Lakshminarayanan, Mike McCullough

PrjSect

• Name = “sftb”

• Scenario = “00001_0001”

• Site = “nvis0306”

Bala Lakshminarayanan, Mike McCullough

SenSect

• Denotes the sensor section of the AGT

• Contains all of the SenUpd

• Name = “in11”– Denotes which sensor being used

Bala Lakshminarayanan, Mike McCullough

SenUpd

• LatLong

• Elevation

• Keyword “Frame #1”

• Keyword “AcousticAz: 0”

• Keyword “Nodeld: 1”

• Azimuth 75.9077301025

• Time 2003 160 16 22 34 587

• Fov 40.0 30.0

Bala Lakshminarayanan, Mike McCullough

TgtSect & TgtUpd

• TgtSect is the sector that contains all the target updates

• TgtUpd– Keyword “Frame: #1”– Time 2003 160 16 22 34 587– Tgt

• Range 48.0• TgtType “MAN”• Aspect 46.0• PixLoc 40 63

Bala Lakshminarayanan, Mike McCullough

ARF Info

• Automatic target recognition working group Raster Format

• Contains header, sub headers, footers, 1 or more images

• Supports multiple frames

• Supports 16 different image types in same file

Bala Lakshminarayanan, Mike McCullough

ARF Info

Rows, cols, version, type, # frames, offset…

Colormap, comments

Image file

Bala Lakshminarayanan, Mike McCullough

ImageJ

• NVL used a plugin for converting image from .arf to other formats– Image processing and analysis in java

• Formats – dicom, pgm, jpg, bmp, tiff, raw…

• Operations – FFT, convolution, fractal box count, morphological…

Bala Lakshminarayanan, Mike McCullough

Acoustic Data

• Raw data in acoustic .dat file

• Contains header information for system time (similar to AGT), node #, longitude/latitude (0’s), and bearing (0’s)

• make_wave.py to convert from .dat to .wav– Changes to specify output file needed within

the python script to make it work properly– Script drops .dat extension and adds .wav

Bala Lakshminarayanan, Mike McCullough

Targets

• Honda CRX (target 1)

• Chevy Cavalier (target 2)

Bala Lakshminarayanan, Mike McCullough

Targets

• Toyota pickup (target 3)

• GMC pickup (target 4)

Bala Lakshminarayanan, Mike McCullough

Targets

• Vehicle (?) (target 5)– Names obtained from AGT files that would eventually

contain a TgtType indicating the target

• Toyota 4Runner (target 6)

• Stake body light truck (target 7)– Dave Rankins

Bala Lakshminarayanan, Mike McCullough

Example Data

• Convert arf files into raw using ImageJ

• Modify raw image into PGM– Switch endianness

• Apply image processing techniques to the image– Very hard to distinguish objects due to having a

dark image

Bala Lakshminarayanan, Mike McCullough

Example Sound Clips

• Normalized data using shareware program

• Target 1 – Node 1– Node 2– Node 3

• Target 6– Node 1– Node 2– Node 3

Bala Lakshminarayanan, Mike McCullough

Example Images (1)

• Sensor placement• FoV of a sensor covers atleast half of total FoV

Bala Lakshminarayanan, Mike McCullough

Example Images (2)

Bala Lakshminarayanan, Mike McCullough

Example Images (3)

Bala Lakshminarayanan, Mike McCullough

Example Images (4)

Bala Lakshminarayanan, Mike McCullough

Future Work

• MatLab acoustical analysis

• Segmentation & shape analysis

• Feature selection & extraction

• Fusion

• Target Recognition algorithms

Bala Lakshminarayanan, Mike McCullough

Questions?