Multimodal sensors &

digital interfaces

CreditsThe original Multimodal Project developed by and credit for:

Zhigang Zhu and Weihong Li (Integration of Laser Vibrometry with Infrared Video for Multimedia Surveillance Display)

Outline• Multimodal System Overview• Multimedia Sensors

• Infrared camera • The LDV sensor• PTZ camera

• Multimodal System Components• System Design Concept• Design Issues• Integration Issues

Multimodal System Overview• The object of this system is to

provide a multimodal integration of audio, visible, thermal for human signature detection.

• The goal is to use the sensing technologies for perimeter surveillance.– Sensors, alarm, response.

• Multimodal system interface – The environment, the sensors,

and the events.

Multimedia Sensors – Infrared CameraInfrared camera

– FLIR ThermoVision A40M IR camera

– Temp Range of -20º to 500ºC, accuracy (% of reading) ± 2ºC or ± 2%

– 320x240 Focal Plane Array– 24º FOV Lens– Firewire Output IEEE 1394– Video output – RS170 EIA/NTSC or CCIR/PAL

composite video for monitoring on a TV screen

– ThermoVision System Developers Kit (C++)– Each thermal image is built from 76,800

individual picture elements that are sampled 60 times per second by the camera's on-board electronics.

Multimedia Sensors – Infrared CameraSamples

Figure 1. A person sitting in dark room can be clearly seen in the IR image. The temperature at Sp1 on the face is 33.1ºC

Figure 2. Two IR images before and after a person standing at about 200 feet. The reading of the temperature at Sp1 changes from 11C to 27C.

Multimedia Sensors – Infrared CameraThermographic measurement techniques• An infrared camera measures and images the

emitted infrared radiation from an object.• The radiation measured by the camera does not

only depend on the temperature of the objects but is also a function of the emissivity.

• Radiation also originates from the surroundings and is reflected in the object

• Radiation from the object and reflected radiation will also be influenced by the absorption of the atmosphere

• Parameters need to take care:– The emissivity of the object– The reflected temperature– The distance between the object and the camera– The relative humidity

Multimedia Sensors – Infrared CameraEmissivity:- How much radiation is emitted from the

object- Object materials and surface treatments

exhibit emissivity ranging from approximately 0.1 – 0.95

- Highly polished (mirror) surface < 0.1- Human skin exhibits an emissivity close to 1- Metal: low, increase with temperature- Non-metal: high, decrease with temperature

Multimedia Sensors – Infrared CameraReflected ambient temperature:- To compensate for the radiation reflected

in the object and the radiation emitted from the atmosphere between the camera and the object.

- If the emissivity is low, the distance very long and the object temperature relatively close to that of the ambient it will be important to set and compensated for the ambient.

Distance- The distance between object and the front

lens of the camera.

Relative Humidity- Normally, default 50%

Multimedia Sensors – Infrared CameraHistory of Infrared Technology- Sir William Herschel (1738-1822)

- Discover of infrared spectrum

- Marsilio Landriani (1746-1815)- As the blackened thermometer was moved

slowly along the colors of the spectrum, the temperature readings showed a steady increase from the violet end to the red end.

- Macedonio Melloni (1798-1854)- Rock salt (NaCl) (to be made into lenses and

prisms) is remarkably transparent to the infrared.

- Sir John Herschel- The first ‘heat-picture’ in 1840, thermograph

- Samuel P. Langley (1834-1906) - Inventor of the bolometer (1880)

Multimedia Sensors – LDV sensorVibrometer types:- Single Point Vibrometers:

- Measure the vibration of an object in the direction of laser beam

- Differential Vibrometers: (dual beam)- Allow vibration measurement between two

points vibrating relative to each other.

- Rotational Vibrometers:- Measure angular vibrations on rotating

structures.

- In-plane Vibrometers:- measure continuous (DC) velocity and

superimposed variable (AC) components perpendicular to the central axis of two converging laser beams.

- 3D Vibrometers

Multimedia Sensors – LDV sensorLaser Doppler Vibrometer (LDV)- Optical instruments for accurately measuring

velocity and displacement of vibrating structures completely without contact.

- Sensor head OFV-505- HeNe laser, = 633.8 nm.- OFV-SLR lens (f=30mm) 1.8m – 200+m, auto

focus

- Controller OFV-5000 with a digital velocity decode card VD-6 - RS232 interface for computer control

- Telescope VIB-A-P05- ±1º vertical tilt and ± 1.5º horizontal tilt

Multimedia Sensors – LDV sensorMeasurement Principle

S is the light source

f is frequency

P is the moving with velocity v and reflects the light

O is the receiver (f + fD)

Resultant frequency shift

For vibrometers: S=O ("backscatter") 1= - 2, therefore,

Multimedia Sensors – LDV sensorLDV schema

- Velocity is directly obtained by demodulation: = 2fm

- Voice frequency f: 300Hz ~ 3000Hz- LDV can detect vibration at a

magnitude as low as m = /2f = 1/(2*3.14*300) = 0.5µm

Multimedia Sensors – PTZ cameraPan/tilt/zoom (PTZ) camera - Human and other target detection at a

large distance- Canon PTZ

- 26X optical zoom lens & 12X digital zoom

- Pan: ±100º, Tilt: +90º/-30º

- Built-in IR light (effective up to 9 feet)- BNC video output- RS-232 computer control interface

Multimedia Sensors – PTZ cameraPTZ Samples

Two images of a person at a distance of about 200 feet, captured by changing the zoom factors of the PTZ camera.

Multimodal System Components• Three components

– The IR/EO imaging video surveillance component

• Human motion tracking, human face detection

• Thermal Camera for daytime and nighttime

• Visible camcorder (sony), Web cam (logitech)

– The LDV audio surveillance component

• Audio signal capture, voice recognition

– The human-computer interaction component

• Cognitive understanding of the environment, the sensors, and the events

System Design Concept• The overall goal the project is to design a

human computer interface for human-centered multimodal (MM) surveillance.

IR Imaging

Human/motion detection

LDV point select/track

Face Detection Recognition

LDV Pointing

Audio signal detection

Voice Extract Analysis

Voice Recognition

Human ID and Event Understanding

Sensor Registration and Modeling HCI VE navigation AR: VE +video Target visuali-zation Face ID display

HCI LDV points Visuali-zation MM: video +audio voice synthesis Voice ID diaplay

Design IssuesIssues need to be considered

– how to use EO camera tracking human motion;

– how to incorporate IR imaging with existing EO captured image;

– how to use IR imaging to help the laser Doppler vibrometer to select the appropriate targets;

– how to select optimal viewpoint from audio detection.

Integration Issues- Target detection and localization

via IR/EO imaging- Set up an IR/EO imaging system with an

IR camera and a PTZ camera for finding vibration targets for LDV listening

- Registration between the IR/EO imaging system and the LDV system.- Two types of sensors need to be

precisely aligned so that we can point the laser beam of the LDV to the target that the IR/EO imaging system has detected

- Future research on automated targeting and focusing.

References- Main multimodal system technical

report- http://www-cs.ccny.cuny.edu/~zhu/LDV/

FinalReportsHTML/CCNY-LDV-Tech-Report-html.htm

- Polytec Laser Vibrometer- http://www.polytec.com/

- FLIR Systems Security ThermoVision Cameras- http://www.flir.com/

- Paper:- Z Zhu, W Li, “Integrating LDV Audio and IR

Video for Remote Multimodal Surveillance”

- Others…

(END)