Image processing for tactical UAV · Image processing for tactical UAV. ... Ku-band data link...
Transcript of Image processing for tactical UAV · Image processing for tactical UAV. ... Ku-band data link...
1
Branche DBranche Dééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Image processing for tactical UAVImage processing for tactical UAVImage processing for tactical UAVM Broekaert, D Duclos
et M Sirieix
RTO SET SYMPOSIUM
Advanced Sensor Payloads for UAV
LISBON, April 2005
Report Documentation Page Form ApprovedOMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.
1. REPORT DATE 01 MAY 2005
2. REPORT TYPE N/A
3. DATES COVERED -
4. TITLE AND SUBTITLE Image Processing for Tactical UAV
5a. CONTRACT NUMBER
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Directeur Technique et Scientifique SAGEM SA Défense et Sécurité LePonant de Paris 27 rue Leblanc 75512 Paris Cedex 15 FRANCE
8. PERFORMING ORGANIZATIONREPORT NUMBER
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release, distribution unlimited
13. SUPPLEMENTARY NOTES See also ADM202032., The original document contains color images.
14. ABSTRACT
15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT
UU
18. NUMBEROF PAGES
37
19a. NAME OFRESPONSIBLE PERSON
a. REPORT unclassified
b. ABSTRACT unclassified
c. THIS PAGE unclassified
Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18
2
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
OUTLINEOUTLINE
SAGEM Tactical UAVs
Imagery chain
3 rd generation IR imaging Payload
Image processing
Automatic target recognition
Target localization
Conclusions
3
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
SAGEM : EUROPEANSAGEM : EUROPEAN LEADER FOR UAVLEADER FOR UAV
BATTLE & FIELD BATTLE & FIELD -- PROVENPROVENMOSTMOST ORDEREDORDERED SYSTEM IN EUROPESYSTEM IN EUROPE
•• FRENCH ARMY (2+1+4 systems),FRENCH ARMY (2+1+4 systems),•• DUTCH ARMY (4 systems),DUTCH ARMY (4 systems),•• SWEDISH ARMY (3 systems),SWEDISH ARMY (3 systems),•• DANISH ARMY (6 systems),DANISH ARMY (6 systems),•• GREEK ARMY (3 systems)GREEK ARMY (3 systems)
ININ FULL RATEFULL RATE PRODUCTIONPRODUCTION
1993199319951995
1997199719991999
20012001 20022002
From Swedish Lapland Operation ...From Swedish Lapland Operation ...
…… to Saudi Arabia Operationto Saudi Arabia Operation
4
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
MAIN SPERWER SYSTEM CHARACTERISTICSMAIN SPERWER SYSTEM CHARACTERISTICS
THE MISSIONS ?Artillery (30m CEP)Intelligence
HIGHLY TACTICAL
•• Catapult launchCatapult launch
•• Parachute recoveryParachute recovery
THE SYSTEM Ground SegmentLauncherAVs
NATO INTEROPERABLE•• C4I integrated (C4I integrated (AdatAdat P3)P3)
•• KuKu--band data link (STANAG 7085)band data link (STANAG 7085)
•• Multiple payload possibleMultiple payload possible
•• Hand Over between ground stationsHand Over between ground stations
•• 2 aircraft simultaneously2 aircraft simultaneously
FLEXIBLE CAPABILITIES
5
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Tactical UAV MissionTactical UAV Mission
Control Transfer
Launch and RecoveryStation
Station 1
Station 2Recovery
400 km
400 km
6
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
SPERWER - LONG ENDURANCESPERWER - LONG ENDURANCE
Endurance 12+ hoursEndurance 12+ hours
Stealthy (RCS, IR, acoustic)
Airworthiness
Robust delta design
Anti icing
Typical time to reach 2000m < 10 mn
Stealthy (RCS, IR, acoustic)
Airworthiness
Robust delta design
Anti icing
Typical time to reach 2000m < 10 mn
SPERWER - STANDARDSPERWER - STANDARD
MTOW 350 kg
Span 6.5 m
Speed 148 km/h
Ceiling 6000 m
Mission radius 200 km
SPERWER AIR VEHICLESSPERWER AIR VEHICLESCommon avionics
Common data link
Common avionics
Common data linkCommon ground segment
Common logistic support
Common ground segment
Common logistic support
MTOW 330 kg
Span 4.2 m
Speed 150 km/h
Ceiling 4500 m
Mission radius 200 km
Endurance 6+ hours
7
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
IR PAYLOAD MUST BE ADAPTED TO THE MISSION IR PAYLOAD MUST BE ADAPTED TO THE MISSION AND THE AIRCRAFT SPECIFICSAND THE AIRCRAFT SPECIFICS
IRLS 8 IRLS 8 -- 12 12 µµ
8 - 12 µ IRIS 2nd Generation & IR 3 - 5 µ MATIS 3rd Generation IR Camera
OLOSP & ARTEMIS
Stabilised gimbal
8
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Payload system control and Payload system control and monitoringmonitoring Image analysisImage analysis
Payload Control Panel
TV SELECT
FOCUS∞ AV1
TV Control
VIDEO OFF
ACTIVE
IR SELECTE. ZOOM ENABLE INV POLAR
AV2
PLD ON
FOCUSZOOM / FOV
C-WTC-BK
SCEN
GAIN / LEVEL AUTO
Image Acquisition
LOS Control
COLOR
Overlay PCP Control
Tracker
Video Selection
PLD Mode Control
PLD ControlAV Request
STOW LOCKED
OPERATE RATE
FIX FORWARD
TEST LIGHT
Sensor Control
E
IR C ontrol
ORBIT CENTER
CONTRAST
GAIN
+
GAIN -
LEVEL
+
LEVEL -
TV ON IR ON AUTOTEST PLD
AUTOTRACK TOP POS SPEED
TRK ALGO TARGET
Tracker Control
Payload Control Panel
PO station
Mission Exploitation (Image analysis)
MP station
Maps display Payload
Video display
DATA BASE
9
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
MATIS 3MATIS 3--5 5 µµm CAMERAS m CAMERAS
BINOCULARS MANPORTABLE /BINOCULARS MANPORTABLE / LONG RANGELONG RANGESTANDARDSTANDARD
10
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
SELECTION CRITERIAS for IR PAYLOAD SELECTION CRITERIAS for IR PAYLOAD
Compactness to be fitted into a platform with other equipment’s (visible camera, Laser Range Finders/Laser Designator,..)Consistency with the mission and the environmentImage quality (high definition)Sensitivity (low NETD)Maturity of technology Affordable cost
11
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
ATMOSPHERIC TRANSMISSION ATMOSPHERIC TRANSMISSION Influence of the Relative HumidityInfluence of the Relative Humidity
Visibility : 25 kmTransmission 1 km,Altitude 0 mTemperature : 20°CPressure 1013 mbar; 1013 hPa
Visibility : 25 kmTransmission 10 km,Altitude 0 mTemperature : 20°CPressure 1013 mbar; 1013 hPa
12
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
DIFFRACTIONDIFFRACTION
Angular Resolution is Limited by Diffraction: α
= 1.22 λ
/ Ø
α
: angular resolution λ: wavelength Ø: pupil diameter
2 possibilities: Increase Ø : Ineffective as already limited by
the available space
Decrease λ : λ : move from 8-12 µm to 3-5 µm thermal imager, X2.5 benefit in term of resolution
13
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Background Background limitedlimited NETD for a 2 D NETD for a 2 D ArrayArray
Nmax : Maximum storagecharge in ROIC capacitance (Nb of electrons)CΔT : Thermal contrast in the bandwidth
maxTmin N
2C
1NETDΔ
= ( )
( )λ∂
λ∂
=Δ ,TLT
,TL
CB
B
B
14
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
MATIS image MATIS image processingprocessing
Usual : non uniformity correction:Usual : non uniformity correction:Gain tableGain table
Shutter to show a uniform sceneShutter to show a uniform scene
Improvement : use of optical flow estimate Improvement : use of optical flow estimate from scene and camera movementfrom scene and camera movement
⇒⇒ Better image quality Better image quality
⇒⇒ New image processing capabilitiesNew image processing capabilities
15
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Optical flowOptical flow
In a visible image sequence , the value of the intensity In a visible image sequence , the value of the intensity I(x,yI(x,y) at a point ) at a point x,yx,y of the image plane may fluctuate of the image plane may fluctuate because of two phenomena :because of two phenomena :
A movement of the scene objects or a movement of A movement of the scene objects or a movement of the camera,the camera,
An intensity variation of the scene objectsAn intensity variation of the scene objects
The optical flow detection consists in finding the The optical flow detection consists in finding the velocity field of the video sequence elements in the velocity field of the video sequence elements in the image plane.image plane.
16
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Optical flowOptical flow
pixels movement 2D Fieldpixels movement 2D Field
Basic equation : Basic equation :
0dt t)y,I(x, t)y,dy(x,t)y,I(x, t)y,dx(x,t)y,I(x,=
∂∂
+∂
∂+
∂∂
tyx
17
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Hypothesis : Hypothesis : The scene optical flow is principally linked to camera The scene optical flow is principally linked to camera
movementsmovementsIn the IR, intensity is a combination of incoming flux In the IR, intensity is a combination of incoming flux
(through the lens) and a parasitic flux ((through the lens) and a parasitic flux (eg narcisuseg narcisus). ). To simplify the equation, we can make two To simplify the equation, we can make two
observations concerning the optical flow of the IR observations concerning the optical flow of the IR camera :camera :
There is no fast movements for the parasitic There is no fast movements for the parasitic radiationradiationIntensity variations are linked to thermal Intensity variations are linked to thermal
phenomena, and consequently have time constants phenomena, and consequently have time constants that are broadly superior to the image periodicity.that are broadly superior to the image periodicity.
IR IR Optical Optical fflow : low : algorithm descriptionalgorithm description
18
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
IR IR Optical Optical fflow : low : algorithm descriptionalgorithm description
– Image pre processing : to avoid aliasing
– Gradient calculation
– Use of a predictive model to reduce the intensity variation
– Select N more convenient pixels
– We express movement in the focal plane as a function of 3 angles (roll, yaw and pitch), 3 translations and a zoom factor by a projection in the image plane of the movement model
– These expressions are injected in the Optical flow equation
– We solve the N equations by least mean square estimate
19
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Application of optical flow processingApplication of optical flow processing
IR image without any correction
• the non-uniformity correction of the IR sensor• the stabilisation in the terrestrial co- ordinate system, or the filtering of the carrier vibrations• the extraction and the tracking of the mobile objects in the scene• the generation of hyper resolved images combining over sampling and restoration
The suggested processing permits to achieve successively or simultaneously the following functions :
20
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Optical flow Optical flow => Calculate 3 angles camera rotation=> Calculate 3 angles camera rotation
algorithm descriptionalgorithm description
NUC 1 pointNUC 1 point
StabilisationStabilisation OverOver--samplingsampling
Mobile target detectionMobile target detection hyperhyper--resolutionresolution
21
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Stabilisation of IR imageStabilisation of IR image
Stabilised and Referenced movie
Original movie Filtered and stabilised movie
22
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
IncreasedIncreased resolutionresolution with motionwith motion
Hyper-resolution
23
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Image Processing ChainImage Processing ChainLow LevelProcessing
Hot spots detection
Edge extract.
Region segmentation
Texture analysis
High Level Processing
Correlation
Voting process
Bayes Class.
Fussy Logic
Fusion (s)
ANN
3D position estimation
Time Processing
Kalman filtering
Tracking / Association
Calibration -Restauration
Contrast Enhancement
Dead pixel correction
Wiener filter
Optical Flow Motion
AnalysisMoving Target Detection
Monocular Stereo
Calibration with OF
Stabilisation
Deblurring
Hyper- resolution
Sensor / Navigation Fusion
24
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Automatic Target Detection, Recognition, Automatic Target Detection, Recognition, Identification and Localization Identification and Localization
Automatic generic target detection and recognition :Automatic generic target detection and recognition :vehicle type target : plane, tank, truck, vehicle type target : plane, tank, truck, ……
infrastructure target : bridge, building, infrastructure target : bridge, building, ……..
MultiMulti--bands, multibands, multi--contexts target identificationcontexts target identificationPrecise complex target localization helped with Precise complex target localization helped with
targettarget’’s area models area model
25
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Generic Target Detection and Generic Target Detection and RecognitionRecognition
Detection
Best Alarm
Nav. Information
IMAGES
AlarmsSegmentation
AlarmsClassification
GenericTargetsModels
Alarms selection
Preliminary designation :Target type
Raw localization
UAV Flight
Target’s precise
localization
26
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Generic Target Detection and RecognitionGeneric Target Detection and Recognition
« Airplane » target « Helicopter» target
27
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
« Tank » target « Truck » target
Generic Target Detection and RecognitionGeneric Target Detection and Recognition
28
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Generic Target Detection and RecognitionGeneric Target Detection and Recognition
Low LevelFeatures Extraction
Alarms Selection
Selected3D Generic
Model
IMAGES
Projected 3D Model
Navigation Information
Research of Model’s Optimal Parameters
DETECTIONS
Alarms Notation and selection
3D GenericModels
29
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
« Bridge » target « Bulding » target
Generic Target Detection and RecognitionGeneric Target Detection and Recognition
30
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
3D Models: geometry (facet) + radiometry: (emissivity, temperature)
Shapes for each position:distance/targetSynthetic Images
(Ray tracing,sensor
model)
Models Data Base
Target Segmentation
Image: distance to edge
Shape recognition
00,5
1
1,52
2,53
3,54
4,5
1 31 61 91 121 151 181 211 241 271 301 331 361 391 421 451 481 511
pré classe classe type
Notes and decisions
OFF-LINE
ON-LINE
Target IdentificationTarget Identification
31
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Target IdentificationTarget IdentificationAMX 10MARDER
Input Image Identification result Input Image Identification result
32
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
TARGET LOCALIZATIONTARGET LOCALIZATIONPerformance Performance constantconstant and and independentindependent from GCS/AV from GCS/AV distancedistanceAutomaticAutomatic real timereal time calculation (no human interpretation)calculation (no human interpretation)Target coTarget co--ordinatesordinates available from video and still imagesavailable from video and still imagesRelative coRelative co--ordinates : ordinates : Accuracy 6 mAccuracy 6 m
1/2 angularcone
65°
10°
TargetLocation Error
< 50 m
< 20 m< 20 m
(CEP 50, altitude 1000 m)
65°
AccurateAccurate33--axes INSaxes INS
and attitude referenceand attitude reference
AV AV LocalisationLocalisation
Ground stationGround stationLocalisationLocalisation
NAVNAV
PLDPLDPositionPosition
LOSLOSOrientationOrientation
TargetTargetLocalisationLocalisation
Accurate LOSAccurate LOSposition measurementposition measurement
Payload / InertialPayload / InertialNavigation UnitNavigation Unit
boresightingboresighting
DGPSDGPS
GPS,GPS,Land NavigationLand Navigation
Systems, ...Systems, ...
DigitalDigitalTerrainTerrain
ElevationElevationDataData
33
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
Target localization improvement by image processingTarget localization improvement by image processing
In-flight processingTARGET
+ENVIRONMENT
Mission planning
Observation Means
Observation Means
InputimagesDTED
Automatic + manual offline modeling process
Target’s area model
Target’s area modelingTarget’s area modeling
Onboard IR sensor Onboard IR sensor
IR images
Low level image processing
Low level image processing
Extracted features
Automatic real-time modeling process
Localisation data
High level matching process
High level matching process
Precise Target localisation
34
BrancheBranche DDééfensefense et et SSéécuritcuritéé
s
ee
20855266-
ConclusionsConclusions
IR image sequence processing can offers new IR image sequence processing can offers new capabalitiescapabalities for target detection.for target detection.Motion is usable to make over sampling and provides Motion is usable to make over sampling and provides better resolution with 2D arrays by aliasing reduction.better resolution with 2D arrays by aliasing reduction.Improved target detection, recognition and Improved target detection, recognition and identificationidentificationPrecise target localisation by mixed processing Precise target localisation by mixed processing between navigation system, image between navigation system, image datasdatas and terrain and terrain modelling modelling
To day, large FPGA permits fast design of hardware To day, large FPGA permits fast design of hardware implementation from new algorithmsimplementation from new algorithms
BrancheBranche DDééfensefense et et SSéécuritcuritéé
TC X
XXX
s
ee
02/08/2000 3520855266-
Questions ?Questions ?Questions ?
R
UNCLASSIFIED/UNLIMITED
Image Processing for Tactical UAV
M. Broekaert, D. Duclos and M. Sirieix* * Directeur Technique et Scientifique
SAGEM SA Défense et Sécurité Le Ponant de Paris
27 rue Leblanc 75512 Paris Cedex 15
FRANCE
This paper was received as a PowerPoint presentation without supporting text.
Broekaert, M.; Duclos, D.; Sirieix, M. (2005) Image Processing for Tactical UAV. In Advanced Sensor Payloads for UAV (pp. 11-1 – 11-2). RTO Meeting Proceedings RTO-MP-SET-092, Paper 11. Neuilly-sur-Seine, France: RTO. Available from: http://www.rto.nato.int/abstracts.asp.
TO-MP-SET-092 11 - 1
UNCLASSIFIED/UNLIMITED