CDR Presentation FINAL - University of Arizona...3 Requirements Overview Elliott Liggett 7o 6f 7...
Transcript of CDR Presentation FINAL - University of Arizona...3 Requirements Overview Elliott Liggett 7o 6f 7...
11/16/2010
1
Machine Vision and Autonomy Integration into an
Unmanned Aerial System
Critical Design Review
1 of 67Hans Hony Elliott Liggett Michael Palmer James Dianics Malcolm Gibson
November 18, 2011
Presentation Objectives
• Project Introduction
• Requirements Overview
• Design Overview
• High-level System Design
Strategic Test Plan
2 of 67
• Strategic Test Plan
• Bill of Materials
• Project Timeline
• Risk Analysis
• Review and Conclusion
Introduction Requirements Design Overview Strategic Planning Conclusion
The AIR Team
Project Introduction
The University of Arizona’s Aerial Robotics Club has sponsored us to help them design and build an aerial vehicle for competition in the
2011 AUVSI Student Unmanned Aerial Systems Competition
The competition vehicle is currently being developed by two teams:
The AVIONICS Team
3 of 67
• Composed of Aerospace Seniors
• Aerodynamic Design & Analysis
• Airframe Design
• Composites Manufacturing
• Flight Testing
• Avatar UAV Platform
• Composed of ECE, OSE, SYE, and ME seniors.
• Autopilot Integration into Aircraft
• Computer Vision System
• Image Acquisition System
• System Integration
(Avionics Module)
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
2
Project Introduction
The 2011 AUVSI Student Unmanned Aerial Systems Competition
Key Performance Parameter
Percent of Score
Autonomy Up to 30%
Imagery Up to 20%
Competition Overview:
• June 16-19th 2011
• Sponsored by industry leaders
• Oral Presentation
S t S f t O i
4 of 67
Imagery Up to 20%
Target Location Up to 20%
Mission Time Up to 10%
In-flight Re-tasking Up to 5%
2010 AUVSI SUAS Competition Breakdown of Flight-Test Score
• Systems Safety Overview
• Pre-Mission Briefing
• Flight Test Evaluation
• Mission Efficiency
• Performance
Introduction Requirements Design Overview Strategic Planning Conclusion
Project Introduction
Mission Breakdown
5 of 67
Webster Annex Field in Maryland
Introduction Requirements Design Overview Strategic Planning Conclusion
Project Introduction
Key Mission Objectives
Autonomy
• Take-Off and Landing
• GPS Waypoint Navigation
• Avoid designated no-fly zones
• In Flight Re Tasking
6 of 67
• In-Flight Re-Tasking
Machine Vision
• Accurately resolve 4’x 4’ targets
• Size, Shape, Location, Colors
• Alphanumeric Character
• Pop-up Target
• Operate autonomously
Camera System
• Remain orthogonal to ground
• Capture clear images
• Sufficient resolution
• Transmit data to ground station
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
3
Requirements Overview
Elliott Liggett
7 of 67
Project Introduction
Must:
Produce an aircraft and machine vision system.
• Aircraft must be capable of flight for 30 minutes under autonomous control
• Must positively identify all target characteristics:
8 of 67
o Color
o Shape
o Alphanumeric Character
Introduction Requirements Design Overview Strategic Planning Conclusion
Project Introduction
9 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
4
Changes since PDR
1. After analyzing several lens and camera options: o Two cameras would be required for wide-
angle (120º) view.o Reduces lens distortions to imageo Increases GIFOV
Recent Changes
10 of 67
2. Since flight time and bandwidth are coupled to power:o Requirements increase dramatically with
slight changes to path. o Thus an optimum path algorithm will be
created.
Introduction Requirements Design Overview Strategic Planning Conclusion
Trade Studies
James Dianics
11 of 67
Trade Studies
Camera System
12 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
5
Camera System
Trade Studies
13 of 67
Resolution vs. FOV
Introduction Requirements Design Overview Strategic Planning Conclusion
Camera System
Trade Studies
14 of 67
Resolution and FOV
Introduction Requirements Design Overview Strategic Planning Conclusion
Weight, Size and Power
Camera System
Trade Studies
15 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
6
Field of View and Resolution
Camera System
Trade Studies
16 of 67
Low Weight and Power Consumption
Introduction Requirements Design Overview Strategic Planning Conclusion
Processing Speed and Compatibility
Onboard Computer
Trade Studies
17 of 67
Weight, Power Consumption, and Size
Introduction Requirements Design Overview Strategic Planning Conclusion
Software Design
Hans Hony
Malcolm Gibson
18 of 67
Malcolm Gibson
Elliott Liggett
Michael Palmer
11/16/2010
7
System Design Overview
19 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Autopilot Concept of Operation
Control Loop Theory:
20 of 67
Control Loop Theory:
• Input – Transfer Function – Output
• Feedback Loop
• Poles and Zeros of the system
• Operational range of the various gains
Introduction Requirements Design Overview Strategic Planning Conclusion
OL
OLGain A
AK
1
Autopilot Concept of Operation
21 of 67
Pros of Closed Loop Controls:
• Less Gain Optimizing and more programming
• Bonus!
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
8
Autopilot Concept of Operation
22 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Autopilot Concept of Operation
23 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Using the GPSData_t Class:
• Access to Latitude, Longitude and Altitude
• Ground Speed and Direction
• Timestamp information
• Double Bonus!
Software Design
Hans Hony
Malcolm Gibson
24 of 67
Malcolm Gibson
Elliott Liggett
Michael Palmer
11/16/2010
9
Machine Vision Process
ma
ges
Gimbaled Camera System
25 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
FitPC Onboard ComputerOpenCV
Co
mb
ine
Im
USB
Filter Image
Machine Vision Process
Read image into the Matlab
W k
26 of 67
Identify Object LocationCrop Region of Interest
Send Image and info to Ground Station
Workspace
Introduction Requirements Design Overview Strategic Planning Conclusion
Machine Vision Process
C
B
1
A
me
ric D
ata
ba
se
Compare to characterdatabase
OUTPUTShape &
Character To USER
Determine Target Shape
Determine Target Character
27 of 67
Perform Analysis on Binary Image
Convert RGB Image to BW
Sh
ap
e D
ata
ba
se
Generate Shape Signature
Binary Im
age
Extract Alphanumeric Character
Compare to Shape database
2
3
Alp
ha
nu
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
10
Machine Vision Process
Target Area ( )ft3 Original Image
Target Background Color
OUTPUT Target Area, Color, and Location
To USER
Original Filtered Image
28 of 67
Remove large objects
Perform “blob” analysis on image
DeterminePixel Area (px)
Determine pxCentroid (X,Y)
AircraftGPS Coordinates(from autopilot)
length( ft) f (px, altitude)
Target GPS Location(orientation)
Extract RGB ValuesFrom Centroid
Introduction Requirements Design Overview Strategic Planning Conclusion
Software Design
Hans Hony
Malcolm Gibson
29 of 67
Malcolm Gibson
Elliott Liggett
Michael Palmer
System Software Integration
while (in_air) {
In-Air Image Acquisition Heartbeat
30 of 67
• Images and commands sent via HPN-SSH
}...
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
11
System Software Integration
Ground Station Image Processing Flow
31 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Software Design
Hans Hony
Malcolm Gibson
32 of 67
Malcolm Gibson
Elliott Liggett
Michael Palmer
Interface Design
33 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
12
Interface Design
34 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
• Easy to Customize GUI
• TCP / IP Network Interface
• Consumer/Producer Queue
Interface Design
Ground Station Interface
35 of 67
Q
• Report Generating Toolkit
• Printer Drivers
Introduction Requirements Design Overview Strategic Planning Conclusion
• Shape Recognition
• Optical Character Recognition (OCR)
Interface Design
Machine Vision Station
36 of 67
• Target Matrix
• Confirmation Countdown
• Return Targets to Labview Ground Station
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
13
• Device to World
o Lab View GUI
o Final Report
Interface Design
Hardware Level
37 of 67
• Device to Human
o Matlab GUI
o Target Recognition
o Confirmation Countdown
Introduction Requirements Design Overview Strategic Planning Conclusion
Analysis and Testing
Michael Palmer
Elliott Liggett
38 of 67
Elliott Liggett
Hans Hony
Malcolm Gibson
Camera Analysis
• Camera Calculations
o Field of View
o Pixel Area of Target
o Optical Overlap
o Resolution (in. per pixel)
LAARK Toolbox
39 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
( p p )
o Lens Error
o Resolution Error
• To Meet Specifications
o Greater than 5MP
o 4.5mm to 6mm C-Mount lens
11/16/2010
14
Camera Analysis
40 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Analysis and Testing
Michael Palmer
Elliott Liggett
41 of 67
Elliott Liggett
Hans Hony
Malcolm Gibson
Data-link Analysis
FSPLdB 20 log10 d 20 log10 f 36.58
Free-Space Loss:
Property Value
FSPL at 2.43GHz and 1 mile radius ‐104.0dB
Signal Strength
42 of 67
• More negative numbers indicate lower signal strength, thus lower speed.
Transmit Strength + 24.0dBm
Ground Station Antenna Gain + 10.0dBi
Aircraft Antenna Gain + 2.7dBi
Loss in Cables (total both ends) ‐ 8.0dB
Total System Losses ‐75.3dB
Express Range 2 performance ‐77.0dB @ 54Mbps
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
15
Data-link Analysis
Signal Strength
43 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Data-link Analysis
Bandwidth
• Must supply the ground with all photos within mission timeframe.
44 of 67
Property Value
Estimated Photos ~ 3MB @ 500 photo = 1.5GB
Transfer rate of 10Mbps ~ 20 minutes 17 seconds
Transfer rate of 54Mbps ~ 3 minutes 45 seconds
Total Mission Time Allowed 40 minutes
Decreasing Scores Result at 30 minutes or more
Introduction Requirements Design Overview Strategic Planning Conclusion
Completed Testing
Thus far…
• Tested machine vision options:o Compiled several test OpenCV appso Preliminary image target detection in Matlabo Image display in LabView
45 of 67
• Image transfer and remote commands:o Tested SSH session sharing (faster connections)o Tested remote command cuing at 20 commands per secondo Tested High Performance SSH (“HPN-SSH”)
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
16
Analysis and Testing
Michael Palmer
Elliott Liggett
46 of 67
Elliott Liggett
Hans Hony
Malcolm Gibson
Design vs. Requirements
Camera Design
Status Requirements
Yes 120° Field Of View (FOV).
Yes FPS faster rate than speed of aircraft and GIFOV between photos.
Yes USB or Ethernet Connected Device
47 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Yes USB or Ethernet Connected Device
Yes Linux Drivers
Design vs. Requirements
Onboard PC
Status Requirements
Yes Small Form Factor.
Yes Onboard PC able to process images and transmit data efficiently.
Yes Onboard storage capable of meeting high demand of photos
48 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Yes Onboard storage capable of meeting high demand of photos.
Yes Expandable to interface with peripherals not accounted previously.
11/16/2010
17
Design vs. Requirements
WiFi Card
Status Requirements
Yes WiFi capable of at least one mile transmission.
Yes Small form factor and compatible with onboard PC.
Yes Free (Although not required)
49 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Yes Free… (Although not required)
Design vs. Requirements
Overall System
Status Requirements
Yes Machine Vision able to perform in under 30 minutes.
Yes Onboard payload less than 3lbs.
Yes Power consumption manageable
50 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Yes Power consumption manageable.
Analysis and Testing
Michael Palmer
Elliott Liggett
51 of 67
Elliott Liggett
Hans Hony
Malcolm Gibson
11/16/2010
18
Test Plan
Optics Test Plan
1. Finalize optics configuration
2. Purchase cameras and lenses
3. System compatibility test
4. Determine optimum overlap ratio
52 of 67
p p
a. Angle between cameras
b. Confirm 120 FOV requirement
5. Integrate into the Aero-dome
6. Measure and quantify distortion
7. Confirm weight and size requirements
8. Aerial Photography test
Introduction Requirements Design Overview Strategic Planning Conclusion
Test Plan
Machine Vision Test Plan
1. Test camera/software interface
2. Construct mock targets
3. Generate simulated images for machine vision testing
4. Prototype algorithms
53 of 67
yp g
a. Image stitching
b. Filtering
c. Object analysis
d. Information output
5. Confirm reliability in determining target characteristics
6. Determine computational power and processing speed requirements
Introduction Requirements Design Overview Strategic Planning Conclusion
Test Plan
Systems Integration Test Plan
1. Test camera/piccolo/computer interface
2. Test gimbal control loop
a. Test response speed
b. Tune sensitivity to eliminate noise
3. Ensure reliable connections between
54 of 67
components
4. Test for any RF interference
5. Test Battery life under various operating conditions
a. Full power draw
b. Estimated mission load
6. Test software integration
a. OpenCV > Matlab > Labview User 1Safety Pilot
User 2Target Analysis
User 3Autopilot Control
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
19
Test Plan
Data-link Test Plan
1. Fly express range wifi card in sponsor’s aircraft to perform rigorous data-link test
2. Measure bandwidth
3. Measure range
55 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
4. Test repeatability / reliability
5. Test potential weather effects
6. Optimize antenna configuration
7. Develop failure protocols to ensure safety and recovery in the event of a connection loss
8. Perform range/reliability tests with Piccolo ground-station
Bill of Materials
Hans Hony
56 of 67
Bill of Materials
57 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
20
Bill of Materials
58 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Bill of Materials
Component Timeline Status
Fit‐PC2 (Onboard PC) 4 to 5 weeks Ordered
Express Range 2 WiFi Card and antennas 2 weeks Arrived
UI‐1495LE‐C (Camera) + 4.5mm Lens 2 weeks TBA
Component Lead Times
59 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
UI 1495LE C (Camera) 4.5mm Lens 2 weeks TBA
Gimbal mounts, servos, and dome 2 weeks TBA
Second Piccolo Autopilot TBA TBA
Additional Ground Computer TBA TBA
Risk Analysis and Future
James Dianics
60 of 67
11/16/2010
21
Risk Analysis
Identified Risks
Risk Severity Likelihood Risk Factor
Optical Dome 0.60 0.30 0.18
Data Link 0.40 0.20 0.08
61 of 67
System Stability 0.80 0.30 0.24
Physical Crash 0.75 0.20 0.15
Data Loss 0.10 0.50 0.05
Introduction Requirements Design Overview Strategic Planning Conclusion
0.4
0.5
0.6
od
Risk Analysis
Highest Risk Factors
Rank Risk
1 System Stability
2 Data‐link
3 O ti l D
Risk Analysis
62 of 67
0
0.1
0.2
0.3
0 0.2 0.4 0.6 0.8 1
Likelihoo
Severity
3 Optical Dome
4 Physical Crash
Introduction Requirements Design Overview Strategic Planning Conclusion
Risk Mitigation
• System Stability:o Testing, Testing, Testingo Watchdog Functionso Robustness
Data link
Risk Analysis
63 of 67
• Data-link:o Preliminary Testingo Reduce Data Transfer
• Optical Dome:o Testing
• Physical Crash:o Reduce Flight Time
Introduction Requirements Design Overview Strategic Planning Conclusion
11/16/2010
22
Gnatt Chart
Future Planning
64 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
Next Steps
Future Planning
• Purchase Specified Components
– Imaging System
– Piccolo II
• Initial Configuration Testing
• Software Development
65 of 67Introduction Requirements Design Overview Strategic Planning Conclusion
p
– Open Source
• Optical Character Recognition
• Shape Detection
– Closed Source
• LAARK Server & Watchdog
• Autopilot Ground Station
• Machine Vision Station
• Test & Refine Algorithms
Conclusion
~ Thank you for your attention ~
Questions or Comments?
66 of 67
Presentation By:
Hans Hony Elliott Liggett Michael Palmer James Dianics Malcolm Gibson
Introduction Requirements Design Overview Strategic Planning Conclusion