Safe Testing of Autonomy in Complex, Interactive Environments (TACE) Flight Software Workshop...
-
Upload
gaige-shillington -
Category
Documents
-
view
229 -
download
0
Transcript of Safe Testing of Autonomy in Complex, Interactive Environments (TACE) Flight Software Workshop...
Safe Testing of Autonomy in Complex, Interactive Environments
(TACE)
Flight Software Workshop
December 16 – 18 2014
David Scheidt [email protected] Robert Lutz [email protected] William D’Amico [email protected] Harmalkar [email protected]
2
Outline
Test & Evaluation Need and Challenges
Background and Definitions
Current Effort under TRMC’s T&E/S&T program
Path to a Testing Capability
Summary
3
From Tele to Auto to Autonomous Operations
Unmanned systems usually require the operator to be a “user-in-the-loop”
APL has demonstrated the autonomous operation of multiple and cooperative UXVs where the operator is a “user-on-the-loop”
There have been many demonstrations of APL’s Mission Level Autonomy (MLA) under JHU/APL internal research and at Camp Roberts (CA)
MLA has been shown to be “platform” and “vendor” agnostic riding above proprietary control/guidance algorithms
MLA research led to insights on what would be needed for rigorous test and evaluation
UXV
Sensor Control
Autonomous Operations
Operationalobjectives
Measure effect
Measureeffect
Operatingenvironmentaldecisions
A measure of effectiveness of an autonomous system is directly related to
“operational objectives” thus making the test methodology and test results applicable to
both developmental and operational test environments – true joint DT/OT is needed
4
Systems with Autonomous Behaviors Are Here
ACTUV: ANTI-SUBMARINE WARFARE CONTINUOUS TRAIL
UNMANNED VESSEL
AMAS: Autonomous Mobility Appliqué System
AACUS: Autonomous
Aerial Cargo/Utility
System
LDUUV: Large Diameter UUV
5
Autonomy Test & Evaluation Challenge
Autonomous systems respond to unpredictable change by devising a course of action. Before we deploy such systems how can we be sure that autonomous decisions that will be produced will always: (1) achieve objectives set by human supervisors and (2) not produce unacceptable unintended consequences. Testing autonomous systems is particularly challenging since we cannot possibly test all interactions between the autonomous system and the natural world.
6
1970 1990 2000
Air-Exjam(later EXDRONE and BMQ-147 Dragon)
Scheer “Beast”
Autonomy S&T Vision Program
Predator/SSN Interoperability Demo
NEAR burn failureDARPA UUV Program
History of Autonomous Unmanned Systems at APL
1965 20102005
4 UGVs @ APG
2 UAVs & 4 UGVs
@ APG
1 UAV 1 UGS @ APG
Tactical Sensing 1 UAV & Multiple UGS
Unmanned SurfaceVehicles
3 UAVs, 2 UGS,3 Mobile Users
@ Dugway & DOE Range
TNT Experiments6 UAVs, 3 UGS
@ Camp Roberts
1 UUV
2015
OPISR – 4 UAVs, 1 UGV, 2 USV, 1 UUV, 3 UGS
@ Webster
Small Ocean-going USVs
In Atlantic, Pacific & Gulf of Mexico
JHU/APL has been developing autonomous unmanned vehicles since 1962.
Since 2002 JHU/APL has conducted dozens of autonomous vehicle flight programs and flown/launched hundreds of sorties. Currently JHU/APL has 24 active autonomy programs.
JHU/APL Proprietary
New Horizons
TACE
DADFS
Agile UxVAutonomy
7
Combined Autonomous Air/Ground Missions2004 Aberdeen Test Center
OSD NII Swarming Unmanned Vehicle Experiment – Cooperative Search, Patrol & Track (4 UGVs and 2 UAV)
Combinations of autonomous air and ground vehicles (an AACUS/AMAS collaboration) are clearly possible and probably required for many military missions
8
Challenges for Autonomous System Test
• Test & Evaluation (T&E) Need- Build an infrastructure for SAFE/LIVE testing of autonomous unmanned vehicles
(AUVs), especially for tactical UAVs that have significant airspace and platform-related range safety issues
- Demonstrate live, virtual, and constructive (LVC) methods for SAFE/LIVE tests
- LVC methods should be Test and Training Network Architecture (TENA) compliant
- To our knowledge there are no AUV established T&E capabilities with these features
• T&E Challenge– Provide an infrastructure that supports safe testing when testing autonomous systems
operating over the horizon or in denied environments.– Provide an infrastructure that supports safe testing when autonomous systems perform
unpredictable actions– Provide an accurate, real-time, live-virtual-constructive environment that interacts with the
system under test’s autonomy in unpredictable ways.
– Reliable command and control (C2) of the systems under test (SUT) and robust 2-way data links are needed to stimulate the SUT, to record behaviors, and to maintain SAFE/LIVE test control
9
Safe Testing of Autonomy in Complex, Interactive Environments (TACE) – Key Elements
Capability TACE is a 3-phase/36-month program (started in April 2013) where the open/general
architecture will be at TRL6 with initial capabilities for the testing (black and white box) of autonomous unmanned vehicles (AUVs) of all types. TACE is “portable” but will rely upon a “thin client” interface to the SUT and the inclusion on the SUT of a minimal “TACE test applique.” There are 3 levels of control/override within TACE: SUT autonomy, TACE automatic experiment controls, test range safety officer control
Relevance to T&E Scenarios The TACE architecture will use rigorous techniques to prevent unsafe AUV operations or
actions while stimulating, measuring, monitoring, and recording the AUV’s autonomous response/performance in complex environments. TACE assurance algorithms provide mathematically rigorous, on-board, real-time, safety guarantees, while TACE dual simulation system provides a complex live-virtual constructive test environment.
Cost Benefits A critical element, as always, is selecting the critical tests. TACE is not a planning tool.
A recent award was made to JHU/APL under the Unmanned and Autonomous System Test (UAST) program for an autonomy planning tool, Rapid Adversarial Planning Tool (RAPT). The “cost” of conducting tests where the “Achilles heel” of the embedded autonomous behavior is unknown would involve very high risk to the loss of the SUT and the delay of system fielding. The combination of tools such as RAPT and TACE should prevent unnecessary and expensive testing
10
TACE System Architecture
11
TACE System Under Test (SUT )Payload
TACE’s payload for the Boeing ScanEagle is a repurposing of JHU/APL’s Autonomy Toolkit (ATK) software, which has been used on more than a dozen types of UAVs in Boeing, Lockheed, Aerovironment manufactured vehicles, and hardware previously developed jointly under internal research funds by Boeing and JHU/APL
TACE SUT payload installed in a Boeing ScanEagle payload bay
ATK payload with Persistent Systems Wave Relay Wireless
Local Area Network Card
12
TACE Flight Tests at Aberdeen Test Center
APL Test Team on the tarmac at Phillips Army Airfield (PAAF) Aberdeen Test Center (ATC)
Hand launch of the Procerus research AUV controlled by
JHU/APL’s Autonomy Tool Kit (ATK)
Five Test Events with Multiple Sorties Were Executed during January/February 2014
13
Sample Results from TACE Testing at ATC
Constraint Violation - Range Safety Executive User Interface
SUT Moving NoGo Virtual Target Safe Loiter Point Fixed NoGo Area
14
Path to a Testing Capability – Phase 2 OV1
Test Director Test
Ground Station
Intruder C2
Restricted Airspace(s)
UnicornUAV
Range Safety
ScanEagle SUT-System Under Test
Native UHF Link
Wave Relay
2400MHz
Live target track to SUT
Intruder GS
UHF Safety Link
Land Lines
Onboard Test
Payload
Autonomous BehaviorsSearch, Track, Avoid
Watchdog Behaviors3D Range BoundariesPlatform Safety Limits
Virtual JIMM Entities 1 to N
Virtual Translated Air Traffic
Live Unicorn UAV
Defense Research & Engineering Network Connection
Tactical EO/IR
UHF PNT Downlink
Test Range Topology
ScanEagle as the SUT at Yuma Test Center
15
10 Miles
1.0Mile
2.5 Miles
7.5 Miles
GeographicConstraintBoundaries
1.0Mile
1.0 Mile
2.0 Miles
Notional Playbox at a Large Test Range
Flight Test Operations
Base
16
Proximity Violation (Virtual Entity)
TrackEstablished
Violation
LiveGroundEntity
VirtualAircraft
Not
e: p
aths
sho
wn
are
not
actu
al p
aths
)
17
Summary for Autonomy and T&E Perspectives
TACE Phase 3 will provide some basic operational functions at TRL 6 – this is an “initial architecture” but not a “range capability”
To our knowledge, there are no “active” programs to extend beyond TRL6 – highly understandable since there are no PORs with autonomous behaviors
At the recent NDIA Annual T&E conference – SHIFT LEFT is coming RFPs will include CONOPS and initial TEMPs - will “autonomy” be a “shift left” – Insights from Dr. Brown, Director TRMC - cited “cyber, big data, hypersonic, and
autonomy” as the T&E challenge areas for the near future
The NATO Science and Technology Organization (STO) promotes and ... Validation and Verification of Autonomous Systems (SCI-274)
18
Acknowledgements
• JHU/APL TACE Team: Robert Chalmers, Robert Bamberger, Kristine Ramachandran, Dean Kleissas, Brendan John, Subodh Harmalkar, William Van Besien, Michael Biggins
• TACE Subcontractors to JHU/APL :
– Trideum: Michael O’Connor
– Boeing Corporation: Gabriel Santander
– InSitu Inc.: Jerry McWithey
• Government Flight Test Support:
– Aberdeen Test Center: John Wiley
– Yuma Test Center: Mary Beth Weaver and Bob Vondell
• Unmanned and Autonomous Systems Test Test Technology Area
– Executive Agent: Vernon Panei
– Deputy Executive Agent: Stephanie Riddle
– Subject Matter Experts: William Hamel, Kirk Bonnevier