© 2017 Arm Limited Taiwan 2017
Using Virtual Platforms To Improve Software
Verification and Validation Efficiency
Odin Shen | Staff FAE | Arm
Arm Tech Symposia
© 2017 Arm Limited 3
Software Costs Increasing
Primary Design Costs By Process Node
The cost of developing and qualifying software increases rapidly with higher complexity IC designs
High-performance SoCs include multiple processor cores and 100s of IP blocks
Source: IBS 2013
© 2017 Arm Limited 4
Software Verification and Validation
Traditional flows involve development followed by testing.
Modern practices such as Continuous Integration and Extreme Programming call for continuous development & test cycles.
• Develop -> Commit -> Test cycles daily, or several times a day
Frequent commits help reduce the number of conflicting changes.
Frequent testing gives clues about where and when the error was introduced.
© 2017 Arm Limited 5
Continuous Integration
Automation Server, e.g. Jenkins
• Commit in a version control system
• Test run scheduling
• Triggered after the other builds in the queue have completed
Scripting
• Configure target
• Connect to target
• Initiate tests
• Stop target and query state of memory, registers, etc
Control and Reporting
• Reset, results presentation, etc
Target
© 2017 Arm Limited 6
Automation is Key
Efficient Continuous Integration programs require good automation.
• Integration with common automation platforms such as Travis, Jenkins and AppVeyor
• Kick off jobs as soon as a commit is made
• Overnight testing
• Results collection and presentation
• Robust, stable, automatable target platforms
© 2017 Arm Limited 7
Targets For Your CI System
Continuous software integration enforces fail-fast, but you have to have the right infrastructure.
Key component of testing software on Arm is a fast, available, cheap, automatable target.
Let’s look at target options for a CI system…
© 2017 Arm Limited 10
Variables to Consider
Accuracy – How much accuracy do you need? What is more important, accuracy or speed?
Performance – What kind of dependency do you have on performance? Can your testing finish within the desired time window?
Debug and Analysis – How stable is your design? How much debug do you expect? How much visibility into hardware/software interface do you need?
Availability – Are the targets available? Are there sufficient quantity to scale out to a large test environment?
Capacity – How big is your design? Will it fit within the target’s capacity? Does cost scale with the capacity required?
Cost – Have you considered scaled out cost? Infrastructure cost? Setup cost? Maintenance cost?
Flexibility – What is turnaround time for changes?
© 2017 Arm Limited 11
Platforms Based on Simulation
If you don’t need cycle accuracy, most will agree that a simulated target is preferable.
• Minimal infrastructure
• Easy to maintain
• Easy to deploy
• No capacity issues
• No expensive probes
• Easily automated
• Excellent debug capabilities
Some testing will require cycle accuracy, but testing functional correctness, software-software and software-hardware correctness don’t.
© 2017 Arm Limited 12
Standard Project Flow Without Virtual Platform
Manufac-turing
SoC Hardware Development
Spec Freeze Tape Out Project FinishedSilicon
Software Development, HW/SW Integration & System Validation
ArchDesign
Hardware–Software Co-design
Gained TTM
Manufac-turing
Standard Project Flow With Virtual Platform
SoC Hardware Development
Spec Freeze Tape Out Silicon
ArchDesign
Software Development, Integration & System Validation
Project Finished
© 2017 Arm Limited 13
Armv8 “Juno” test chip bring-up timeline
Development on virtual platform before siliconDevelopment on hardware
2 years 10 days
Full software stackand tools validated
on hardware
Virtual Prototype Availability
Fast Models
Arm DS-5 Arm TrustZone
© 2017 Arm Limited 14
The Real Value of Simulation Platforms
Better debug capabilities. Why? Visibility and control.
• Non-intrusive, deterministic debug
• Control, visibility and determinism are guaranteed
• Immediate halt of entire system on watchpoint or breakpoint
• No expiring watchdogs or transient states
• Capture state for testing, then carry on running as if the stop never occurred
• VP Debug callback
• Checkpointing
• Reverse execution
• Built in or 3rd party tools
© 2017 Arm Limited 15
Scripting Example
Model connection and configuration
Set breakpoints
Run the application
Get status of variables, registers, memory, etc.
Variable status determines test pass/fail
© 2017 Arm Limited 16
Debug Example
Virtual Platforms provide almost unlimited tracing capabilities. Virtually every signal, register, memory or CPU program flow can be traced.
• No need for special hardware or software instrumentation
A user can define his/her own monitors and query information about OS state and threads to give more context when printing out results.
A watchpoint can be set on a signal between a core cluster and a subsystem.
Callbacks from the model can non-intrusively report system state.
Stop on access
© 2017 Arm Limited 17
Virtual Prototype Availability
Cycle Accurate Debug Interface (CADI) defines debug interface to model.
Model Trace Interface (MTI) defines trace interface to model.
CADI and MTI are open and freely available.
– Enables a debug user experience similar to hardware debug.
– Scripting interface for advanced capability.
Source Level Debugger
© 2017 Arm Limited 18
Biggest Challenge with Virtual Platforms
Where’s the model?
• Companies like Synopsys, Arm, Intel have large modelling teams who create models of their IP
• But what about the rest of the system: UART, keyboard, display, speaker, etc?
• Someone has to model them, or, you exclude those peripherals from your testing
• As hardware evolves, the models must also evolve
© 2017 Arm Limited 19
Arm Models
Arm Models come from various places.
• Fixed Virtual Platforms (FVP)
• Roll your own with Fast Models, OVP, QEMU
• Ecosystem partners
• Silicon partners
The closer to the actual system, the better.
What if I don’t have a full system simulation?
• Test what you can
• Stub out code
© 2017 Arm Limited 20
Regression Testing
Regression farms also benefit from models.
Automated defect and performance regression testing.
Same advantages as CI scenario.
• Easy to setup and deploy
• Highly automatable
• Superior debug-ability
• Easy to maintain
Performance Regression Testing – Android Boot
© 2017 Arm Limited 21
Success Stories
Linaro
• System Guidance
• Armv8-A FVP
Customers
• Intel, Qualcomm, Hewlett Packard, Texas Instruments, Cirrus, Nokia, Ericsson, ST, HiSilicon, Samsung, and many more
Arm internal usage (compiler, models, system software, IP development and testing)
Arm System Guidance Platform
© 2017 Arm Limited 23
NN on Arm?• Deploy NN inference on Cortex v8.2 platform w/o real hardware
• How to select solution?
Accuracy PerformanceDebug and AnalysisAvailabilityCapacityCostFlexibility
© 2017 Arm Limited 24
Use MNIST as Example• The MNIST database of handwritten digits, available from this page, has a training set
of 60,000 examples, and a test set of 10,000 examples.
• It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.
http://yann.lecun.com/exdb/mnist/https://www.tensorflow.org/get_started/mnist/beginners
© 2017 Arm Limited 25
How To Deploy on Arm?• Purpose:
• Prototyping on Arm platform
• Early exploration NN engine design
• SW framework profiling
• Script
• Load image
• Load parameter
• Automation
• Profiling
.....
…
6
AMBA®4 ACE or AMBA5 CHI
SCU
Cortex-A55ARMv8.2-A
32b/64b CPU
L1 cache
Core 5
Shared L3 cache
7 8
ACP
Multicore Debug and Trace
Private L2 cache
Snoop Filter Async BridgesPeripheral Port
…
2
Cortex-A75ARMv8.2-A
32b/64b CPU
L1 cache
Core 13 4
Private L2 cacheDebugTrace
Streamline
Trace and Debug
© 2017 Arm Limited 27
Let’s See• DS5 live demo
• Variables to Consider:
Accuracy PerformanceDebug and AnalysisAvailabilityCapacityCostFlexibility
.....
…
6
AMBA®4 ACE or AMBA5 CHI
SCU
Cortex-A55ARMv8.2-A
32b/64b CPU
L1 cache
Core 5
Shared L3 cache
7 8
ACP
Multicore Debug and Trace
Private L2 cache
Snoop Filter Async BridgesPeripheral Port
…
2
Cortex-A75ARMv8.2-A
32b/64b CPU
L1 cache
Core 13 4
Private L2 cacheDebugTrace
Streamline
Trace and Debug
VV
VV
VV
V
© 2017 Arm Limited 29
Summary
Growing software complexity is creating the need for new test strategies.
Continuous Integration enables “fail-fast”, greatly reducing software integration efforts.
Models are easy to automate and provide extended access and visibility into the system, are more reliable, and easier to maintain than hardware.
Freely available Arm system models at https://developer.arm.com/products/system-design/fixed-virtual-platforms with software stacks available from Linaro(www.linaro.org)
Looking for more about the demo?
Check the git and free feel to contact [email protected] for the real demo.
3131 © 2017 Arm Limited
The Arm trademarks featured in this presentation are registered trademarks or trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners.
www.arm.com/company/policies/trademarks
Top Related