Heuristic Stimuli Generation For Coverage Closure ... · [email protected]. GOAL • To...
Transcript of Heuristic Stimuli Generation For Coverage Closure ... · [email protected]. GOAL • To...
Heuristic Stimuli GenerationHeuristic Stimuli GenerationFor Coverage Closure For Coverage Closure
Exploiting Simulation FeedbackExploiting Simulation FeedbackGiovanni Squillero
Politecnico di Torino - ItalyCAD Group ( )
GOALGOAL
• To propose a methodology for coverage-directed stimuli generation based on simulation feedback
• Such stimuli could be added as new content to improve existing validation suites
DVClub 2010-01-18 [email protected] 2
AcknowledgementsAcknowledgements
• Danilo Ravotto• Ernesto Sanchez• Matteo Sonza Reorda• Alberto Tonda
• + many others
DVClub 2010-01-18 [email protected] 3
OutlineOutline
• Proposed methodology• Case studies• Conclusions
DVClub 2010-01-18 [email protected] 4
Design ChoicesDesign Choices
• Being able to tackle real problems• Develop a versatile and broadly applicable
methodology – Compatible with different environment– Compatible with any coverage metric
• Minimize effort to change goal/target– Exploit common aspects
DVClub 2010-01-18 [email protected] 5
Feedback-Based ApproachFeedback-Based Approach
• Simulation-based approach• Exploits feedback from simulation • Incremental improvement/refinement of the
solution (trial-and-error)• Trade-off between computational resources
and confidence• May exploit heuristics or problem-specific
knowledge
DVClub 2010-01-18 [email protected] 6
Proposed MethodologyProposed Methodology
DVClub 2010-01-18 [email protected] 7
StimuliGenerator
System
Stimuli
Feedback
StimuliStimuli
• Sequences of bits• Sequences of keys • Full fledged assembly language programs• VHDL test case• External world• …
DVClub 2010-01-18 [email protected] 9
Stimuli GeneratorStimuli Generator
DVClub 2010-01-18 [email protected] 10
StimuliGenerator
System
Stimuli
Feedback
Stimuli GeneratorStimuli Generator
• Exploit an Evolutionary Algorithm to generate stimuli to maximize a given function
DVClub 2010-01-18 [email protected] 11
Evolutionary AlgorithmsEvolutionary Algorithms
• Meta-heuristic optimization algorithm based on the concept of population and exploiting some principles of natural evolution
DVClub 2010-01-18 [email protected] 12
Evolutionary AlgorithmsEvolutionary Algorithms
• Succession of random and deterministic steps– A systematic way of throwing dices– Better than pure random
“The great effect produced by the accumulation in one direction, during successive generations, of differences absolutely inappreciable by an uneducated eye”
DVClub 2010-01-18 [email protected] 13
Evolutionary AlgorithmsEvolutionary Algorithms
• Population– Multiple solutions considered in each step– More resistant than pure hill-climbing– Different solutions may interbreed
DVClub 2010-01-18 [email protected] 14
Evolutionary AlgorithmsEvolutionary Algorithms
• Why using an evolutionary algorithm?– Adaptative– Able to find unexpected solutions– Better than randomBetter than random
DVClub 2010-01-18 [email protected] 15
Evolutionary AlgorithmsEvolutionary Algorithms
• Problem: Fitness function– GOAL: Optimize the wheel– FITNESS: Minimize the number of “bumps”
DVClub 2010-01-18 [email protected] 16
Evolutionary AlgorithmsEvolutionary Algorithms
• Problem: Black magic
DVClub 2010-01-18 [email protected] 17
µGP (MicroGP)µGP (MicroGP)
• CAD Group general-purpose evolver– 3 versions (only 2 released under GPL)– Project started in 2002– 11 developers + contractors, students, …
• Current version– ≈ 300 file, > 40,000 lines in C++
DVClub 2010-01-18 [email protected] 18
µGP (MicroGP)µGP (MicroGP)
Evolutionary Optimization: the µGP toolkitE. Sanchez, M. Schillaci, G. Squillero
Springer, 2010
ISBN: 978-0-387-09425-0
DVClub 2010-01-18 [email protected] 19
µGP (MicroGP)µGP (MicroGP)
DVClub 2010-01-18 [email protected] 20
http://ugp3.sourceforge.net/
MicroGP++ (aka. ugp3, µGP3)• Information• Download• Credits
System & FeedbackSystem & Feedback
DVClub 2010-01-18 [email protected] 21
StimuliGenerator
System
Stimuli
Feedback
SystemSystem
• Strongly problem dependant• Model via simulation/emulation– HDL (netlist to high-level)– HW accelerated (e.g., exploiting FPGA)– Architectural simulator– ISA simulator
• Real device
DVClub 2010-01-18 [email protected] 22
Feedback (examples)Feedback (examples)
• From simulation– Code coverage metrics (e.g., instruction coverage)– HW specific metrics (e.g., toggle coverage)– High-level information (e.g., FSM coverage)
• From the real system– Performance counters– Physical measures (e.g., temperature, time, power
consumption)
DVClub 2010-01-18 [email protected] 23
OutlineOutline
• Proposed methodology• Case studies• Conclusions
DVClub 2010-01-18 [email protected] 24
Design VerificationDesign Verification
• Devise a set of programs maximizing different coverage metrics
DVClub 2010-01-18 [email protected] 25
FeedbackFeedback
• Code coverage metrics– Statement coverage (SC)– Branch coverage (BC)– Condition coverage CC)– Expression coverage (EC)
• HW specific metric– Toggle coverage (TC)
DVClub 2010-01-18 [email protected] 26
SystemSystem
• DLX/pII– Cleaned and simplified MIPS intended primarily
for teaching purposes– 32-bit load/store architecture, 5-stage pipeline– VHDL RTL description• 4,558 statements• 3,695 branches • 193 conditional statements (1,764 expressions)• 8,283 logic bits
DVClub 2010-01-18 [email protected] 27
Experimental ResultsExperimental Results
DVClub 2010-01-18 [email protected] 28
Experimental ResultsExperimental Results
DVClub 2010-01-18 [email protected] 29
230Kinstructions
2,978instructions
Experimental ResultsExperimental Results
DVClub 2010-01-18 [email protected] 30
20h 33h
27h
37h
95h
≈1 week
Post-silicon VerificationPost-silicon Verification
• Generate functional test programs for post-silicon verification
• Activity performed in collaboration with the ETM Group, Intel (Phoenix)
• Intel Pentium 4– 42-55 millions of transistors– 2GHz clock– NetBurst architecture
DVClub 2010-01-18 [email protected] 31
System & FeedbackSystem & Feedback
• Real Pentium 4 (no simulation) • Performance counters as metric– Introduced in 1993 in IA32 architecture – 48 event detectors + 18 programmable counters– Instruction-tagging (for discriminating non-
speculative performance events)
DVClub 2010-01-18 [email protected] 32
Experimental ResultsExperimental Results
• Two sets of experiments:– Mispredicted branches over the total branches– Clock cycles in which the trace cache is delivering
µops to the execution unit instead of decoding or building traces (hard metric!)
• Intrinsic features of the µarchitectural design• Time (both): 12h/program
Haifa 2008 G. Squillero 33
Mobile PhoneMobile Phone
• 12m-activity performed in collaboration with Motorola Research Center in Turin (“Automation of Current Drain Measures”)
DVClub 2010-01-18 [email protected] 34
SystemSystem
• Real prototypical mobile phone– P2K platform– GSM/3G networks– Complex applications
• Radio Communication Analyzer, anechoic chamber
• Controllable power supply• Custom hardware
DVClub 2010-01-18 [email protected] 35
StimuliStimuli
• Sequence of keys• Power supply• Network status
DVClub 2010-01-18 [email protected] 36
FeedbackFeedback
• Physical measures– Power consumption (+ autocorrelation)– Network status
• FSM transition coverage
DVClub 2010-01-18 [email protected] 37
Experimental ResultsExperimental Results
• Major bugs discovered in the prototype– Test infrastructure– LCD display– Fake deep-sleep state
DVClub 2010-01-18 [email protected] 38
OutlineOutline
• Proposed methodology• Case studies• Conclusions
DVClub 2010-01-18 [email protected] 39
ConclusionsConclusions
• Simulation-based & feedback-based• Evolutionary algorithm• Broadly applicable• Human resources– Limited (set-up )
• Computational resources– Easily parallelizable (generation level)– Trade-off quality vs. effort
DVClub 2010-01-18 [email protected] 40
ConclusionsConclusions
• May exploit other methodologies– Useful starting point– Effective completion
• May be coupled with other methodologies– Rule-based instruction randomizers– Simulation-based approaches
DVClub 2010-01-18 [email protected] 41