What's with All This Talk About Coverage?
-
Upload
dvclub -
Category
Technology
-
view
120 -
download
0
Transcript of What's with All This Talk About Coverage?
© 2006 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice
What’s with all this talk about coverage?David Lacey and Rob PorterHewlett Packard CompanyJune 20, 2006
2 April 13, 2023
“You have this awesome generation that pseudo-randomly creates all sorts of good scenarios. You also have created equally awesome scoreboard and temporal checker infrastructure that will catch all the bugs. Next, you run it like mad, with all sorts of seeds to hit as much of the verification space as possible.”Peet James
Verification Plans
3 April 13, 2023
Given all that, what really happened?• Where did all those transaction go?• Which lines of RTL were exercised?• Which sections of the specification were tested?• Which corner cases of my implementation were hit?• What was the distribution of transaction types
issues?• Do I need to create new tests?• Can I stop running simulations?
Coverage helps provides the
answers!Coverage is a piece of the puzzle, not the final answer
4 April 13, 2023
Coverage provides…• An understanding of which portions of the
design have been exercised• Increased observability of simulation
behavior• Feedback on which tests are effective• Feedback to direct future verification
efforts
5 April 13, 2023
Agenda• Coverage terms and tools• How to get started• Coverage planning• Coverage execution• Coverage analysis• Coverage results
6 April 13, 2023
Coverage terms and tools
7 April 13, 2023
Coverage terms• Coverage strategy
− Approach defined to utilize coverage technology − Generate, gather, and analyze coverage data
• Coverage model− Collection of coverage spaces− Definition of one or more coverage spaces of interest
• Coverage space− Set of coverage points associated with a single aspect of the design and
a single coverage technology• Coverage technology
− Specific mechanism such as code, functional, assertion, transaction• Coverage point
− A specific named aspect of the design behavior− FCP, line of code, state transition, transaction or sequence of transactions
• Coverage data− Raw data collected from all coverage points and coverage spaces
• Coverage results− Interpretation of coverage data in context of coverage model
8 April 13, 2023
Coverage model vs. coverage tools
•WHAT−Coverage model High Level
ArchitectureSpecification
Design detailLow Level
TransactionBug rates
Code
Coverage
ModelFunctional
Assertion
Coverage tools
Sim cycles
•HOWCoverage tools
9 April 13, 2023
Code coverage• Line/block, branch, path, expression, state• Measures controllability aspect of our stimulus
−i.e. What lines of code have we exercised
• Does not connect us to the actual functionality of the chip −No insight into functional correctness
• Takes a blind approach to coverage (low observability)−Activating an erroneous statement does not mean the error will
propagate to an observable point during the course of a simulation
• Generates a lot of data−Difficult to interpret what is significant and what is not
10 April 13, 2023
Assertion coverage Assertions monitor and report undesirable
behavior • Ensures that preconditions of an assertion
check have been met
// SVA: if asserting stop or flush, no new requestassert property (@(posedge clk) disable iff (rst_n)
((Flush | SMQueStop) |->SMQueNew))
else $error(“Illegal behavior”); precondition
check
11 April 13, 2023
Functional coverage• Similar in nature to assertions
Assertions monitor and report undesirable behavior
Functional coverage monitors and reports desirable behavior
• Functional coverage−Specific design details
−Corner cases of interest to engineers
−Architectural features
12 April 13, 2023
Transaction coverage• A transaction is the logging of any data structure
−A packet on a bus−Does not have to be a system packet
• Example transaction coverage points−All transaction types were seen on each interface−Transactions with specific data were seen
• Source, destination, address, address ranges• Sequences of transactions
−Have recording monitor watch for sequence−Implement advanced queries to look for sequence
• Two parts to transaction coverage−Record the right data−Correct queries
13 April 13, 2023
EDA tools• Code, FCPs and transactions are recorded
into vendor specific databases−Tools are provided to look at coverage data−Report engines provide text reports
• Debug tools for FCPs and assertions• Tools to encourage coverage-driven
methodologies• Coverage is still a young technology
−Tools still expanding set of capabilities−Development areas such as data aggregation,
multiple view extraction
14 April 13, 2023
How do I get started with this coverage stuff?
15 April 13, 2023
Coverage roadmap – getting started
PSL / SVA / OVL
Coverage tools
Code / Assertion / FCP / Txn
TxnFCP
Code4. Collect data
1. Choose specificatio
n form
2. Identify
coverage model
3. Implement coverage
model
5. Analyze
data
6. React to data
Spec, design
Adjust stimulus
Planning
Execution
Consumption
16 April 13, 2023
Coverage planning
PSL / SVA / OVL
Coverage tools
Code / Assertion / FCP / Txn
Txn
FCP
Code4. Collect
data
1. Choose specification form
2. Identify
coverage model
3. Implement coverage
model
5. Analyze
data
6. React to data
Spec, design
Adjust stimulus
Planning
Execution
Consumption
17 April 13, 2023
Coverage planning
• Identify content of the coverage model−Coverage types to be used
• Identify required tools • Coverage infrastructure• Coverage execution• Maintenance and tool enhancements• Define coverage goals and metrics• Coverage reviews
Start looking at coverage up front!Coverage results only as good as coverage
model
18 April 13, 2023
Who, what, when, where, why• Who creates coverage model?
−Who analyses the data?
−Who owns coverage?
Add with RTLAnalyze continuously
Logic and DV engineers
Concern areas
spec, design, assertions, test plan
Because…
• What to cover in the model?
• When to add coverage points?−When to analyze coverage data?
• Where to look for ideas?
• Why mess with coverage?
19 April 13, 2023
For FCPs, ask yourself…−What should be covered?
−Where is best place to put FCPs?
−When to look for condition?
−Why have coverage point?
20 April 13, 2023
Watch out for…• Too much data
−Need information, not data−Need supporting tools to get
correct views of data• Ineffective use of coverage
−FCPs that fire every clock cycle−Duplication of coverage with
different tools• Reading too much into grading
tests−Random tests produce different
results with different seeds
21 April 13, 2023
Cost of coverage• Plan for the costs of using coverage
−Get solid infrastructure setup
−Plan for slower simulations
• Some level of cost is acceptable−Getting value back for investment
• Be smart−Architect coverage plan up front to ensure
success
22 April 13, 2023
Coverage execution
PSL / SVA / OVL
Coverage tools
Code / Assertion / FCP / Txn
Txn
FCP
Code4. Collect data
1. Choose specification form
2. Identify coverage model
3. Impleme
nt coverage
model
5. Analyze
data
6. React to data
Spec, design
Adjust stimulus
Planning
Execution
Consumption
23 April 13, 2023
Describing coverage model• Code coverage
−RTL code, pragmas
• Assertion and functional coverage−Use assertion language or library (PSL, SVA,
OVL)
• Transaction−Use hooks into Transaction Level Modeling
// PSL cover exampledefault clock = (posedge clk);sequence qFullCondition =
{reset_n ? (q_full : 1’b0);cover qFullCondition;
// SVA cover examplealways @(posedge clk) begin
if (reset_n)myQfull: cover (q_full)
$info (“queue was full”);end
24 April 13, 2023
Data collection• Collect data across volume simulation• Aggregate multiple databases• Location of coverage data repository• Manage volume of data
25 April 13, 2023
Coverage analysis PSL / SVA /
OVL
Coverage tools
Code / Assertion / FCP / Txn
Txn
FCP
Code4. Collect
data
1. Choose specification form
2. Identify
coverage model
3. Implement coverage
model
5. Analyze
data
6. React to data
Spec, design
Adjust stimulus
Planning
Execution
Consumption
26 April 13, 2023
The analysis• Easy to generate a
ton of data−Want information,
not data
Need to organize the data Can’t look at it all at
once Determine views
needed
27 April 13, 2023
Views of coverage data• Un-hit coverage• Functionality groups• Block, chip, system• Current milestone
functionality• Instance or module specific• Across environments, time,
model releases• Cross views
28 April 13, 2023
Our use of coverage• Aggregate data for each verification environment• Views: Verification effectiveness
−Verification environment• Views: TR readiness
−Major sub-blocks and chip• Filtering infrastructure
−Milestone specific functionality−Unreachable
• Aggregate coverage data across windows of time• Metrics provided for each team and full chips
29 April 13, 2023
Analysis is done… now what?• Understand all un-hit coverage• Fill coverage holes• Look for hard to hit coverage• Track coverage metrics
Don’t play games with metrics just to get coverage goals
met
Really understand the results
30 April 13, 2023
Coverage results PSL / SVA /
OVL
Coverage tools
Code / Assertion / FCP / Txn
Txn
FCP
Code4. Collect
data
1. Choose specification form
2. Identify
coverage model
3. Implement coverage
model
5. Analyze
data
6. React to data
Spec, design
Adjust stimulus
Planning
Execution
Consumption
31 April 13, 2023
Success stories• They exist!• Check them out in Assertion-Based Design
32 April 13, 2023
HP coverage data• SX1000 chipset
−6,500 FCPs• SX2000 chipset
−25,000 FCPs• Current efforts
−135,000 assertions and 650,000 FCPs−56,000 transaction points
• Coverage goals−100% coverage with understood exceptions−Team defined goals per milestone
33 April 13, 2023
Resources−J. Bergeron, Writing Testbenches: Functional Verification
of HDL Models, Second Edition, Kluwer Academic Publishers, 2003.
−H. Foster, A. Krolnick, D. Lacey, Assertion-Based Design, Second Edition, Kluwer Academic Publishers, 2004.
−P. James, Verification Plans: The Five-Day Verification Strategy for Modern Hardware Verification Languages, Kluwer Academic Publishers, 2004.
−A. Piziali, Functional Verification Coverage Measurement and Analysis, Kluwer Academic Publishers, 2004.
−B. Cohen, Using PSL/Sugar with Verilog and VHDL, Guide to Property Specification Language for ABV, VhdlCohen Publishing, 2003.
David Lacey, Hewlett Packard, [email protected]
Rob Porter, Hewlett Packard, [email protected]