Transcript of 12005 MAPLDDesign Integrity Concepts. 22005 MAPLDDesign Integrity Concepts Unit Agenda Consistent...
- Slide 1
- 12005 MAPLDDesign Integrity Concepts
- Slide 2
- 22005 MAPLDDesign Integrity Concepts Unit Agenda Consistent
terminology, consistent results Introduction and definitions What
does it have to do? Specifying the design Letting constraints work
for you Proportional design More than the sum of its parts
Partitioning a design You mean were still working on it? Sustaining
a design Whats the exit strategy? Verifying a design What do you
mean, it doesnt do what we thought? Validating a design
- Slide 3
- 32005 MAPLDDesign Integrity Concepts Who is this guy? John
Stone Chief Engineer (Department of Space Systems SwRI ) 18 years
experience (Exclusively space-flight) C&DH design (hardware and
software) Instrument electronics design Box-level integration and
test Hardware project management Product assurance (parts,
radiation, reliability) Process improvement Broad interest in
improving what we do
- Slide 4
- 42005 MAPLDDesign Integrity Concepts What do we want to
accomplish? Discuss techniques that may help us to do our jobs
better Based on general principles Broad applicability (logic,
board, box, system) Foster necessary discussion / brainstorming To
extent possible (as time allows) No one person has all of the
answers Suggestions appreciated
- Slide 5
- 52005 MAPLDDesign Integrity Concepts Consistent Terminology,
Consistent Results Introduction and Definitions
- Slide 6
- 62005 MAPLDDesign Integrity Concepts Agenda Introductions and
Definitions Design integrity A working definition Why is this
important? A tale of two designs Has anything changed? Why should I
care? The miracle cloud The alternative Overview Implications
Additional Definitions Summary
- Slide 7
- 72005 MAPLDDesign Integrity Concepts Definition and Goal Design
The invention and disposition of the forms, parts, or details of
something according to a plan. (AH dictionary online) Integrity The
state of being unimpaired; Soundness; The quality or condition of
being whole or undivided; completeness (AH) This seminar is
intended to talk about techniques and issues that ensure the
soundness and completeness of both the end product and the means
used to produce it.
- Slide 8
- 82005 MAPLDDesign Integrity Concepts Design 1 Radarsat 1
ACP
- Slide 9
- 92005 MAPLDDesign Integrity Concepts Radarsat 1 ACP Overview
Program dates: late 1990 late 1992 Specifications Processor: 8 MHz
80386/80387 Memory:128 k x 8 SRAM, 192kx8 EEPROM, 16k x 8 PROM
Interfaces: A/D (16), D/A (4-12), Synchronous serial (3 input, 3
output), RS-232 GSE Function: Attitude control processor for the
RADARSAT1 satellite Logic Implementation MSI logic + PALs (16L8,
16R6, 16R8) Additional functions (cross-strap, control)
external
- Slide 10
- 102005 MAPLDDesign Integrity Concepts PAL Reminder
- Slide 11
- 112005 MAPLDDesign Integrity Concepts Design 2 - Command
Telemetry Board (CTB)
- Slide 12
- 122005 MAPLDDesign Integrity Concepts CTB Overview Program
Dates: early 2001 late 2003 Specifications: Processor: RTX2010 (16
MHz) Memory: 4M x 8 (random), various FIFOs (16k x8) as necessary
Interfaces MIL-STD-1553B Synchronous Serial (command / telemetry)
Analog input High-level discrete (output) Low-level discrete (input
/ output) Functionality: S/C command/telemetry (level 0 and active)
Logic Implementation: 4 54SX32S FPGAs Additional resources (in same
box): RAD 750, Mass Memory, Instrument Interface Card
- Slide 13
- 132005 MAPLDDesign Integrity Concepts Whats Changed? Capability
/ Complexity Logic Density Specificity RADARSAT (1 small
specification with interface focus) CTB (1 large specification with
interface, s/w, operations focus) Software Centricity Initial
Errors RADARSAT: 3 jumpers; 1 PAL design change CTB: 14 FPGA
revisions 2 spec change 5-6 mistakes 6-7 data dependency
- Slide 14
- 142005 MAPLDDesign Integrity Concepts Whats Not Changed?
Overall program schedules Proportional budget Expectation of
correctness Pain from mistakes
- Slide 15
- 152005 MAPLDDesign Integrity Concepts What Explains the (error)
Difference? Engineers arent as capable? Insulting! Everything is
just more complex? Copout! Methodology? Methodology hasnt changed
Always inadequate, we just got lucky Adequate for old designs, no
longer adequate Methodology has changed Used to be adequate, but we
lost the recipe Design philosophy of systems has changed?
Predicated on maximum flexibility Expectation of extreme complexity
Over-specification almost impossible to meet
- Slide 16
- 162005 MAPLDDesign Integrity Concepts What Do These Examples
Illustrate? The incidence of initial correctness for designs seems
to be decreasing Design changes seem to be more common Problems
late in the verification/validation cycle seem to be more frequent
Perhaps a combination of the factors presented explains this, but
Desired complexity is not going to decrease Budgets are not going
to get bigger The expectation of excellence isnt going to go away
The only solution is to develop and improve a consistent
methodology for implementing robustly designed products Based on
basic principles Applicable to a variety of conditions
- Slide 17
- 172005 MAPLDDesign Integrity Concepts Why Should I Care? Why do
we work? Self-actualization (fun, monetary reward, interest) Why do
people want us to work for them? They need what we produce What do
people want engineers (especially in Aerospace) to produce? A
quality product that satisfies the customers needs How do they want
us to produce such a product? Consistently and efficiently
- Slide 18
- 182005 MAPLDDesign Integrity Concepts The Laymans View The
Miracle Cloud
- Slide 19
- 192005 MAPLDDesign Integrity Concepts The Miracle Cloud Method
Note that too many engineering schools teach this approach without
meaning to Advantages to the miracle cloud method Total creative
freedom Disadvantages to the miracle cloud method Product quality
is variable Team makeup dependent Team mood/morale dependent
(Monday morning car) Luck dependent Product is not produced in a
repeatable manner Product is not produced in an efficient manner
Result Quality low Customer Satisfaction Low
- Slide 20
- 202005 MAPLDDesign Integrity Concepts How Do We Replace the
Miracle Cloud? Provide structure to the development effort Evaluate
the effort and the product produced Improve the effort and the
product Repeat
- Slide 21
- 212005 MAPLDDesign Integrity Concepts Definitions of Importance
From Q9000-2000 Process A set of interrelated and interacting
activities which transforms inputs to outputs [in our case ideas to
devices] Product The result of a process
- Slide 22
- 222005 MAPLDDesign Integrity Concepts Implications From These
Definitions If we want a consistent product, we must have a
consistent process If we want to improve a product, we must improve
the process If our company has no (or inadequate) process and we
must produce a quality product, then we must establish a process
[personal responsibility] Developing, imposing, and improving a
process is not an end (in and of itself) it is only a means to an
end
- Slide 23
- 232005 MAPLDDesign Integrity Concepts A Model for Discussing
the Design Process
- Slide 24
- 242005 MAPLDDesign Integrity Concepts Notes on the Model
Feedback / iteration are not shown for clarity Model may be
recursive Board development process includes FPGA requirement
definition, FPGA development, instantiation, etc. Board
verification process includes the FPGA validation product Successes
and failures are cumulative Good requirements + successful
development => successful instantiation Bad requirements +
failed development => failed instantiation Complexity multiplies
Complex requirements increase design complexity which, in turn,
increases verification complexity Processes are absolute gates to
the next stage of development
- Slide 25
- 252005 MAPLDDesign Integrity Concepts Implications From the
Model All processes must be addressed at all levels of design
[there are no shortcuts!] Does not imply same formality at all
levels Does imply same rigor at all levels Up front work on
requirements is essential! Must provide adequate time and money
Must gain team buy-in to the process* Benefits compound throughout
the rest of the activities Simplicity is an essential virtue!
Complexity inevitably multiplies A product is not qualified until
both verification and validation are complete
- Slide 26
- 262005 MAPLDDesign Integrity Concepts Additional Useful
Definitions (courtesy of Q9000-2000) Specification A document*
stating requirements, needs, or expectations that are obligatory
Quality The degree to which a set of inherent characteristics
fulfill requirements Customer satisfaction Customers perception of
the degree to which the customers requirements have been fulfilled
Verification Confirmation, through the provision of objective
evidence, that specified requirements have been fulfilled
Validation Confirmation, through the provision of objective
evidence, that the requirements for a specific intended use or
application have been fulfilled Objective evidence Data supporting
the existence or verity of something Continual Improvement
recurring activity to increase the ability to fulfill requirements
Note the importance of Requirements
- Slide 27
- 272005 MAPLDDesign Integrity Concepts Summary I have no
assurance that my product will have consistent quality without:
Well-defined requirements A well planned approach to implementing
the requirements A clearly defined plan for verification and
validation of the requirements The ability to improve the process
that produces the product Without quality product, customer
satisfaction is impossible Without customer satisfaction, I wont
work! Therefore, I must care about ensuring design integrity
- Slide 28
- 282005 MAPLDDesign Integrity Concepts What Does It Have to Do?
Specifying the design
- Slide 29
- 292005 MAPLDDesign Integrity Concepts Agenda Specifying the
Design Basic concepts and implications What should we include in a
specification? What should we avoid in a specification? The role of
the design engineer Summary FPGA specifications
- Slide 30
- 302005 MAPLDDesign Integrity Concepts Basic Specification
Concepts #1 A specification has a limited set of purposes: It is
intended: To capture product requirements that are essential to
meeting a need functionally and interfacially To ensure
traceability of the requirements from higher levels To provide a
fixed framework for verifying product conformance To provide a
framework for derived requirements It is not intended: To impose a
detailed implementation for a design To provide unnecessary theory
of operation statements To become a users manual As a box to check
on the schedule checklist
- Slide 31
- 312005 MAPLDDesign Integrity Concepts Basic Specification
Concepts (cont.) #2 Specification language must meet certain
criteria Structure Similar requirements must be grouped together
All requirements must stand on their own Verbs must have a
consistent specific meaning Shall, must -> impose imperatives
Should, might -> define preferences Will -> defines objective
information Phraseology Phrases must be unambiguous Phrases must
define one (and only one) requirement Phrases must be short and
understandable All must agree on the meaning of what is
written
- Slide 32
- 322005 MAPLDDesign Integrity Concepts Basic Specification
Concepts (cont.) #3 Specifications are inherently intrusive
Requirements exclude options in a design The number and specificity
of requirements determine the level of exclusion Specifications
impose non-negotiable verification requirements Every requirement
must be verified The narrower the requirement, the more specific
the verification The more requirements, the more work needs to be
performed A well designed specification assists the design and
verification process A poorly designed specification Unnecessarily
shuts down trade space Increases verification time and effort Makes
for an unhappy engineer
- Slide 33
- 332005 MAPLDDesign Integrity Concepts What is Included in a
Specification? Interfaces Number Type Timing / Handshaking
Interrelationships Temporary storage Data Flow Software Support
(careful) System impact Quality and Reliability Parts, materials,
and processes
- Slide 34
- 342005 MAPLDDesign Integrity Concepts Examples of Things to
Include Interfaces The unit shall provide 16 low-level discrete
interfaces The unit shall configure 8 low-level discrete interfaces
as pulsed interface Pulse interfaces shall generate pulses between
2 and 20 ms long Interrelationships The unit shall provide on-board
temporary storage sufficient for 2 packets from the XYZ interface
The unit shall forward data from the XYZ to QRS interfaces on a
packet by packet basis The unit shall provide an interrupt to the
S/W upon receipt of a complete package from the XYZ interface
System Integration The unit shall have a.75 probability of success
as calculated per the parts count method of MIL-HDBK-217 The Unit
shall use only EEE parts that meet the requirements of EEE-
INST-002
- Slide 35
- 352005 MAPLDDesign Integrity Concepts What Should Not Be
Included in the Specification? Implementation details (unless
essential to safety or reliability) The unit shall implement
standard pyrotechnic fire circuit #2 as found in (marginally
acceptable) The unit shall use 4 7206 FIFOs made by IDT corporation
370 ns after the last bit of a packet is received the unit shall
raise bit 3 of interrupt control register 21 Requirements based on
inherent capability of devices used The unit shall provide a 14-bit
A/D converter for interface X (when based upon the part being used)
Note: Decisions on precision or accuracy etc. should be made for
interface characteristics (physical measurement range and
accuracy)
- Slide 36
- 362005 MAPLDDesign Integrity Concepts What Shouldnt Be Included
in the Specification (cont)? Ambiguous, unclear, or interpretive
requirements The unit shall be constructed using techniques found
in generally accepted space hardware guidelines The unit shall
interface to other components per figure 2 (picture based
requirements) The unit shall provide the best performance possible
Overly complex requirements The unit reset shall be immediately
triggered upon expiration of the watchdog timer unless the watchdog
timer has timed-out five times (8 times if in mode 2) in which case
the circuitry shall look at discrete bit 2 to determine whether to
delay 3 s (discrete bit 2 = 1) before timing out Extraneous
information The unit shall interface with the visible CCD used in
spectrographic mode The unit shall replace the functionality
provided by p/n 276401-01
- Slide 37
- 372005 MAPLDDesign Integrity Concepts Why Shouldnt These Be
Included? This type of requirement increases cost [time, money,
emotional] to develop a product Increase design effort (less trade
space, inefficient design, overly complex design) Increase
verification effort [planning, implementing, configuring] due to
design complexity Increase probability of re-design or re-build
[validation uncertainty] This type of requirement reduces the
overall quality of the product developed Implemented design not per
intention [interpreted requirements] Implemented design is not 100%
verifiable to specification [unclear requirements] Implemented
design requires significant number of waivers and deviations
- Slide 38
- 382005 MAPLDDesign Integrity Concepts The Role of the Design
Engineer The design engineering team must buy in to the
specification development process They have to deal with the
consequences if it is done poorly Lack of buy in produces a
non-controlled process Buy in requires early review and
participation by the design team Therefore, design engineers have a
significant role to play in the specification development process
What is this role?
- Slide 39
- 392005 MAPLDDesign Integrity Concepts Role of the Design
Engineer (cont.) Know how to write a specification Allows
constructive review of the imposed specification Improves qualities
of derived requirements Try to understand the why behind the
various requirements Improves efficiency of design trade studies
Allows intelligent suggestion of alternatives Suggest alternatives
when necessary Expose more efficient ways of producing equivalent
functions Support trade offs between software and hardware
functionality
- Slide 40
- 402005 MAPLDDesign Integrity Concepts Role of the Design
Engineer (cont.) Think forward Take the lead in defining derived
requirements Ask: What implications does this have for the design?
What implications does this have for testability? Will this let me
sleep at night? Push back Seek clarification of ambiguous
requirements Inform others of impractical or costly driving
requirements Ask for relief from impossible requirements Remain
engaged in the process Be thorough in review Dont be passive with
suggestions
- Slide 41
- 412005 MAPLDDesign Integrity Concepts Summary Specifications
are critically important to the design engineer and must not be
ignored Design teams must be intimately involved in the
specification development process Detailed design and
implementation will succeed or fail largely based on successful
application of the specification process
- Slide 42
- 422005 MAPLDDesign Integrity Concepts FPGA Specifications -
Rationale FPGAs are a major part of modern day spaceflight designs
Primary control functionality Equivalent of multiple boards of
traditional circuitry Major schedule driver FPGAs are impossible to
verify completely from external signals Too many buried modes and
functions Too much dependence on simulation Correcting FPGAs is
conceptually simple Temptation to sloppiness First time right is an
essential virtue The FPGA specification is a means to achieve first
time right
- Slide 43
- 432005 MAPLDDesign Integrity Concepts FPGA Specification
Prototype Interface Section Specific pinout requirements I/O type
definition (Names, Direction, Logic levels) Imposed Software
requirements (addressing, etc.) Do not include non-imposed
addressing / register mapping / software interfaces Functionality
Section Interface interaction requirements Data flow requirements
Exclusion requirements Do not include Implementation details Theory
of operations Usage instructions
- Slide 44
- 442005 MAPLDDesign Integrity Concepts FPGA Specification
Prototype (cont.) Fine timing section All timing imposed by board
peripheral circuits Include Min and max delay between I/Os Set-up
and hold for controlled peripherals Fine timing requirements should
be an input to the FPGA development effort, not outputs of the
concluded design
- Slide 45
- 452005 MAPLDDesign Integrity Concepts Why Include Fine Timing?
Reduces influence of changes external to FPGA (encapsulation) Board
implementation can change significantly FPGA implementation can
change significantly Changes ok as long as interfaces remain stable
- but fine timing must be a controlled item Simplifies verification
Verification between implemented FPGA performance and fixed
requirements defined at the board level can be easily accomplished
Reduces reliance on complex back-annotated simulations at the board
level for FPGA specific verification Increases reliability of
static timing analysis Note this only works if a structured design
approach is used such that valid requirements can be imposed on the
FPGA early in the process.
- Slide 46
- 462005 MAPLDDesign Integrity Concepts Stand-Alone or Integral?
When should an FPGA specification stand alone? When the FPGA design
engineer is not the board design engineer When there are multiple
interrelated FPGAs on a board When the FPGA design will be used in
multiple board applications When the FPGA will become an ASIC When
the development schedule between the board and FPGA are disjoint
When the complexity of the FPGA is greater than that of the board
support circuitry
- Slide 47
- 472005 MAPLDDesign Integrity Concepts Stand Alone or Integral
(cont.)? When should an FPGA be specified as part of the board
specification? When the FPGA is so intertwined with the peripheral
circuitry that writing a stand-alone specification is not practical
When the FPGA functionality is easily expressed by simple gate
level schematics When the FPGA and board are designed by the same
person When heritage design with adequate specifications exist
- Slide 48
- 482005 MAPLDDesign Integrity Concepts Letting Constraints Work
For You Proportional Design
- Slide 49
- 492005 MAPLDDesign Integrity Concepts Agenda Proportional
Design Conceptual background Types of constraints Examples The
proportional design mindset Summary
- Slide 50
- 502005 MAPLDDesign Integrity Concepts Conceptual Background
Three parts to solving a problem: Need, solution set, constraints
All parts have a role to play in the solution Ignoring any of them
will lead to problems
- Slide 51
- 512005 MAPLDDesign Integrity Concepts Conceptual Background
(cont.) Example Need:means of conveyance to work Solution set:
Skateboard, bicycle, bus, jogging shoes, mid-size sedan, luxury
car, helicopter Constraints: Distance (6 miles), $, not on bus
route, $, not in very good shape, $ Solution: 1992 Honda Accord
(120 kmiles, 4 k$) The constraints guide selection of the solution
from the solution set The particular solution is not necessarily -
The cheapest (roller skates) The most desired (Lexus LS400) What is
perceived as best for society (bus) But the best overall fit to the
needs
- Slide 52
- 522005 MAPLDDesign Integrity Concepts Conceptual Background
(cont.) Definitions Constraint: the state of being checked,
restricted, or compelled to avoid or perform some action (AH)
Proportional: corresponding in some degree or intensity (AH)
Proportional design is design that results in a product sized
appropriately to the needs and restrictions of the specification
The concept of proportional design: Accepts the reality of
constraints Attempts to optimize the solution given the constraints
Accepts that the constraints provide benefits (more later) More
efficient designs More thorough designs More correct designs Caveat
All other things being equal
- Slide 53
- 532005 MAPLDDesign Integrity Concepts Types of Constraints
External (mass, power, cost, quality) Internal Derived (packaging,
architecture, component availability, maximum clock speed)
Self-imposed Design rules/guidelines (free space, clock use, logic
structure, HDL language) Documentation style (pre-design, post
design) Component acceptability (maturity of part, limited use of
various features
- Slide 54
- 542005 MAPLDDesign Integrity Concepts Examples (1) Problem:
provide decoding logic for memory map 0-3FFF = SRAM; 4000-4FFF =
Peripheral; E000-FFFF = PROM Constraint: use minimum amount of
logic But what about Unused addresses, future expansion, etc.
Doesnt matter given the constraints
- Slide 55
- 552005 MAPLDDesign Integrity Concepts Examples (2) Problem:
provide all combinational / sequential logic for the RADARSAT ACP
Constraint: Only low density high speed logic available (16X8 PALs,
MSI/SSI logic) What was forced by the constraint? Careful mapping
of peripherals into available address space Careful partitioning
between: Programmable logic and MSI/SSI MSI/SSI functionality
Efficient data bus partitioning (tri-state enable issues) Special
attention to component delays at the gate level
- Slide 56
- 562005 MAPLDDesign Integrity Concepts The Proportional Design
Mindset Constraints inevitably foster attention to detail
(creativity inside the box) With respect to methodology With
respect to level of planning With respect to implementation
Attention to detail is of inherent value because it produces
carefully structured, well-thought out designs Improved up-front
correctness Decreased design post-processing time (simulation,
verification, validation, lab time) Efficient designs that meet the
stated requirements Increased reliability Therefore, constraints
are welcomed, whether externally imposed or self-imposed
- Slide 57
- 572005 MAPLDDesign Integrity Concepts The Proportional Design
Mindset (cont.) Examples of self-imposed constraint Ignoring
achievable flexibility (when not necessary) Removing non-specified
capability Avoiding gratuitous cleverness (especially with abstract
design techniques) Rejecting brute force solutions without
analysis
- Slide 58
- 582005 MAPLDDesign Integrity Concepts The Proportional Design
Mindset (cont.) Characteristics of the right mind set Planning
before starting Reviewing before finalizing Simplifying ruthlessly
Making the design do only what it must Viewing resources as
precious commodities to be used only to the extent needed
Understanding the implication of the designs level of abstraction
Being satisfied with the result
- Slide 59
- 592005 MAPLDDesign Integrity Concepts The Proportional Design
Mindset (cont.) Why arent self-imposed constraints more common?
They arent absolutely essential because we have: Lots of logic
space [FPGAs, ASICs] Lots of memory space [DOS file systems,
complicated operating systems] Lots of bandwidth [fast data busses,
general purpose communications protocols] They dont match the
current paradigm Flexibility is all-important [re-use,
re-configure, adapt] Specifications are malleable late in the game
Software changes, why cant hardware? We can catch problems in
simulation and reprogram the part They arent fun We dont train
people to value constraints and work within them This is
unfortunate because constraints can make our job easier without
degrading the end product
- Slide 60
- 602005 MAPLDDesign Integrity Concepts Summary The proportional
design mindset is important because it: Focuses on fulfilling
needs, not wants [specification orientation] Deepens understanding
of the final design [ownership oriented] Avoids unnecessary effort
[efficiency oriented] Fosters simplicity that aids verification and
validation [quality oriented]
- Slide 61
- 612005 MAPLDDesign Integrity Concepts More Than the Sum of Its
Parts Design Partitioning
- Slide 62
- 622005 MAPLDDesign Integrity Concepts Agenda Design
Partitioning Definitions and examples Why partition an electronic
design? Guidelines for partitioning an electronic design Why isnt
it done more often? Summary
- Slide 63
- 632005 MAPLDDesign Integrity Concepts Definitions and Examples
Partition to divide into parts or shares Examples: budgeting,
outlining, WBS development Design partitioning refers to the
deconstruction of a design into various sub-designs in an ordered
and logical manner Goal to simplify the whole by optimizing the
parts and thus increase: Efficiency Reliability
Maintainability
- Slide 64
- 642005 MAPLDDesign Integrity Concepts Why Partition (General)?
Complexity interferes with ready comprehension Comprehension of a
complex system depends on ability to impose order upon it But a
given mind has a finite capability to impose order which depends on
the quantity and structure of the data related to the system The
more complex the system, the more difficult its comprehension
Partitioning a design introduces a piece-wise reduction in
complexity Reduces quantity and complexity of design to manageable
chunks Improves comprehension of the various parts of the design
Increased comprehension of the parts leads to better comprehension
of the whole Better comprehension of the whole increases the
ability to Verify correctness of the design Correct errors in the
design Update the design when necessary
- Slide 65
- 652005 MAPLDDesign Integrity Concepts Why Partition (cont.)?
Programmatic advantages Refines scope of work Identifies unexpected
effort Identifies reuse possibilities Identifies staffing
requirements Identifies schedule dependencies Improves allocation
of resources Identifies parallelization and schedule enhancement
opportunities Promotes management visibility
- Slide 66
- 662005 MAPLDDesign Integrity Concepts Why Partition? - Design
Improved interface organization / more formal structure Interfaces
between functions have more predictable characteristics Expansion
by addition of functions is more controlled Enhanced functional
encapsulation Individual functions have predictable results when
invoked Functional design enhancements have limited side-effects
when installed Effects of faults are more easily predicted and
mitigated Efficient design implementation Functional (fewer
functions and types of functions) Data flow (fewer, more sensible
data busses) Control flow (simpler address decoding and state
machines) All are side effects of additional thought put into
How?
- Slide 67
- 672005 MAPLDDesign Integrity Concepts Why Partition? -
Correctness Simpler inspection Functionality may be obvious at a
glance Error space is more limited Within the function or within
the interconnect Simpler Qualification Verification can begin with
encapsulated modules or circuit subsets Overall functionality
correctness becomes less of a late concern because subset
functionality is proven correct early in the process Simpler / more
thorough review Structure provides orientation to peer reviewers
Encapsulation allows easier review Peer input more likely to be
useful
- Slide 68
- 682005 MAPLDDesign Integrity Concepts Why Partition?
Maintainability Clearer documentation Documentation has smaller
parts to focus on Structure of documentation grows from the design
structure Simpler maintenance Changes affect only enhanced area
Interactions between changed area of design and remainder of
original design is controlled by a formal structure Enhanced reuse
Sub-circuits / functions usable in other applications as long as
the interface structure is observed Inherent capability of design
better understood
- Slide 69
- 692005 MAPLDDesign Integrity Concepts Guidelines for
Partitioning Take advantage of organizational strengths Expertise
(analog, digital, software, etc.) is seldom the same across
organizations Partition the design in accordance with
organizational strengths according to primary functions Divide
auxiliary functions (those that can be assigned to multiple
organizations) so that Interfaces are simplified Workload is
equalized Functions are easily tested without requiring all of the
hardware
- Slide 70
- 702005 MAPLDDesign Integrity Concepts Guidelines for
Partitioning (cont.) Example: Alice UV spectrograph electronics
(Rosetta) Core expertise Sensor Sciences: Detector and front end
electronics Mobius Systems: Embedded software SwRI space systems:
Digital, low speed data acquisition SwRI space science: LV and HV
power supplies Primary partition per core expertise
- Slide 71
- 712005 MAPLDDesign Integrity Concepts Guidelines for
Partitioning (cont.) Alice example (work breakdown chosen) Sensor
Sciences: detector electronics through parallel digital output
simplified interface Mobius: instrument control / protocol
servicing (some error decoding partitioned to H/W) Space systems:
microcontroller; s/c interface; heater, motor, door digital
control; analog high-voltage control Space sciences: LVPS; HVPS;
motor, door, and heater drive circuitry
- Slide 72
- 722005 MAPLDDesign Integrity Concepts Guidelines for
Partitioning (cont.) Alice example partitioning decisions Auxiliary
expertise split along sensible interfaces C&DH to detector
analog or digital? (noise, ease of test) - digital C&DH to HVPS
analog or digital? (space, noise susceptibility, ease of test) -
analog C&DH to S/W protocol decoding? (performance margin,
logic capability) hardware error decoding, s/w protocol
encoding
- Slide 73
- 732005 MAPLDDesign Integrity Concepts Guidelines for
Partitioning (cont.) Use specific functionalities Combine
functionality with very similar characteristics Low-level analog /
discrete bi-level (comparator is difference) Example: Spacelab flex
interfaces Cluster related functionalities Shared data-flow
direction, shared logical control Examples Constant level discretes
/ pulsed discretes Low-level discretes / high-level discretes
- Slide 74
- 742005 MAPLDDesign Integrity Concepts Guidelines for
Partitioning (cont.) Use specific functionalities (cont.) Isolate
related functional sub-groups from other items Ex. analog data
acquisition group (multiplexers, converters, signal processing,
control logic) Isolate Through appropriate data/address buffering
Through separate programmable logic sub-sets Exploit directionality
Write functions; read functions; read/write functions appropriate
buffering Examples Write: Analog output, digital output, telemetry
Read: Analog input, digital input, command R/W: Memory,
bi-directional discretes, GSE
- Slide 75
- 752005 MAPLDDesign Integrity Concepts Guidelines for
Partitioning (cont.) Exploit operational considerations Operational
considerations often determine the specific configuration of a set
of common components Example: memory system Components: memory,
write control, read control, sequencing, buffering Application:
telemetry system Packetized / unpacketized? Asynchronous timing /
TDM ? Science data / engineering data? Pushed / pulled ? The type
of telemetry system determines the partitioning
- Slide 76
- 762005 MAPLDDesign Integrity Concepts Example Science Data
Storage and Readback System How should the logic be partitioned? Is
write logic part of science data process or memory process? Is read
logic part of telemetry system or memory process? How complicated
is the arbitration? How it is partitioned depends on specific
operational requirements
- Slide 77
- 772005 MAPLDDesign Integrity Concepts Example - New Horizons
Alice Science Data Alice Operation Slow source accumulation
relative to output speed Push interface initiated by instrument
Alice Implementation (others likely in different circumstances)
Block 1: handshake and latch data Block 2: Ingest data, process,
write to memory Block 3: Read, serialize, send, blank memory
Arbitration simplified to a switched double buffered memory access
(no real-time arbitration)
- Slide 78
- 782005 MAPLDDesign Integrity Concepts Guidelines for
Partitioning (cont.) Ensure encapsulation is reflected in the form
of the engineering documents Functions contain many types of
operations Example: Telecommand interface De-serialization,
decoding, error determination, re-packetization, temporary storage
Real time functionality [level 0], forwarding to software Storage
access / arbitration, status log maintenance, data-bus handshaking
Partitioning should ensure that all reasonable aspects of a
function are in one locality (we are ordering data for
understanding) One (or a few) pages in a schematic One module in
HDL One object in software Benefits: readability, error
determination, testability, maintainability
- Slide 79
- 792005 MAPLDDesign Integrity Concepts Ordering Example Bus
Structure A logical bus structure Simplifies data flow Eases
expansion / enhancement Identifies bottlenecks / opportunities for
efficiency Ensures signal compatibility Reduces timing uncertainty
(capacitive loading) Reduces power Simplifies control logic /
arbitration Simplifies analysis
- Slide 80
- 802005 MAPLDDesign Integrity Concepts Ordering Ex. Bus
Structure (cont.) Before: After:
- Slide 81
- 812005 MAPLDDesign Integrity Concepts Ordering Ex. Bus
Structure (cont.) Directional data flow is clustered High-speed
access devices (SRAM) not buffered Exclusive functions clustered
(PROM/ EEPROM) Simplification of Timing Loading Control
- Slide 82
- 822005 MAPLDDesign Integrity Concepts Why is Partitioning Not a
Priority? It is at the S/C, sub-system, and box level (sometimes)
Why not at the board level? Requires potentially significant
planning effort (schedule / cost) The tool syndrome (CAD / CAE)
Crush creativity by forcing an early start to design Primarily a
way to communicate between software packages rather than humans
(schematic => PWB) Too much junk being placed in too little
space (simplification may not always be space efficient) Lack of
emphasis from senior engineers Boards arent where the action is
(FPGAs) less effort placed on them
- Slide 83
- 832005 MAPLDDesign Integrity Concepts Why is Partitioning Not a
Priority? (cont.) At the FPGA level Lack of solid design
methodology Methodology must be tailored to a tool VHDL / Verilog
are functional descriptions VHDL / Verilog dont inherently enable
data flow visualization or block oriented deconstruction The
synthesizer can understand non-partitioned design Perception of
expansive resources (no / few constraints) The tool syndrome (I
must start coding) Lack of effective design quality gates prior to
start of detailed design
- Slide 84
- 842005 MAPLDDesign Integrity Concepts Summary A design is only
as solid as its weakest part Proper planning and partitioning of a
design: Ensures individual functions are logical and complete
Ensures interconnects are ordered and efficient Provides for
improved reliability / verifiability Allows easier modification and
enhancement Enhances detailed understanding of how things work
Partitioning requires ordering the design by considering The
capabilities of the team The functionality of the modular pieces
How the design will operate The individual components of a
particular function Partitioning a design ensures that all parts
are solid resulting in a solid whole
- Slide 85
- 852005 MAPLDDesign Integrity Concepts You Mean Were Still
Working On It? Sustaining a Design
- Slide 86
- 862005 MAPLDDesign Integrity Concepts Agenda Design
Sustainability Definitions Sustainable Capable of being supported
(AH) Sustainability The characteristic of an item that allows it to
be supported Why is this important? Suggestions for sustainability
Summary
- Slide 87
- 872005 MAPLDDesign Integrity Concepts Why Sustainability?
Missions may be multiple decades long Flight systems may develop
anomalous behavior Ground equivalents may need repair An
understanding of the design is necessary to ensure mission success
Original designers will not be available for debugging Other
critical assignments Working in telecon Cruising the Pacific
Sustainable designs allow analysis and correction without the
access to the original designers
- Slide 88
- 882005 MAPLDDesign Integrity Concepts Why Sustainability?
(cont.) Many designs are derivative Reuse of unmodified circuits
essential for similar performance in modified designs Acceptable
modification depends on creative incorporation of what IS
Derivation may take many years Example Alice UVS Design 1 Rosetta
(1997-2001) Design 2 New Horizons (2002-2005) Design 3 LRO
(2005-2008) Design 4 Juno (2006 - ???) Staffing will not be
constant Human memory will not be precise Sustainability ensures an
ability to efficiently build on past successes
- Slide 89
- 892005 MAPLDDesign Integrity Concepts Why Sustainability
(cont.) You may not be the person who has to make it work Staffing
is dynamic You may quit You may get re-assigned Somebody with more
clout may be needed to satisfy the customer Teams produce a product
and share debugging Test technician S/W designer I&T team
Self-interest and common courtesy You dont want 18 questions per
day Ethics Do unto others Example (ICB)*
- Slide 90
- 902005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability Remember the dual nature of design input The CAD
perspective Schematic => PCB layout package => circuit board
HDL => Synthesizer => Fuse file => Programmed FPGA The
human perspective Schematic Interrelationships (time, space,
connection) Debugging tool Functionality description HDL Describes
functions and interaction Renders constraints understandable Ensure
Readability!
- Slide 91
- 912005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Record the design process Keep a notebook
(type not vital) Describe everything of importance Why? Is this bus
used for this function? Is this function implemented like this?
Etc. How? Do these things talk to one another? Does this sequential
logic work (state diagrams)? Is the address map decoded? Are errors
handled? Etc.
- Slide 92
- 922005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Record the Design Process (cont.) Describe
(cont.) What? Signals are needed to perform this function? Do the
waveforms look like? Timing do I expect to observe? Changes have I
made? Etc. When? Record chronology Provide a way to reproduce what
was done Make part of permanent project record Example Radarsat 1
Notebook
- Slide 93
- 932005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Schematics Provide an overview of the design
Schematic table of contents Block diagram (hierarchical design if
available) Provide consistent naming scheme Descriptive of signal
direction / function / polarity Consistent across logic gates and
within various blocks Cluster sub-circuits on contiguous pages Make
connections between components explicit Add comments where
necessary for clarification Remove unused circuitry (for FPGA
schematics)
- Slide 94
- 942005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Dont:
- Slide 95
- 952005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Do:
- Slide 96
- 962005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Help!
- Slide 97
- 972005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) HDL (Must be done from beginning!) Provide
overall orientation to design Provide top-level comments on Level
of use (top, intermediate, etc.) Overall purpose of function /
block / module Signal function and origination (external and
internal) Provide operational comments on State machine purpose and
configuration (how? why?) Transition logic (theory and reasoning)
Function of particular sequences State to control signal
translation Clarify obscure references Remove superseded code (dont
comment out) and explain uncommon structures Improve readability
Create logical file names Minimize file, logic block, function
sizes Include related functions together (error generation, data
interface, basic function, etc.
- Slide 98
- 982005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Comments?
- Slide 99
- 992005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Header from same design! (After the fact
documentation)
- Slide 100
- 1002005 MAPLDDesign Integrity Concepts Suggestions for
Sustainability (cont.) Post process documentation Theory of
Operation / Users Manual Generate One Include Design concept /
features Operational Constraints Appropriate Uses Complete
Engineering Documentation Update, release and correct as necessary
Create design archive Self-consistent and complete Place under
revision control Control changes
- Slide 101
- 1012005 MAPLDDesign Integrity Concepts Summary Useful designs
will be corrected, modified and evaluated FOR A LONG TIME! By
people besides you Sustainability measures must be implemented to
make this happen efficiently Sustainability requires Adequate
conceptual documentation and records Clear and readable
implementation records Finalized and controlled configuration
records Ensuring sustainability will preserve your legacy
- Slide 102
- 1022005 MAPLDDesign Integrity Concepts Whats the Exit Strategy?
Verifying a Design
- Slide 103
- 1032005 MAPLDDesign Integrity Concepts Agenda Design
Verification Concepts and implications The specification connection
Verification before test Test thoroughness Tiered verification
Concluding the process Summary
- Slide 104
- 1042005 MAPLDDesign Integrity Concepts Verification Concepts
Verification Confirmation, through the provision of objective
evidence, that the specified requirements have been fulfilled
(Q9000-2000) Confirmation the act of ratifying or corroborating
(AH)
- Slide 105
- 1052005 MAPLDDesign Integrity Concepts Verification
Implications The verification process is not intended to be a craps
shoot Verification is primarily intended to highlight correctness
Verification is NOT primarily intended to reveal incorrectness The
mindset is important! Doubt that a design will work expresses a
lack of confidence in the designer and the design process Waiting
until verification to guarantee correctness is an excuse for being
sloppy We should expect our designs to work! Verification simply
puts the seal of confirmation on the expectation
- Slide 106
- 1062005 MAPLDDesign Integrity Concepts Verification
Implications (cont.) Verification addresses two processes Design
Primarily a one-time, iterative process in which the developed
concept is proven sound Fundamental correctness can be proven
analytically Inspection confirms logical correctness in all
circumstances Peer reviews are a manual form of inspection
Functional simulation is an automated form of inspection
Inspections must be tailored to design Analysis verifies
correctness in the presence of variability Reliability (WC,
Derating, FMEA, etc.) Timing over environments and age
- Slide 107
- 1072005 MAPLDDesign Integrity Concepts Verification
Implications (cont.) Verification addresses two processes (cont.)
Instantiation Particular instance must be sound Particular
correctness must be demonstrated experimentally Inspection confirms
that instructions are followed Correct components installed Correct
configuration selected Correct processes performed Test and
demonstration confirms that operation matches expectations
Functionally and parametrically Predicted by nominal analytical
predictions
- Slide 108
- 1082005 MAPLDDesign Integrity Concepts Verification
Implications (cont.) Verification after the fact is too late
Analysis and test are NOT equivalent Design analysis and inspection
must proceed in lockstep with design processes Implementation tests
and inspections should simply confirm what is known Design efforts
fail because they occur in a vacuum Creativity takes primacy over
correctness Functionality comes first with operability being
shoehorned after the fact The monster simulation syndrome
Conclusion: Verify as you proceed
- Slide 109
- 1092005 MAPLDDesign Integrity Concepts Verification
Implications (cont.) Correctness is a matter of personal integrity
for the designer Verification is not simply externally imposed task
Verification is something that the designer must want to do (a part
of his/her ethos) Verification is confirmation that the designer is
worth his/her salt
- Slide 110
- 1102005 MAPLDDesign Integrity Concepts The Specification
Connection Requirements are confirmed during verification The
designer does not have a free hand with respect to how a design is
confirmed Specification provides the constraint set under which
verification is prosecuted Therefore, verification must be defined
prior to start of design Not simply in a general manner
Specifically Since specification is a customer and vendor document
Both customer and vendor must be involved in the verification
process Trust me is not a reasonable phrase when it comes to
verification
- Slide 111
- 1112005 MAPLDDesign Integrity Concepts The Specification
Connection (cont.) Specification contains a complete [ideally] set
of requirements for the design Some requirements are very specific
All discrete outputs shall have a source impedance of 2 kOhms. An
adequate test is fairly obvious Some requirements are not very
specific The unit shall provide 512 kbytes of SRAM Note the implied
requirement The SRAM shall function correctly What is an adequate
functional test for this requirement? (walking 1s, 0s addressing,
etc?) Some requirements lend themselves to quantitative analysis
The unit shall provide a.1 C accuracy at end of life Some
requirements lend themselves to simple inspection The unit shall
provide 4 analog output channels When do we decide adequacy of the
method of verification and the individual verification cases?
Before we start designing
- Slide 112
- 1122005 MAPLDDesign Integrity Concepts The Specification
Connection (cont.) Therefore, verification planning must begin with
specification creation Each requirement must be assigned a method
or methods Each requirement must be assigned a test or analysis
case Must answer question What is an adequate verification? Must
establish answer to question When are we done? Each requirement
must be verified at one or more levels [FPGA, board, box,
sub-system, system] Customer must concur with decision Note that
modern verification databases make this relatively
straightforward
- Slide 113
- 1132005 MAPLDDesign Integrity Concepts The Specification
Connection (cont.) Advantages to this type of verification planning
More verifiable / verified designs Up-front focus on
programmatically difficult verification can be instituted Earlier
analysis Simplification of overly complex circuit designs Guidance
for lower-level verification efforts can be established Sub-module
vs. module level Module vs. unit level Integral self-test
provisions can be included in design Earlier simulation vector
definition (FPGAs / logic)
- Slide 114
- 1142005 MAPLDDesign Integrity Concepts The Specification
Connection (cont.) Advantages (cont.) More robust test sets Early
knowledge regarding end-item function communicated Capability of
test set matched to need Precision (timing, voltage, current, etc.)
Speed Test level (component, unit, system) Test flow design
considerations included in test set design Knowing verification
requirements early makes the entire development process more
efficient
- Slide 115
- 1152005 MAPLDDesign Integrity Concepts The Specification
Connection (cont.) Defining requirements and test cases at the same
time as the specification clarifies and simplifies the FPGA
verification process Allows early start to simulation vector design
Creates early knowledge of additional functional models (SRAM,
peripherals, etc.) needed Clarifies decision regarding what can be
functionally simulated and what must be simulated with back-
annotation Defines levels at which simulations must be run
- Slide 116
- 1162005 MAPLDDesign Integrity Concepts Verification Before Test
It is a relatively bad thing to discover a design flaw during test
(better than nothing) Late in the development cycle Correction is
more complicated / expensive Personal stress is higher Yet a
surprising lack of verification occurs early (before test) Types of
pre-test verification Inspection - conformity evaluation by
observation and judgment accompanied, as appropriate by measurement
or testing (ISO) Personal self-check Peer Review Basic functional
verification (Signal audits, functional simulation, etc.)
- Slide 117
- 1172005 MAPLDDesign Integrity Concepts Verification Before Test
(cont.) Types of pre-test verification (cont.) Analysis Conformity
evaluation of the predictions produced by realistic models of the
system Worst case analysis (voltage, current, frequency, time)
using models that incorporate real world degradation of function
Derating analysis using models that reduce stress Other Note that
pre-test verification is fundamentally conceptual Not tool
dependent Not design dependent Can be accomplished many ways
- Slide 118
- 1182005 MAPLDDesign Integrity Concepts Verification Before Test
(cont.) Two approaches to pre-test verification Verify full design
at once (one big peer review, code walkthrough, analysis,
simulation) Benefits Complete design reduces need to develop
assumptions When proved correct, does not need to be repeated When
design is solid, less work Reduces final documentation for
verification Drawbacks Allows small design flaws to propagate for a
long time Complexity reduces visibility (verification more prone to
mistakes and oversights) When design is flawed, it increases work
Leads to corrections that are global (hard to test effectively and
predict side effects) May take longer to execute than an aggregate
of smaller verification activities
- Slide 119
- 1192005 MAPLDDesign Integrity Concepts Verification Before Test
(cont.) Two approaches to pre-test verification (cont.) Verify
incremental designs Benefits Supports development of known good
sub-designs Meshes with a partitioned design approach Facilitates
visibility (fewer mistakes, clearer goals) Allows small flaws to be
fixed quickly (minimizes downstream impact) Drawbacks Is more work
in the ideal case (no/few flaws) Increases documentation if
formality is vital Either approach can be made to work, but one or
the other must be chosen and implemented
- Slide 120
- 1202005 MAPLDDesign Integrity Concepts Test Thoroughness
Principles for test thoroughness Every shall in the specification
that meets the following criteria must be tested Items that may be
affected by the instantiation process (subject to fabrication
error) Items for which the only extant verification is model based
Items for which the testing does not pose unacceptable risk
(radiation, overstress, etc.) Every instance of a shall must be
tested 18 interfaces require 18 specific tests Requirements that
have an exclusive character Must be tested for correct operation
Should be tested at some level for abnormal operation Example: If
you write a 5Ah to something to set it on, you should verify that
writing other values does not set it on Requirements must be tested
over a representative range of conditions
- Slide 121
- 1212005 MAPLDDesign Integrity Concepts Test Thoroughness
(cont.) A thorough test is complicated and time consuming Requires
some level of automation to be comprehensive Requires some level of
compromise to be practical Requires team agreement to be
acceptable
- Slide 122
- 1222005 MAPLDDesign Integrity Concepts Test Thoroughness
(cont.) Ensuring thorough, yet practical, tests Define which tests
require manual interaction and which can be automated Output
impedance or analog fine precision may need to be manual Signal
level or analog coarse precision may be automatable Define which
tests must be performed over environmental conditions and which can
be performed at room temperature only Make decision based on
character of measurement Do not decide based purely on practicality
Define level of complexity for all measurement sets Example: 24
analog inputs (3 eight channel muxes) Test full range of values for
24, or limited range for 7 of 8 mux inputs and full range for
remaining Define need for repetition of measurement at higher level
of integration (tiered verification) Example: voltage level /
signal quality / clock jitter at board level Repeat at box
level?
- Slide 123
- 1232005 MAPLDDesign Integrity Concepts Test Thoroughness
(cont.) Test thoroughness issues affect everything Customer
relationship Test set design (board, box, system) Schedule / cost
An adequate test program is not trivial Cannot wait until system
level Must incorporate lowest level of design Must be a constant
background activity during design process All programs must balance
perfect and good enough A good test program Will ensure that the
specification is met Will verify everything that can be affected by
fabrication Will build on the analysis and inspection effort Will
maximize automation without sacrificing test accuracy
- Slide 124
- 1242005 MAPLDDesign Integrity Concepts Tiered Verification
Tiered verification is the process of matching the correct type of
verification to the appropriate level of integration Tiered
verification incorporates all types of verification (before and
during testing) Tiered verification applied to the test regime
attempts to have thoroughness with practicality Test a requirement
at the lowest conceivable level Determine which tests must be
repeated at higher levels by Establishing the purpose of a test at
a particular level Determining whether the next level has the
potential to change previous measured quantities Determining what
test set capabilities are Tiered verification cannot be retrofitted
into the process Must be planned up front Must be implemented
throughout the development process
- Slide 125
- 1252005 MAPLDDesign Integrity Concepts Tiered Verification -
Example Need: A set of 16 high-level pulsed discretes to control
latching relays used to change the spacecraft power configuration
Basic Requirements Quantity:16 Level:+30V +/- 2 V Load:2 kOhm, 4 mH
Duration of pulse:50 ms Rise / Fall time max:1 ms Command
capability:Unique bit pattern, only one at a time, arm first S/W
requirements:Various (beyond scope of example)
- Slide 126
- 1262005 MAPLDDesign Integrity Concepts Tiered Verification
Example (cont.) Tier 1 Basic Verification (functional simulation)
Verify each channel fires for a specific bit pattern / address
Verify cannot be fired without an arm command Verify that other
complementary patterns dont accidentally fire the pulse Analyze
drive capability of FPGA and input / output requirement /
capability of translation chip Evaluate glitch hardness of logic
selected
- Slide 127
- 1272005 MAPLDDesign Integrity Concepts Tiered Verification
Example (cont.) Tier 2 Board level Test to confirm signal level
between FPGA and driver chips is compatible Test rise, fall,
duration, level with dummy load Repeat test for all 16 outputs
Replace dummy load with relay, verify that the relay switches for
all 16 outputs Repeat over temperature if necessary
- Slide 128
- 1282005 MAPLDDesign Integrity Concepts Tiered Verification
Example (cont.) Tier 3 Box level Attach GSE (verifies with relays)
Functionally pulse all outputs Verify relay switches Verify gross
drive duration Tier 4 Box and operational software level Verify
functionality commanded through software GSE verifies correct relay
switch Tier 5 System Verify gross functionality as needed for
system operation
- Slide 129
- 1292005 MAPLDDesign Integrity Concepts Tiered Verification -
Notes Most complex and detailed functionality is verified at lower
levels Performance Error cases Predictive of future operation
Higher levels focus on functionality / operation Lower level
verification requires more manual touch Higher level verification
is more automated STE and test procedure developed as appropriate
Purpose is maximum test completeness As well as bang for the
buck
- Slide 130
- 1302005 MAPLDDesign Integrity Concepts Concluding the Process (
through the provision of objective evidence) The importance of
traceability Something is only verified (formally) when it is
documented The tie between verification item and specification must
be made (verification matrix or database) The importance of
planning Establish the links between requirement, method, level,
test case early Use the established structure to develop
verification items The importance of buy in All responsible must
complete verification and report results Systems engineers must
track and close People must keep up with the process Only personal
commitment from all involved will ensure that the process is
successful
- Slide 131
- 1312005 MAPLDDesign Integrity Concepts Summary Verification is
to prove success, not avoid failure Verification planning is an
integral part of the project lifecycle Must be initiated with
specification (including customer buy-in) Must be included in
design effort Must be used to develop documentation and appropriate
test hardware Verification is more than final test Begins at lowest
possible level Incorporates analysis and inspection throughout
design process Develops proof of compliance at all levels of
operation A thorough test is complicated Requires extensive work
Must trade perfect for practical (a tiered test approach can help)
Verification processes must be tracked and documented The whole
team must buy in
- Slide 132
- 1322005 MAPLDDesign Integrity Concepts What Do You Mean It
Doesnt Do What We Thought? Validating a Design
- Slide 133
- 1332005 MAPLDDesign Integrity Concepts Agenda Design Validation
Concepts and implications Specification mitigations Design
mitigations Test set mitigation Summary
- Slide 134
- 1342005 MAPLDDesign Integrity Concepts Concepts Validation
Confirmation, through the provision of objective evidence, that the
requirements for a specified intended use or application have been
fulfilled
- Slide 135
- 1352005 MAPLDDesign Integrity Concepts Implications The issues
relating to validation encompass those of verification Additional
concerns with validation Caused by the need to match the
application to the product Desired application has been translated
to specification through the requirements process Requirements
process is by nature imperfect Sometimes the specification does not
satisfy the needs of the application Result a verified product
might be invalid May require significant rework to the product May
require accepting reduced functionality (waiver) A goal of the
development process is to minimize validation failures Begins in
review of the requirements process (hopefully primary point)
Mitigate by design activities Reduce by robust test set design
- Slide 136
- 1362005 MAPLDDesign Integrity Concepts The Implication
Illustrated Alice electronics (detector component and C&DH
component) Joint requirement: process > 10 k sustained events
per second Individual requirements defined for detector and
C&DH processing Both met individual requirements for processing
When combined only 6-7 k sustained events per second Verification
of individual units led to invalid system What went wrong? The
overall requirements were not broken down correctly The C&DH
and DE test sets were not high fidelity Verified functionality, not
performance We got lucky that a waiver was acceptable
- Slide 137
- 1372005 MAPLDDesign Integrity Concepts Specification Mitigation
Only list requirements, not desirements State unambiguous
performance requirements Build in adequate margin Not open-ended
enhancement, but Enough to accommodate performance shortfalls
Ruthlessly remove TBDs Insist on definite test methods for
mitigation Remember Unless application needs can be unambiguously
specified, they wont be met!
- Slide 138
- 1382005 MAPLDDesign Integrity Concepts Design Mitigation
Implement specification exactly Isolate various sub-sections
Minimizes corner cases and negative interactions Allows correction
with minimal impact when things dont work right Verify complex
functions early, thoroughly, and completely Allows early look at
potential problems Analysis / simulation / what ifs should be as
realistic as possible Insist on end-user review of implementation
Allows user community to comment Minimizes misunderstandings upon
delivery Develop test plans that have high fidelity to the end
application
- Slide 139
- 1392005 MAPLDDesign Integrity Concepts Test Set Mitigation
Ensure interfaces are maximally flight-like Precludes
misunderstandings of characteristics Provides early indication of
problems Dont emulate only one characteristic of interface (R vs.
R+L) Make test set reasonably sophisticated Sufficient complexity
to reproduce operational timing Adequate functionality for stress
testing Run all interfaces at maximum speed with margin Dont let
the same group build the tested unit (design) and the unit tester
(test bench) Identical assumptions might go into both ends of an
interface Faithful reproduction is dependent on familiarity (if
possible, test bench should be provided by end user)
- Slide 140
- 1402005 MAPLDDesign Integrity Concepts Test Set Mitigation
(cont.) Make the control interface as application like as possible
Forces correct command structures / types Allows all test scripts
to be reproduced at higher levels If at all possible, incorporate
early interface tests of real engineering hardware Keep the test
(or simulation) environment constant unless the flight system
changes Dont change test equipment hardware configurations Apples
to apples comparisons during tests vital Ensure that flight changes
are reflected in test set design
- Slide 141
- 1412005 MAPLDDesign Integrity Concepts Test Set Mitigation
(cont.) Use the same controls for test set development as for
flight unit development Configuration management Software
development Peer reviews Build in diagnostics so that anomalies can
be traced to test equipment or unit under test Ensure that test
results mean something Pass / fail criteria clear Allowable flight
parameter variations included Reasonable displays (with significant
information clearly shown) Ensure that test set accommodates
calibration
- Slide 142
- 1422005 MAPLDDesign Integrity Concepts Summary Successful
verification does not always guarantee successful validation
Techniques can be incorporated that improve the likelihood that
validation will succeed Careful specification development Thorough
and cautious design techniques Extensive test set fidelity to
flight requirements Effective techniques for validation are extra
effort More time consuming More expensive But, definitely worth
it.