Using Symbolic PathFinder at Using Symbolic PathFinder at NASANASA

Corina PCorina PããssããreanureanuCarnegie Mellon/NASA AmesCarnegie Mellon/NASA Ames

IntroductionIntroduction Modern softwareModern software

Multi-core, cloud-based, cyber-physicalMulti-core, cloud-based, cyber-physical Needs to be highly Needs to be highly reliable and securereliable and secure

Testing Testing Most widely usedMost widely used May May miss important errorsmiss important errors

Formal analysis Formal analysis theorem proving, model checking, symbolic execution, theorem proving, model checking, symbolic execution,

static analysis, etc.static analysis, etc. Strong guaranteesStrong guarantees of reliability and safety of reliability and safety

GoalGoal Use formal techniques to make testing more effectiveUse formal techniques to make testing more effective

Symbolic PathFinder (SPF)Symbolic PathFinder (SPF) PerformsPerforms

Automated test case generationAutomated test case generation Property checkingProperty checking

Integrates Integrates Symbolic execution, model checking and constraint solvingSymbolic execution, model checking and constraint solving

Uses analysis engine of Ames’ JPF toolUses analysis engine of Ames’ JPF tool

Test case generation – the easiest path towards the Test case generation – the easiest path towards the infusion of advanced formal verification infusion of advanced formal verification technologiestechnologies

JPF

Uses of SPFUses of SPF Analysis/test generation forAnalysis/test generation for

Byte-codeByte-code Models (Simulink/Stateflow, UML) Models (Simulink/Stateflow, UML)

Translation into JavaTranslation into Java Polyglot Framework – w/ U. Vanderbilt, U. Polyglot Framework – w/ U. Vanderbilt, U.

MinnesotaMinnesota SPF handlesSPF handles

Pre-conditions, data structures, concurrencyPre-conditions, data structures, concurrency User customizesUser customizes

Coverage, search strategy, output formatCoverage, search strategy, output format Used at NASA, Fujitsu, academiaUsed at NASA, Fujitsu, academia

Orion orbits the moon (Image Credit: Lockheed Martin)

Polyglot

Experience: Onboard Abort Executive (OAE)Experience: Onboard Abort Executive (OAE)Prototype for CEV ascent abort handling being developed by JSC Prototype for CEV ascent abort handling being developed by JSC

GN&CGN&C

Inputs

Pick Highest Ranked Abort

Checks Flight Rules to see if an abort must occur

Select Feasible Aborts

OAE StructureResultsResults BaselineBaseline

Manual testing: time consuming Manual testing: time consuming Guided random testing could not cover all abortsGuided random testing could not cover all aborts

Symbolic PathFinderSymbolic PathFinder Generates tests to cover all aborts and flight rulesGenerates tests to cover all aborts and flight rules Found major bug in new version of OAEFound major bug in new version of OAE

Environment constraintsEnvironment constraints Input data is constrained by environment/physical lawsInput data is constrained by environment/physical laws

Example: inertial velocity can not be 24000 ft/s when Example: inertial velocity can not be 24000 ft/s when the geodetic altitude is 0 ftthe geodetic altitude is 0 ft

Need to encode these environment constraints Need to encode these environment constraints explicitlyexplicitly

SolutionSolution Use simulation runs and machine learning to get data Use simulation runs and machine learning to get data

correlations correlations

Lessons LearnedLessons Learned Not enough to generate test cases that Not enough to generate test cases that

achieve good coverageachieve good coverage Should also Should also make sense to developersmake sense to developers

Environment assumptions need to be Environment assumptions need to be made made explicitexplicit Using both input from the Using both input from the useruser and and automatedautomated

inference techniquesinference techniques True for any analysis tool that can only analyze True for any analysis tool that can only analyze

components (not entire systems)components (not entire systems)

Usability ChallengesUsability Challenges Tool UsabilityTool Usability Research ChallengesResearch Challenges Domain SpecificDomain Specific

Tool UsabilityTool Usability Easy set-up -- Not true for JPF/SPFEasy set-up -- Not true for JPF/SPF Portability for multiple platformsPortability for multiple platforms Animation and/or graphical display of resultsAnimation and/or graphical display of results

Intermediate resultsIntermediate results Important for tools that do not give “instant” results like SPFImportant for tools that do not give “instant” results like SPF

Informative counterexamples/Explanation of errorsInformative counterexamples/Explanation of errors Documentation easily accessible on-line -- JPF’s wikiDocumentation easily accessible on-line -- JPF’s wiki Integration with other tools familiar with developersIntegration with other tools familiar with developers

JPF’s Eclipse plug-in; SPF outputs JUnit tests and Antares JPF’s Eclipse plug-in; SPF outputs JUnit tests and Antares scriptsscripts

Clear display of different options --Not true for JPF/SPFClear display of different options --Not true for JPF/SPF Availability --JPF is open-sourceAvailability --JPF is open-source ……

Research ChallengesResearch Challenges Scalable and timely verificationScalable and timely verification

Analysis as efficient as possibleAnalysis as efficient as possible Parallelization, compositional techniques, fast constraint Parallelization, compositional techniques, fast constraint

solverssolvers New constraint solversNew constraint solvers

Solve complex, non-linear constraints (NASA applications)Solve complex, non-linear constraints (NASA applications) Solve strings combined with numeric constraints (web Solve strings combined with numeric constraints (web

applications, security)applications, security) Automatic discovery of interfacesAutomatic discovery of interfaces

With (minimal) inputs from userWith (minimal) inputs from user Combinations of automata and machine learning techniquesCombinations of automata and machine learning techniques

……

Domain Specific KnowledgeDomain Specific Knowledge Add the Add the useruser to the picture to the picture

Integration of domain specific knowledgeIntegration of domain specific knowledge Adaptation of tools to users’; specific needsAdaptation of tools to users’; specific needs

E.g. NASA uses models; adapt SPF from code to E.g. NASA uses models; adapt SPF from code to models; customize search strategies/output models; customize search strategies/output formatformat

……

??