Moving towards Continuous Benchmarking (CB)icl.utk.edu/jlesc9/files/STA1.3/jlesc9_anzt.pdf · 3...
24
KIT – The Research University in the Helmholtz Association Anzt, Chen, Cojean, Dongarra, Flegar, Nayak, Quintana-Orti, Tsai, Wang www.kit.edu Moving towards Continuous Benchmarking (CB) 9th JLESC Workshop April 15th-17th, 2019 | Knoxville, TN
Transcript of Moving towards Continuous Benchmarking (CB)icl.utk.edu/jlesc9/files/STA1.3/jlesc9_anzt.pdf · 3...
KIT – The Research University in the Helmholtz Association
Anzt, Chen, Cojean, Dongarra, Flegar, Nayak, Quintana-Orti, Tsai, Wang
www.kit.edu
MovingtowardsContinuousBenchmarking(CB)9th JLESC WorkshopApril 15th-17th, 2019 | Knoxville, TN
JLESC20192 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Developer
A Healthy Software Development Cycle
JLESC20193 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Developer
A Healthy Software Development Cycle
SourceCodeRepository
Push
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
JLESC20194 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
Push
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
JLESC20195 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
• Unittestsensurefunctionalityandvalidityofallbuildingblocks.
JLESC20196 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
• Unittestsensurefunctionalityandvalidityofallbuildingblocks.
• Goal:Automatedbenchmarkrunseverytimenewcodeisadded.
JLESC20197 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
• Unittestsensurefunctionalityandvalidityofallbuildingblocks.
• Goal:Automatedbenchmarkrunseverytimenewcodeisadded.
JLESC20198 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
SecurityThrust!• ExternalcodeisrunonHPCsystem• Possiblepointofattack
• Versioncontrolsystemfortrackingchangesandcoordinatingcollaborativedevelopment.
• CIBuildsystemcontinuously checksthecompilationsuccessondifferentarchitectures.
• Unittestsensurefunctionalityandvalidityofallbuildingblocks.
• Goal:Automatedbenchmarkrunseverytimenewcodeisadded.
JLESC20199 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
CodeReview
TrustedReviewer
MergeintoMasterBranch
• ManualcodereviewsbytrustedreviewerswhotakeresponsibilityforexecutingcodeontheHPCsystem.
JLESC201910 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
CodeReview
TrustedReviewer
MergeintoMasterBranch
• ManualcodereviewsbytrustedreviewerswhotakeresponsibilityforexecutingcodeontheHPCsystem.
• Howtoarchivetheresultsandmakethemaccessible?
JLESC201911 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
CodeReview
TrustedReviewer
MergeintoMasterBranch
PerformanceDataRepository
• Storedatainseparaterepository.• UseJSONdataexchangeformat.
JLESC201912 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
ContinuousIntegration(CI)
CIBuild
Developer
A Healthy Software Development Cycle
SourceCodeRepository
PushCITest
Schedule inBatchSystem
HPCSystem
CIBenchmarkTests
CodeReview
TrustedReviewer
MergeintoMasterBranch
PerformanceDataRepository
Users
• Storedatainseparaterepository.• UseJSONdataexchangeformat.• Cloningahugerepository is
expensive.• ”Interestinmyexperiments,only.”
JLESC201913 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
CITest
PerformanceDataRepository
ContinuousIntegration(CI)
DeveloperCodeReview
CIBuildSourceCodeRepository
Push
Schedule inBatchSystem
Web-Application
HPCSystem
TrustedReviewer
Users
MergeintoMasterBranch
CIBenchmarkTests
A Healthy Software Development Cycle
• Storedatainseparaterepository.• UseJSONdataexchangeformat.• Cloningahugerepository is
expensive.• ”Interestinmyexperiments,only.”• Interactiveweb-basedGinkgo
PerformanceExplorer(GPE).
JLESC201914 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
CITest
PerformanceDataRepository
ContinuousIntegration(CI)
DeveloperCodeReview
CIBuildSourceCodeRepository
Push
Schedule inBatchSystem
Web-Application
HPCSystem
TrustedReviewer
Users
MergeintoMasterBranch
CIBenchmarkTests
A Healthy Software Development Cycle
GPEGinkgoPerformanceExplorer
JLESC201915 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo Performance Explorer (GPE)
DataSelectionTab
TransformationScriptEditor
DataandPlotViewer
GPE
JLESC201916 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo Performance Explorer (GPE)
DataSelectionTab
TransformationScriptEditor
DataandPlotViewer
1.SelectDatainGitrepository.
GPE
JLESC201917 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo Performance Explorer (GPE)
DataSelectionTab
TransformationScriptEditor
DataandPlotViewer
1.SelectDatainGitrepository.
2.WriteJSONatascripttovisualizedata(examplesareprovided).
GPE
JLESC201918 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo Performance Explorer (GPE)
DataSelectionTab
TransformationScriptEditor
DataandPlotViewer
1.SelectDatainGitrepository.
2.WriteJSONatascripttovisualizedata(examplesareprovided).
3.Analyzedatavisually.
GPE
JLESC201919 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo Performance Explorer (GPE)
GPE
JLESC201920 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo Performance Explorer (GPE)
GPE
JLESC201921 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo Performance Explorer (GPE)
GPEDolan & More: Benchmarking optimization software with performance profiles
JLESC201922 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo’s Performance Evaluation Framework
GPE
ContinuousBenchmarkingBenefits
• Archivingperformancedataalongwithexecutionparametersensuresfullbenchmarkreproducibility.
• Comparing theperformanceresultsoverthecodelifetimeidentifiesperformancedegradations.
• Easeofuse:thesetupallowstolaunchbenchmarkwithfewclicks.
JLESC201923 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
Ginkgo’s Performance Evaluation Framework
GPE
ContinuousBenchmarkingBenefits
• Archivingperformancedataalongwithexecutionparametersensuresfullbenchmarkreproducibility.
• Comparing theperformanceresultsoverthecodelifetimeidentifiesperformancedegradations.
• Easeofuse:thesetupallowstolaunchbenchmarkwithfewclicks.
GinkgoPerformanceExplorer(GPE)Benefits• ThedesignofGPEefficientlyrealizestheanalysisaswebservice,removingtheneedfordownloading
performancedatatolocaldiskorinstallingadditionalsoftware.
• ExternaldeveloperswithoutaccesstoHPCsystemscantestandengineertheircodesonHPCresources.
• Extensibility:Optiontocompareperformancewithothersoftware libraries.
Anzt etal:“TowardsContinuous Benchmarking:AnAutomatedPerformanceEvaluationFrameworkforHighPerformanceSoftware”,PASC2019,accepted.
JLESC201924 04/15/2019 Hartwig Anzt:TowardsContinuous Benchmarking(CI)
• Open-sourceC++frameworkforsparselinearalgebra.
• Sparselinearsolvers,preconditioners,SpMV etc.
• Genericalgorithmimplementation:+referencekernelsforcheckingcorrectness;+architecture-specifichighlyoptimizedkernels.
• FocusedonGPUaccelerators(i.e.NVIDIAGPUs).
• SoftwarequalityandsustainabilityeffortsguidedbyxSDK communitypolicies:
•
•
•
•
https://xsdk.info/
PratikNayakTerryCojean GoranFlegar ThomasGrützmacher
TobiasRibizel
https://ginkgo-project.github.io/
Learn More about Ginkgo
https://bssw.io/
