Post on 10-Jan-2017
Jordan Moldow and Nadeem Ahmad PuppetConf 2016
FromPainToGain:APuppetUnitTes4ngStory
Boxisamoderncontentmanagementpla=ormthattransformshoworganiza4onsworkand
collaboratetoachieveresultsfaster.
WhoWeAre
NadeemProduc'vityEngineering
JordanDesktopEngineering
Unittes4ngmonolithicPuppetrepositories
Orperhapsspaghe6repos
hFps://www.flickr.com/photos/oeuildebarzo/3026845290/MaksBarzo
Unittes4ngmonolithicPuppetrepositories
Characteris8csofspaghe6monoliths
• Inter-connectedmodules
• Nospecifica4ons• Organicgrowth• Businesslogiceverywhere
Thistalkisnotaunittes8ngtutorial
Puppet @ Box
Agenda
Why unit tesBng?
GeFng started
Making it work
ConBnuous integraBon
1
2
3
4
5
Rolling it out 6
Puppet@BoxImportance and scale
Puppet@Box
8,000+NODES
300+MODULES
200+CHANGESPERWEEK
115AVERAGECOMMITTERSPER
MONTH
EnsuringPuppetcorrectnessiscri8cal
Whyunittes4ng?Our current tesBng process and how we wanted to improve
ManualPuppettes8ng@Box
~/puppet $ git commit
~/puppet $ git push origin $BRANCH
$ ssh test.server.com
user@test $ sudo puppet agent –-test --environment=$BRANCH
ManualPuppettes8ng@Box
1. Waitforcatalogtocompile
2. Waitforno-opruntocomplete
3. Reviewandacceptno-opresults
4. WaitforPuppetruntocomplete
5. Reviewstateofmachine
Running Puppet on a test machine
ManualPuppettes8ng@Box
3:00:00
Time spent by engineers
Otherissueswithmanualmonolithtes8ng
• Cannotmanuallytesteveryaffectedrole
• Testsarenoteasilyrepeatable
ModuleM
ModuleCModuleBModuleA
RoleR3RoleR2RoleR1
Cluster1 Cluster2 Cluster3
ApproachestoPuppettes8ng
1. Manual
2. Acceptance(Beaker)
3. Unit/Integra4on
Projectgoals
1. TestscanbewriPenformostfiles
2. Testsrunautoma8cally
3. Testsrunquickly
4. Testsfailifsomethingisbroken
5. Failingtestsarereportedtocodereview
6. Developerswritetestswiththeircode
GefngstartedFirst experiment with rspec-puppet
Toolsweused
• puppet(3.6.2)
• puppetlabs_spec_helper(0.10.2)
• rspec(3.1.0)– Rubyspectes4ngframework
• rspec-puppet(2.3.0)– Extensiontorspecfortes4ngPuppetmodules
Basictes8ng Rakefile – Command definiBons
Basictes8ng.fixtures.yml –Pointerstoyourmodules
Basictes8ngspec/spec_helper.rb – Configure rspec-puppet
Basictes8ngspec/classes/foo/bar_spec.rb
Runningtests
~/puppet $ rake spec
Runalltestsinadirectory
~/puppet $ rake spec SPEC=spec/classes/foo
Runalltestsinafile
~/puppet $ rake spec SPEC=spec/classes/foo/bar_spec.rb
Runalltests
Runningtests
Basictes8ng
• Thisworks!• …onsomesimplemanifests
• Fortechnicalbenefits,thisneedstoworkonmostmanifests
• Forculturalchange,itneedstoworkANDbefric4onless
MakingitworkProblems we encountered and our soluBons to them
box_spec_helper hSps://github.com/jmoldow/box_spec_helper
Problem#1:Undefinedfacts
• rspec-puppetinvokesPuppetwithabareminimumoffactsprovided
• Factvaluesdefaulttoundefinmanifests
• Manifestswon’tcompileasexpected
Solu8on#1:Defaultfactsspec/spec_helper.rb
Problem#2:DefaultrequiredaPributes
• ResourcecollectorssefngdefaultaFributes
• Inproduc4on,execcommandsdon'tneedabsolutepath
• Buttestswillfailwithoutit
Solu8on#2:site.ppsymlink
• Allofourglobaldefaultsareinmanifests/site.pp • Createdsymlinkfromspec/fixtures/manifests/site.pp
Problem#3:Produc8on-onlyfunc8ons
• Somecustomfunc4onsmayonlyworkinproduc4on,e.g.‒ Queryingaproduc4ondatastore‒ AccessingfilesthatonlyexistonPuppetmasters
• Anymanifestthatusessuchafunc4oncannotbetested‒ Norcananymanifeststhatincludessuchamanifest
Solu8on#3:Globally-stubbedfunc8ons
• Produc4on-onlyfunc4onsshouldalwaysbestubbed• AnytestcanbewriFenwithouttest-specificstubs• Allotherfunc4ons,classes,typesshouldactastheywouldinprod
Solu8on#3:Globally-stubbedfunc8ons
• module_pathcanincludemul4pledirectories
• Customfunc4onscannotberedefined;firstdefini4onwins
• Addedextratreetofrontofmodule_path • Addednewmoduleswithredefinedfunc4ons
Stubbing via module paths
Globally-stubbedfunc8on-Example
Otherbigenhancements
• Simpletestauto-generator
• Accumula4ngfact,paramdefini4onsfrommul4pleblocks‒ Includingsharedcontexts
• Emula4ngotheropera4ngsystems‒ Wewererestrictedonwhathardwarewecouldruntests
Con4nuousIntegra4on
FirstItera8onofCI Define small beta group of Puppet developers
FirstItera8onofCI Run unit tests for their code reviews, report feedback
FirstItera8onofCI Run unit tests for their code reviews, report feedback
FirstItera8onofCI
• Iterateonframework
• Encouragebetatesterstoalsowritenewtests
FirstItera8onofCI
• Testsuitetakes6-8minutestorun
• Goal:orderofseconds
UsingClusterRunnertoRunTestswww.clusterrunner.com
ClusterRunnerPuppetModule
54
UnlockingParalleliza8onFind individual test unit
~/puppet $ find spec -name "*_spec.rb"
spec/classes/os_hardening/init_spec.rb spec/classes/rsyslog/init_spec.rb spec/classes/rsyslog/package_spec.rb spec/classes/dashboard/sync_hive_query_runner_spec.rb spec/classes/pod/delivery_pipeline_spec.rb spec/classes/yum/centos_extras_spec.rb spec/classes/dev_env/repo/devtools_for_windows_spec.rb spec/classes/dev_env/repo/devtools_spec.rb spec/classes/snmp/server_spec.rb spec/classes/bind/staging_spec.rb spec/classes/bind/base_spec.rb …
Runningtests
~/puppet $ rake spec
Runalltestsinadirectory
~/puppet $ rake spec SPEC=spec/classes/foo
Runalltestsinafile
~/puppet $ rake spec SPEC=spec/classes/foo/init_spec.rb
Runalltests
WTF??
UnlockingParalleliza8onrake spec is actually a combinaBon of three subcommands:
~/puppet $ rake spec_prep
Createsfixturesneededbyalltests
UnlockingParalleliza8onrake spec is actually a combinaBon of three subcommands:
~/puppet $ rake spec_clean
Deletesalltestfixtures
UnlockingParalleliza8onrake spec is actually a combinaBon of three subcommands:
~/puppet $ rake spec_standalone
Runstheactualtest(assumestestfixturesarepresent)
clusterrunner.yaml
Ini8alTime(Serial)FinalTime(Parallel)
420
20TotalTestT
ime(secon
ds)
TestRun8mefor~400PuppetUnitTests
Workflow
$ git push
post-receive
Git hook
GitServer
ClusterRunnerMaster
JenkinsMaster
BuildStateinJenkins
RollingItOut
Roadblocks
• Testresultsneedtobepostedtocodereviewforalldevelopers• Post-mergetestresultsneedtobeeasilyavailable
Conclusion
ProjectGoalsChecklist
ü TestscanbewriPenformostfiles
ü Testsrunautoma8cally
ü Testsrunquicklyü Testsfailifsomethingisbroken
q Failingtestsarereportedtocodereviewq Developerswritetestswiththeircode
Possiblefuturework
• Contributetorspec-puppet,puppetlabs_spec_helper• Createstandalonerubygem
hSps://github.com/jmoldow/box_spec_helper
Ques8ons?
• box_spec_helper ‒ hFps://github.com/jmoldow/box_spec_helper
• Contact‒ Box
‒ oss@box.com‒ opensource.box.com‒ www.box.com/careers/
‒ Nadeem‒ nadeem@box.com‒ github.com/nadeemahmad
‒ Jordan‒ jmoldow@box.com‒ github.com/jmoldow
UsingtheTemplateandGuidelines
• Refertothisdocumentforrulesandrecommenda4onsforthetreatmentofgraphicelements
• Alwaysstartanewpresenta4onusingtheBoxcorporatetemplateandmakeeditsfromthere;thiswillensureyoualwayshavethecorrectstyles,colorpaleFeandfontsinuse
• Forlayoutinspira4on,pleaserefertothekeyslidespresenta4on
ColorandFontThemesHow to Save Themes…
StartbyopeningtheBoxCorporateTemplateandclickingontheDesigntab.
1
Savethecolorsfromthetemplatetoyoursystemby
clickingColorsandCreateNewThemeColorsandsaveasBox.
2
3
Dothesamethingforthetemplatefonts.CreateNewTheme
FontsandsaveasBox.
BulletSlideTreatment(CalibriRegular24pt)
• Bulletsaresentencecase(CalibriRegular20pt)‒ Second-linebulletsareCalibri16pt
‒ Third-linebulletsareCalibri14pt
• Limitthenumberofbulletsonaslide
• TexthighlightsareCalibriBold,butnotunderlined• Trynottogobelowtherecommendedfontsizes
BulletSlideWithSub-8tleTreatment
• Trytokeepyouruseofbulletslidestoaminimum
• Becrea4veandthinkvisually• IfyouneedtosourcesomethingcopyandpastethetextboxattheboFomlesontoyourpage
SubBtle Text Placeholder Title Case (Calibri 20pt)
Source:GrayCalibriRegular10pt
CenteredBulletSlideTreatment
• Bulletsaresentencecase(CalibriRegular20pt)‒ Second-linebulletsareCalibri16pt
‒ Third-linebulletsareCalibri14pt
• Limitthenumberofbulletsonaslide
• TexthighlightsareCalibriBold,butnotunderlined• Trynottogobelowtherecommendedfontsizes
CenteredBulletSlideWithSub-8tleTreatment
• Trytokeepyouruseofbulletslidestoaminimum
• Becrea4veandthinkvisually• IfyouneedtosourcesomethingcopyandpastethetextboxattheboFomlesontoyourpage
SubBtle Text Placeholder Title Case (Calibri 22pt)
Source:GrayCalibriRegular10pt
SegueSlideTitleGoesHereandCanBeaFewLinesinLengthSubBtle goes here in sentence case
SegueSlideTitleGoesHereandCanBeaFewLinesinLengthSubBtle goes here in sentence case
SegueSlideTitleGoesHereandCanBeaFewLinesinLengthSubBtle goes here in sentence case
SegueSlideTitleGoesHereandCanBeaFewLinesinLengthSubBtle goes here in sentence case
SegueSlideTitleGoesHereandCanBeaFewLinesinLengthSubBtle goes here in sentence case
SegueSlideTitleGoesHereandCanBeaFewLinesinLengthSubBtle goes here in sentence case
SegueSlideTitleGoesHereandCanBeaFewLinesinLengthSubBtle goes here in sentence case
Enter Btle for secBon one here and use sentence case
AgendaSlideDesign
Enter Btle for secBon two here and use sentence case
Enter Btle for secBon three here and use sentence case
Enter Btle for secBon four here and use sentence case
Enter Btle for secBon five here and use sentence case
1
2
3
4
5
Enter Btle for secBon one here and use sentence case
AgendaSlideDesignWithHighlightforEachSec8on
Enter Btle for secBon two here and use sentence case
Enter Btle for secBon three here and use sentence case
Enter Btle for secBon four here and use sentence case
Enter Btle for secBon five here and use sentence case
1
2
3
4
5
Enter Btle for secBon one here and use sentence case
AgendaSlideDesignWithHighlightforEachSec8on
Enter Btle for secBon two here and use sentence case
Enter Btle for secBon three here and use sentence case
Enter Btle for secBon four here and use sentence case
Enter Btle for secBon five here and use sentence case
1
2
3
4
5
Enter Btle for secBon one here and use sentence case
AgendaSlideDesignWithHighlightforEachSec8on
Enter Btle for secBon two here and use sentence case
Enter Btle for secBon three here and use sentence case
Enter Btle for secBon four here and use sentence case
Enter Btle for secBon five here and use sentence case
1
2
3
4
5
Enter Btle for secBon one here and use sentence case
AgendaSlideDesignWithHighlightforEachSec8on
Enter Btle for secBon two here and use sentence case
Enter Btle for secBon three here and use sentence case
Enter Btle for secBon four here and use sentence case
Enter Btle for secBon five here and use sentence case
1
2
3
4
5
Enter Btle for secBon one here and use sentence case
AgendaSlideDesignWithHighlightforEachSec8on
Enter Btle for secBon two here and use sentence case
Enter Btle for secBon three here and use sentence case
Enter Btle for secBon four here and use sentence case
Enter Btle for secBon five here and use sentence case
1
2
3
4
5
Enter4tleforsec4ononehereandusesentencecase
AgendaSlideColorCodingOp8onUse different colors to color code each secBon/segue slide
Enter4tleforsec4ontwohereandusesentencecase
Enter4tleforsec4onthreehereandusesentencecase
Enter4tleforsec4onfourhereandusesentencecase
Enter4tleforsec4onfivehereandusesentencecase
1
2
3
4
5
ChartSlideWithMul8pleColorsSub-Btle or Chart Title Here in Title Case
76
60
4457
65
86
72
0
20
40
60
80
100
Category1
Axis Title Goes Here in Title Case Calibri 16pt
Series 1
Series 2
Series 3
Series 4
Series 5
Series 6
Series 7
ChartSlideWithMul8pleDataBarsWithinEachCategorySub-Btle or Chart Title Here in Title Case
76 7569
88
60 6557
63
4450
6272
0
20
40
60
80
100
Category1 Category2 Category3 Category4
Axis Title Goes Here in Title Case Calibri 16pt
Series1
Series2
Series3
“
Placeaquotefromsomeonereally,reallyimportantanditcanbeuptothreelines…
AuthorNameHereAuthorTitleand/orDateHere
HEADERSuppor4ngtextgoeshereunderthenumber
HEADERSuppor4ngtextgoeshereunderthenumber
HEADERSuppor4ngtextgoeshereunderthenumber
2M 15M 7B
BigNumberTreatment
ProcessDiagramTreatmentSee Style Page for More Color OpBons
96
1 2 3 4Suppor4ngtextgoeshereunderthenumber
Suppor4ngtextgoeshereunderthenumber
Suppor4ngtextgoeshereunderthenumber
Suppor4ngtextgoeshereunderthenumber
ProcessDiagramTreatmentYou Can Use the Blue Bar to House Heavy Content
TitleGoesHere• Onebullet• Twobullet• Threebullet
TitleGoesHere• Onebullet• Twobullet• Threebullet
TitleGoesHere• Onebullet• Twobullet• Threebullet
TitleGoesHere• Onebullet• Twobullet• Threebullet
1 2 3 4
BoxWithBulletTreatment
• Onebullet• Twobullet• Threebullet
• Onebullet• Twobullet• Threebullet
• Onebullet• Twobullet• Threebullet
TitleHere TitleHere TitleHere
TableLayoutTreatmentSubBtle Text Placeholder Title Case (Calibri 20pt)
Header Header Header Header
TitleOne Informa4on Informa4on Informa4on
TitleTwo Informa4on Informa4on Informa4on
TitleThree Informa4on Informa4on Informa4on
TitleFour Informa4on Informa4on Informa4on
TitleFive Informa4on Informa4on Informa4on
TitleSix Informa4on Informa4on Informa4on
TitleSeven Informa4on Informa4on Informa4on
BigIdeaGoesHereWithHighlightinCalibriBold(40pt)
Usethisareaforabigidea,makesureyourimagehasenoughclearspace+contrastsoyoucanensurereadability
Youcanusethisareaforatexttreatmentthatsupportsyourchosenimagery
Youcanusethisareaforatexttreatmentthatsupportsyourchosenimagery
Youcanusethisareaforatexttreatmentthatsupportsyourchosenimagery
Youcanusethisareaforatexttreatmentthatsupportsyourchosenimagery
Youcanusethisareaforatexttreatmentthatsupportsyourchosenimagery
Youcanusethisareaforatexttreatmentthatsupportsyourchosenimagery
Youcanusethisareaforatexttreatmentthatsupportsyourchosenimagery
ImageTreatmentWithCap8onLayoutHow to Add Your Own Photos and Crop Properly…
InsertoneofthePicturewithCap4onlayouts.Clickontheiconatthecentertoaddyourimage.
1Yourimagewillpopulatethecontainerbutyouwilllikelyneedtoadjustthecrop.
2Clickontheimageandtheformattabandthenselectcrop.
3
ImageTreatmentWithCap8onLayout(Con8nued)How to Add Your Own Photos and Crop Properly…
Clickonthegrayedoutpor4onoftheimageanddragtothelesorrightun4lyouarehappywiththecrop.
4Clickoutofthecroppedimagetosetitandyourimageisreadytogo.
5
ImageWithBulletsTreatmentInsert Your Own Photo, See Slide 39 for InstrucBons
• Bullettexthere
• Somemorebullettext
• Anothershortbullet
• Lastbulletgoeshere
LogoWithBulletsTreatmentInsert Your Chosen Logo to the Lek of the Line
• Bullettexthere
• Somemorebullettext
• Anothershortbullet
• Lastbulletgoeshere
ImageTreatmentforTwoImagesSub-Btle or Secondary InformaBon Here
ImageTreatmentforThreeImagesSub-Btle or Secondary InformaBon Here
Mul8pleImageTreatmentSub-Btle or Secondary InformaBon Here
Mul8pleImageTreatmentHow to Insert MulBple Images…
Thislayoutwillappear.Next,clicktheicontoinsertanimage.
Navigatetoyourimageandinsertit.Repeatforalltheimageboxes.Forinstruc4onsonhowtocropyourimageseepage39.
ClicktheNewSlidetabandselectMul4pleImageLayout.
3
2
1
ScreenShotTreatmentWithBrowserWindow
ScreenShotTreatmentWithBrowserWindowHow to Drop in Your Screen Shot…
Thebrowserwindowislikeaframesoanythingyoudropbehinditwillshowthrough.
1
Dropinyourscreenshot,gototheformatmenuandcropittoshowonlywhatyouwant.
2
Last,besuretorightclickonyourscreenshotandsendtobacksoittucksbehindthebrowserwindowframe.
3
ColorPalePe
Hex# 22A7F0 Hex# 2A62B9 Hex# 2D3D4F Hex# BCC1C7
Hex# 3FB87F Hex# 955CA5 Hex# F7931D Hex# E33D55
Hex# 74808C
Primary
Secondary
StylesandTreatments
Clickontheselogos/iconsandyoucanrecolorthemwithinPowerPointasneeded.Besuretofollowbrandguidelinesforusageandcolors.
Varioustemplatecolorscanbeusedforshapes.Shapesshouldhavea4ptlinestroke.Gradientlinecanbeusedtoseparatecontentandshouldhavea1ptlinestroke.
Primary
Secondary
VideoinYourPresenta8onHow to Insert Video and Set Your Playback SeFngs…
Toinsertavideoclickinsertonthemenubar
Nextselectvideoandnavigatetoyourvideo.
Onceyourvideoisontheslideandyouclickonit,youwillgetanewmenutab.Youcannowmakechangestoboththeformatandplaybackofyourvideo.
Usethesesefngsforvideoplayback:StartAutoma4cally,PlayFullScreen,andHideWhileNotPlaying.ThiswillensureaconsistentlookwithinallBoxpresenta4ons.
1
2
3
4