Defining your own build system With Shake · Fast when everything changes •If everything changes,...
Transcript of Defining your own build system With Shake · Fast when everything changes •If everything changes,...
![Page 1: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/1.jpg)
Defining your own build system With Shake
Neil Mitchell
http://shakebuild.com
![Page 2: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/2.jpg)
Who has heard of Shake?
• Competitor to Make, Ant, Scons, Waf, Ninja…
• Better, because:
– Expressive (powerful dependencies)
– Fast (faster than all the above*)
– Robust (big test suite, large users)
– Haskell library (nice abstractions)
– …
![Page 3: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/3.jpg)
The tale of a large project
Day 1: Simple code, simple build system
Day 1000: Either repetitive code, or complex build system (usually both?)
• Little repetition => one source for data => generated files => hard for build systems
• Abstractions => types and higher-order => hard for build systems
![Page 4: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/4.jpg)
Generated files are hard
foo.c : foo.xml gen.sh gen.sh foo.xml > foo.c
foo.o : foo.c ??? gcc -c foo.c
Before you start, what does foo.c #include?
![Page 5: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/5.jpg)
Monadic dependencies
foo.c : foo.xml gen.sh gen.sh foo.xml > foo.c
foo.o : foo.c gcc -M foo.c | need gcc -c foo.c
After generating foo.c, what does it #include?
![Page 6: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/6.jpg)
Monadic dependencies
Determine future dependencies based on the results
of previous dependencies
![Page 7: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/7.jpg)
out : in cp in out
Simple Shake
"out" %> \out -> do
need ["in"]
cmd "cp in out"
:: Rule () Monad Rule :: Action ()
Monad Action
(%>) :: FilePattern -> (FilePath -> Action ()) -> Rule ()
![Page 8: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/8.jpg)
Congratulations
You now know Shake. (At least enough to start with)
![Page 9: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/9.jpg)
Your Goals for your Company
![Page 10: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/10.jpg)
Why sneak in with Shake?
• Robust software in commercial use for > 6 years
• Has a nice underlying theory
• Build system is always hairy and unloved
• Speeding up the build gives measureable gain
– 10 sec per build, 60 builds/day, 30 devs = 1 extra dev
• Easy to replace alongside
• Not production code, no license/distribute issues
• Only need one or two Haskellers (this talk) * Some of these apply to QuickCheck
![Page 11: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/11.jpg)
Build systems (Makefiles)
![Page 12: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/12.jpg)
Separate out metadata
Baked in (Haskell) Metadata (config)
How to run gcc Hack for Win98
Add Java binding Ship carrot.exe
Ship sprout.exe Ship mushroom.exe
carrot.hs comes from src/
mushroom.exe uses gcc
Haskell expert, changes rarely Everyone, ~10% of commits
![Page 13: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/13.jpg)
Metadata Example
• Bob’s green grocers build a set of .exe’s from C files.
• Identify the metadata!
– (What would be different if I had said Haskell files?)
![Page 14: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/14.jpg)
Some Metadata
carrot = veg orange anti_rabbit
mushroom = fungus mushroom
sprout = veg yuk green
build.cfg
![Page 15: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/15.jpg)
Prototype (1/4) - imports
import Development.Shake import Development.Shake.Config import Development.Shake.Util import System.FilePath
![Page 16: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/16.jpg)
Prototype (2/4) - main
main = shakeArgs shakeOptions $ do usingConfigFile "build.cfg" action $ do xs <- getConfigKeys need ["obj" </> x <.> "exe" | x <- xs]
![Page 17: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/17.jpg)
Prototype (3/4) - linking
"obj/*.exe" %> \out -> do Just xs <- getConfig $ takeBaseName out let os = ["obj" </> x <.> "o" | x <- words xs] need os cmd "gcc -o" [out] os
![Page 18: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/18.jpg)
Prototype (4/4) - compiling
"obj/*.o" %> \out -> do let src = takeBaseName out <.> "c" need [src] cmd "gcc -c" [src] "-o" [out]
![Page 19: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/19.jpg)
Prototype (5/4) - running it
cabal update && cabal install shake nano Shakefile.hs runhaskell Shakefile.hs
![Page 20: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/20.jpg)
Feedback from the team
• It works, it’s quick, and it’s already fully featured
– Profiling, progress prediction, parallelism
– Changes to build.cfg are tracked
– Supports most make command line options
• What’s missing?
![Page 21: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/21.jpg)
Enhancements (1/3) – header tracking
let src = takeBaseName out <.> "c" need [src] - cmd "gcc -c" [src] "-o" [out] + let m = out <.> "m" + () <- cmd "gcc -c" [src] "-o" [out] "-MMD -MF" [m] + neededMakefileDependencies m
![Page 22: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/22.jpg)
Enhancements (2/3) – cleaning
+ phony "clean" $ do + removeFilesAfter "obj" ["*"]
![Page 23: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/23.jpg)
Enhancements (3/3) – add lex
- let src = takeBaseName out <.> "c" + b <- doesFileExist $ takeBaseName out <.> "lex" + let src = (if b then ("obj" </>) else id) $ + takeBaseName out <.> "c"
+ "obj/*.c" %> \out -> do + let src = takeBaseName out <.> "lex" + need [src] + cmd "flex" ["-o" ++ out] src
![Page 24: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/24.jpg)
Winning over developers
• Must do everything actual developers want to do
• Must be more correct (less over/under building)
• Must be faster
• Win developers one-by-one
• After a few switch, go for the lead dev
• Old system quietly dies quite rapidly
![Page 25: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/25.jpg)
Progress prediction
• Guesses how long the build will take
– 3m12s more, is 82% complete
– Based on historical measurements plus guesses
– All scaled by a progress rate (guess at parallel setting)
– An approximation…
![Page 26: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/26.jpg)
Ready for primetime
• Standard Chartered have been using Shake since 2009, 1000’s of compiles per day.
• factis research GmbH use Shake to compile their Checkpad MED application.
• Samplecount have been using Shake since 2012, producing several open-source projects for working with Shake.
• CovenantEyes use Shake to build their Windows client.
• Keystone Tower Systems has a robotic welder with a Shake build system.
• FP Complete use Shake to build Docker images.
Don’t write a build system unless you have to!
![Page 27: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/27.jpg)
Tips for the conversion
• Preserve the same directory/filepath structure
– Even if it is crazy
• Focus on a single platform to start with
• Convert bottom-up
• Config file is a good approach
• Ask if you get stuck
– Mailing list
– Stack Overflow
![Page 28: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/28.jpg)
The GHC conversion (in progress)
• Following the previous slides (or vice versa)
• https://github.com/snowleopard/shaking-up-ghc
– Lead by Andrey Mokhov
alexArgs = builder Alex ? mconcat [ arg "-g" , package compiler ? arg "--latin1" , arg =<< getInput , arg "-o", arg =<< getOutput ]
![Page 29: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/29.jpg)
Speed
• Shake is typically faster than Ninja, Make etc.
• What does fast even mean?
– Everything changed? Rebuild from scratch.
– Nothing changed? Rebuild nothing.
• In practice, a blend, but optimise both extremes and you win
![Page 30: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/30.jpg)
Fast when everything changes
• If everything changes, rule dominate (you hope)
• One rule: Start things as soon as you can
– Dependencies should be fine grained
– Start spawning before checking everything
– Make use of multiple cores
– Randomise the order of dependencies (~15% faster)
• Expressive dependencies, Continuation monad, cheap threads, immutable values (easy in Haskell)
![Page 31: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/31.jpg)
Fast when nothing changes
• Don’t run users rules if you can avoid it
• Shake records a journal, [(k, v, …)]
• Avoid lots of locking/parallelism
– Take a lock, check storedValue a lot
• Binary serialisation is a bottleneck
unchanged journal = flip allM journal $ \(k,v) ->
(== Just v) <$> storedValue k
![Page 32: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/32.jpg)
Poll
• I am already using Shake
• I intend to start using Shake
• I won’t be using Shake
– I don’t have a suitably sized project
– The existing system works fine
– Not enough time to try it out
– Management won’t agree to it
– I want to use something else
– Other
![Page 33: Defining your own build system With Shake · Fast when everything changes •If everything changes, rule dominate (you hope) •One rule: Start things as soon as you can –Dependencies](https://reader033.fdocuments.us/reader033/viewer/2022053017/5f1b9a05a8f62f4c0d2b3738/html5/thumbnails/33.jpg)
Questions?
http://shakebuild.com