Make makefiles
description
Transcript of Make makefiles
![Page 1: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/1.jpg)
Make makefilespipelining for the masses
![Page 2: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/2.jpg)
Make is based on set of rules
# this is a remarktarget : prerequisites ...
recipe_line1recipe_line2...
<Tab>
![Page 3: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/3.jpg)
Simple example
Leg.pdf : Leg.treeR < plotTree.R Leg.tree
Leg.tree : Leg.alnRAxML -f a -m GTRGAMMA -s Leg.aln > Leg.tree
Leg.aln : Lpne.aln Ldra.aln Lisr.aln Ljor.alnperl fastaMsaConcat.pl Lpn.aln Ldra.aln … > Leg.aln
Lpne.aln : Lpne.fastaprank -F Lpne.fasta > Lpne.aln
Ldra.aln : Ldra.fastaprank -F Ldra.fasta > Ldra.aln
Lisr.aln : Lisr.fasta
prank -F Lisr.fasta > Lisr.aln
Lsha.aln : Lsha.fasta
prank -F Lsha.fasta > Lsha.aln
Recipe will be executed if either:
1. Target does not exist
2. Target is older than prerequisite
![Page 4: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/4.jpg)
Makefile is a tree
Leg.pdf
Leg.tree
Leg.aln
Lpne.aln
Lpne.fas
Ldra.aln
Ldra.fas
Lisr.aln
Lisr.fas
Lsha.aln
Lsha.fas
Recipe will be executed if either:
1. Target does not exist
2. Target is older than prerequisite
![Page 5: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/5.jpg)
Simple example
Leg.pdf : Leg.treeR < plotTree.R Leg.tree
Leg.tree : Leg.alnRAxML -f a -m GTRGAMMA -s Leg.aln > Leg.tree
Leg.aln : Lpne.aln Ldra.aln Lisr.aln Ljor.alnperl fastaMsaConcat.pl Lpn.aln Ldra.aln … > Leg.aln
Lpne.aln : Lpne.fastaprank -F Lpne.fasta > Lpne.aln
Ldra.aln : Ldra.fastaprank -F Ldra.fasta > Ldra.aln
Lisr.aln : Lisr.fasta
prank -F Lisr.fasta > Lisr.aln
Lsha.aln : Lsha.fasta
prank -F Lsha.fasta > Lsha.aln
Makefile makes the first target (the order of the rest is not important
![Page 6: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/6.jpg)
Variables
alns = Lpne.aln Ldra.aln Lisr.aln Ljor.aln
Leg.tree.pdf : Leg.treeR < plotTree.R Leg.tree
Leg.tree : Leg.alnRAxML -f a -m GTRGAMMA -s Leg.aln > Leg.tree
Leg.aln : $(alns)perl fastaMsaConcat.pl $(alns) > Leg.aln
Lpne.aln : Lpne.fastaprank -F Lpne.fasta > Lpne.aln
Ldra.aln : Ldra.fastaprank -F Ldra.fasta > Ldra.aln
Lisr.aln : Lisr.fasta
prank -F Lisr.fasta > Lisr.aln
Lsha.aln : Lsha.fasta
prank -F Lsha.fasta > Lsha.aln
![Page 7: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/7.jpg)
Variablesdir = ~/Legionella/Genomes/phase1/
alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln …
$(dir)Leg.tree.pdf : $(dir)Leg.treeR < plotTree.R $(dir)Leg.tree
$(dir)Leg.tree : $(dir)Leg.alnRAxML -f a -s $(dir)Leg.aln > $(dir)Leg.tree
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $(alns) > $(dir)Leg.aln
$(dir)Lpne.aln : $(dir)Lpne.fastaprank -F $(dir)Lpne.fasta > $(dir)Lpne.aln
$(dir)Ldra.aln : $(dir)Ldra.fastaprank -F $(dir)Ldra.fasta > $(dir)Ldra.aln
$(dir)Lisr.aln : $(dir)Lisr.fasta
prank -F $(dir)Lisr.fasta > $(dir)Lisr.aln
$(dir)Lsha.aln : $(dir)Lsha.fasta
prank -F $(dir)Lsha.fasta > $(dir)Lsha.aln
![Page 8: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/8.jpg)
Special Variablesdir = ~/Legionella/Genomes/phase1/
alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln …
$(dir)Leg.tree.pdf : $(dir)Leg.treeR < plotTree.R $^
$(dir)Leg.tree : $(dir)Leg.alnRAxML -f a -s $^ > $@
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $^ > $@
$(dir)Lpne.aln : $(dir)Lpne.fastaprank -F $^ > $@
$(dir)Ldra.aln : $(dir)Ldra.fastaprank -F $^ > $@
$(dir)Lisr.aln : $(dir)Lisr.fasta
prank -F $^ > $@
$(dir)Lsha.aln : $(dir)Lsha.fasta
prank -F $^ > $@
Useful special variables:
$@ - the target
$^ - all prerequisites
$< - first prerequisite
![Page 9: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/9.jpg)
Rulezdir = ~/Legionella/Genomes/phase1/
alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln …
$(dir)Leg.tree.pdf : $(dir)Leg.treeR < plotTree.R $^
$(dir)Leg.tree : $(dir)Leg.alnRAxML -f a -s $^ > $@
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $^ > $@
%.aln : %.fastaprank -F $^ > $@
Another useful special
variable is $* that
matches %
Rules will simplify your makefile and raise level
of abstraction
![Page 10: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/10.jpg)
Rulezdir = ~/Legionella/Genomes/phase1/
alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln …
%.tree.pdf : %.treeR < plotTree.R $^
%.tree : %.alnRAxML -f a -s $^ > $@
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $^ > $@
%.aln : %.fastaprank -F $^ > $@
General rules can be transferred among
makefiles
Intermediate files, which are created by rules and are not
specified as targets themselves, are by default deleted after no
longer needed.(in a few slides we'll see how to change
that behavior)
![Page 11: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/11.jpg)
Some more variablesdir = ~/Legionella/Genomes/phase1/
alns = $(dir)Lpne.aln $(dir)Ldra.aln $(dir)Lisr.aln …
ALIGN = prank -F
RECONSTRUCT = RAxML -f a -s
%.tree.pdf : %.treeR < plotTree.R $^
%.tree : %.aln $(RECONSTRUCT) $^ > $@
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $^ > $@
%.aln : %.fasta $(ALIGN) $^ > $@
Use variables for things you think you
might want to change between runs (e.g.,
BLAST flags)
![Page 12: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/12.jpg)
Some functions (there are many many more…)
$(addsuffix <SUFF>,<LIST>)
$(addsuffix .ext,a dir/b) => a.ext dir/b.ext
$(addprefix <PREF>,<LIST>)
$(addprefix dir/,a b.ext) => dir/a dir/b.ext
$(dir <LIST>)
$(dir tmp/a dir/b.ext c) => temp/ dir/ ./
$(notdir <LIST>)
$(notdir dir/b.ext c) => b.ext c
$(basename <LIST>)
$(notdir a.R dir/b.ext c) => a dir/b c
$(shell <command>)
![Page 13: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/13.jpg)
Function usagedir = ~/Legionella/Genomes/phase1/
alns = $(addprefix $(dir), Lpne.aln Ldra.aln Lisr.aln …)
ALIGN = prank -F
RECONSTRUCT = RAxML -f a -s
%.tree.pdf : %.treeR < plotTree.R $^
%.tree : %.aln $(RECONSTRUCT) $^ > $@
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $^ > $@
%.aln : %.fasta $(ALIGN) $^ > $@
![Page 14: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/14.jpg)
Function usagedir = ~/Legionella/Genomes/phase1/
alns = $(addprefix $(dir),$(addsuffix .aln, Lpne Ldra Lisr …))
ALIGN = prank -F
RECONSTRUCT = RAxML -f a -s
%.tree.pdf : %.treeR < plotTree.R $^
%.tree : %.aln $(RECONSTRUCT) $^ > $@
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $^ > $@
%.aln : %.fasta $(ALIGN) $^ > $@
![Page 15: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/15.jpg)
More functions$(shell <command>)
Executes shell command and returns values as list
(newlines are turned to spaces).
$(shell cat b.ext) => the content of b.ext(as a list)
$(LIST:.fas=.aln)
Changes all extensions of LIST
Legs = Lpne.fas Ldra.fas Lisr.fas Ljor.fas
$(Legs:.aln=.fas) => Lpne.aln Ldra.aln Lisr.aln Ljor.aln
![Page 16: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/16.jpg)
Function usagedir = ~/Legionella/Genomes/phase1/
alns = $($(shell ls $(dir)*.fasta):.fasta=.aln)
ALIGN = prank -F
RECONSTRUCT = RAxML -f a -s
%.tree.pdf : %.treeR < plotTree.R $^
%.tree : %.aln $(RECONSTRUCT) $^ > $@
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $^ > $@
%.aln : %.fasta $(ALIGN) $^ > $@
![Page 17: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/17.jpg)
Special targetsdir = ~/Legionella/Genomes/phase1/
alns = $($(shell ls $(dir)*.fasta):.fasta=.aln)
ALIGN = prank -F
RECONSTRUCT = RAxML -f a -s
All : $(dir)Leg.tree.pdf
%.tree.pdf : %.treeR < plotTree.R $^
%.tree : %.aln $(RECONSTRUCT) $^ > $@
$(dir)Leg.aln : $(alns)perl fastaMsaConcat.pl $^ > $@
%.aln : %.fasta $(ALIGN) $^ > $@
clean:
rm -vf $(dir)Leg.tree.pdf $(dir)Leg.tree $(dir)*.aln
.SECONDARY: <targets>Intermediate targets not to removed (no target will save all)
.Phony : <targets>A phony target is one that is not really the name of a file. It is just a name for some commands to be executed when you make an explicit request
![Page 18: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/18.jpg)
Invoking makefileBy default run make to execute the first target of the file called makefile
> make
You can invoke a specific target in a specific file:
> make -f <makefile name> <TARGET>
make -f makeTree.mk clean
You can also pass values of variables (will override variables with in the makefile
> make -f <makefile name> <VAR>=<value>
make -f makeTree.mk DIR=/mydir/
![Page 19: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/19.jpg)
Useful flags for make file-j <n>: use n threads (without n - as many as possible)
as makefile knows the dependencies he can run the non-dependent recipes in parallel…
-n : just echo commands, do not excute
-d : debug (shows you what makefile thinks)
![Page 20: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/20.jpg)
REAL-LIFE EXAMPLESLet's check out some
![Page 21: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/21.jpg)
Here are some makefiles I did
① A real makefile example:http://www.tau.ac.il/~davidbur/makePhylogeny.mk
② Huge (but well documented) makefile:http://www.tau.ac.il/~davidbur/makeCompileFeaturesByPtts.makefile
③ I use such a makefile to run the one in (2):http://www.tau.ac.il/~davidbur/makefile
④ This makefile creates a makefile similar to (3) and executes it (getting weird, isn't it…):http://www.tau.ac.il/~davidbur/makeLearningHappen.makefile
![Page 22: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/22.jpg)
useful tips and hints - before the commnad will tell make to continue even if the
execution of that line fails When using "$", for example in a Perl oneliner, use $$ to let
makefile know you mean the character '$' Beware of extra spaces at the end of lines You can use the program/script itself as part of prerequisit Some time it is useful to invoke make from within a makefile You can use include FILE to add the content of FILE to this
point in the makefile
![Page 23: Make makefiles](https://reader036.fdocuments.us/reader036/viewer/2022081505/568166cc550346895ddad910/html5/thumbnails/23.jpg)
ReferencesGNU make tutorial, everything you want to know about make:
http://www.gnu.org/software/make/manual/make.html
Eyali's Blog in the post (my initial inspiration):
http://lifeofadataminer.blogspot.co.il/2009/08/writing-makefile-instead-of-pipeline.html
A pipeline is a make (I think his initial inspiration):
http://archive.nodalpoint.org/2007/03/18/a_pipeline_is_a_makefilehttp://archive.nodalpoint.org/2007/03/18/a_pipeline_is_a_makefile