Post on 15-Feb-2016
description
CS 294-73 (CCN 27241)Software Engineering for
Scientific Computinghttp://www.cs.berkeley.edu/~colella/CS294
Lecture 4: Development Tools
08/25/2011 CS294-73 – Lecture 4
Revision Control Systems• Modern scientific computing is no longer a solo effort
- Most interesting modeling questions that could be simulated by the heroic individual programming scientist have already been investigated
- “Productivity language” that are meant to alleviate the complexity of programming high performance software have not delivered yet
- Thus, coding is complicated and requires division of roles and responsibilities.
• Working together on a common code is very error prone without some technology assistance.
• Two tools to be discussed in this part- Concurrent Version System: cvs- Subversion: svn- Very similar in user interface
2
08/25/2011 CS294-73 – Lecture 4
Concurrent Version System• Central Repository system.
- There is one master version of the state of the code
• Users have “check outs” or “working copy” of the master respository
• Can access the master respository via several mechanisms
- rsh connection- ssh connection- cvsserver - All user interaction is considered a client-side operation- Transactional protocol
3
08/25/2011 CS294-73 – Lecture 4
SSH protocol interaction• Setting up keychains (ssh-keygen, ssh-add, ssh-agent)
- Typing in your password for every single transaction with the central respository is a pain
ssh-keygen >ssh-keygen –t dsa // you will be prompted for passphrase>ssh anag.lbl.gov>mkdir .ssh:mkdir: .ssh: File exists>exit>cd .ssh>scp id_dsa.pub –l username anag.lbl.gov:.ssh/tempfile>ssh –l username anag.lbl.gov ;cd .ssh>cat authorized_keys2 tempfile >authorized_keys2>exit
ssh-add>ssh-add // provide passphrase
- Try ssh again and see if your keychain is working.
4
08/25/2011 CS294-73 – Lecture 4
Setting up for CVS repository work• Environment
- CVSROOT=:ext:username@anag.lbl.gov:/usr/local/cvsroot- CVS_RSH=ssh- EDITOR=vi
• Commands- >cvs checkout Chombo- >cvs add [filename|directory]- >cvs update- >cvs update –d // by default update only updates directories that were
in this level of the repo when you did your checkout. -d says bring in any new directories.
- M: Locally Modified or Merging, apparently without conflicts. This is the essence of concurrent version system
- C: merge got confused, Conflicts remain. Your local file has been modified and >>> delimiters have been put in the file telling you where you need to pick a winner, and merge by hand.
- U or P: Repo version has some updated file that you haven’t touched, your copy replaced.
5
08/25/2011 CS294-73 – Lecture 4
• Commands Cont.- >cvs diff
- Show me the difference between my local file and the current repo version as of when I took an update.
- >cvs log- Show a modification history of a file
- >cvs stat- Show the repo info of my working copy of this file
– Tags! - >cvs commit
- Commit my local working copy of file(s) as the new repo version of these files
- User prompted to use EDITOR to create a commit comment- >cvs remove // and watch the horror of ‘remove’ing directories
• You can also use GUI-based tools that invoke these commands for you and help keep things straight
6
08/25/2011 CS294-73 – Lecture 4
Tags• Creates a string to associate with a collection of files and
their individual revisions• >cvs tag backpoint1 .
- This is a “working copy” tag. You keep this info in your own local copy
• >cvs rtag -D <date> <new_tag> [file | directory | module]- This is a repository tag. It modifies the repository to associate the
new_tag with some revision of the [file|directory|module]- -D is a handy way of setting a tag of the repo at some point in time.
• HEAD is a special repo tag reserved for the trunk (non-branched part of the repo)
- We use this this special tag when working with branches
7
08/25/2011 CS294-73 – Lecture 4
Branches• …. uhm, if nobody objects, I’ll leave this one for now.
- Branches in CVS are a confusing issue
• Better to work in Subversion if you are needing to work with complicated branches
8
08/25/2011 CS294-73 – Lecture 4
Subversion: SVN• Central Repository system.
- There is one master version of the state of the code
• Users have “check outs” or “working copy” of the master respository
• Can access the master respository via several mechanisms
- rsh connection- ssh connection- svnserver - All user interaction is considered a client-side operation- Transactional protocol
9
08/25/2011 CS294-73 – Lecture 4
Working with SVN• >svn checkout svn+ssh://anag.lbl.gov/usr/local/svnroot/Chombo4 MyLocalName
• >svn update - Also a merging/concurrent process, as with CVS- No –d process here. Better, think of it as always applied
• >svn log filename• >svn diff [filename|directory]• >svn add [filename|directory]• >svn commit [ |filename|directory]• >svn delete [filename|directory]
10
08/25/2011 CS294-73 – Lecture 4
SVN Branches and Tags: copy• Tags and Branches and similar concepts are all handled
by the universal copy command• >svn copyhttp://svn.example.com/repos/calc/trunk http://svn.example.com/repos/calc/branches/my-calc-branch -m "Creating a private branch of /calc/trunk.”
- Here we are using the http protocol for svn. The same command using ssh protocol would be similar
- svn copy svn+ssh://svn.example.com/repos/calc/trunk svn+ssh://svn.example.com/repos/calc/branches/my-calc-branch -m "Creating a private branch of /calc/trunk.”
• Tags are just the same as branches. Everything is a copy
• Subversion database is clever to not really copy everything. Copies are “shallow” until you modify a file
11
08/25/2011 CS294-73 – Lecture 4
Merging• In svn, everything is just a copy.
- Branches- Tags- Other SVN repos
• Full Syntax- >svn merge left-tree@rev right-tree@rev working-set- users will almost never use fully specified version
- But, it is good to know that the full format exists
• Typical syntax- >svn merge svn+ssh://anag.lbl.gov/usr/local/svnroot/Chombo4- Only one argument given here, and no rev specified, so…..
- right-tree and working-set is assumed to be the current directory- The left-rev is assumed to be the most recent- You’ve done an >svn update
12
08/25/2011 CS294-73 – Lecture 4
Branch and Merge• >export sroot=svn+ssh://usr/local/anag/svnroot• >svn copy $sroot/Chombo4/trunk $sroot/Chombo4/branches/GPU_Experiment
• >svn checkout $sroot/Chombo4/branches/GPU_Experiment gpuExperiment
• Edit and develop code in gpuExperiment• Other users can check out your branch and your
updates and commits and add etc. operations are seen by each other
• >svn merge $svnroot/Chombo4/trunk- Keep up to date with edits made to the main development trunk
• Is the branch code suitable for the core trunk code ?• >cd tmp; svn checkout $svnroot/Chombo4/trunk trunk• >cd trunk; svn merge $sroot/Chombo4/branches/GPU_Experiment
13
08/25/2011 CS294-73 – Lecture 4
GNU Make• A tricky bit of script parsing to manipulate files specialized to work
well with compiling code- lots of features to let you do simple things simply.- complicated things without too much work.- almost impossible to figure out what is going wrong.- Main purpose: turn a set of source code into a library or executable.
• Only two kinds of objects in a Makefile- Variables (lists of strings)- Rules
• Only a few kinds of flow control- ifeq/ifneq/else/endif- No forms or looping available, no jumps, no recursion.
• Most difficulties arising from make are related to- Non-trivial variable parsing of the makefile(s)- Rules can fire and trigger in non-obvious ways- The mysteries of regex
14
08/25/2011 CS294-73 – Lecture 4
The Two type of Variables in GNU Make• Recursively Expanded Variables “=“
foo = $(bar)bar = $(ugh) ugh = Huh? all:;echo $(foo)
> make allHuh?
• Variable is executed at the time it is used in a command• = means build up a symbol table for this name• Notice $. Like in shell, there is the value ‘bar’ and the variable
named ‘bar’
15
08/25/2011 CS294-73 – Lecture 4
• Good points:- Order doesn’t matter! Go and try it.- Can declare a variable as the composite of many other variables that
can filled in by other parts of the Makefile- CFLAGS = $(DEBUG_FLAGS) $(OPT_FLAG) $(LIB_FLAGS)- Lets a makefile build up sophisticated variables when you don’t know
all the suitable inputs, or what parts of the Makefile they will come from- >make all DIM=3
• Bad points:- No appending
- # error, causes infinite loop- CFLAGS = $(CFLAGS) –c
- Future = declarations can clobber what you specified- The last = declaration in the linear parsing of a Makefile is the only one
that matters
16
08/25/2011 CS294-73 – Lecture 4
• Simply Expanded Variables “:=“- Immediate mode variable.- The variable is assigned it’s value based on the current state of the
Makefile parsing- No symbol chain is created.- Specific to GNU Make
• Often just an easier to understand variable. - It acts like variables you know in other languages.- can use for appending
- CFLAGS := $(CFLAGS) –c –e –mmx
17
08/25/2011 CS294-73 – Lecture 4
Rulestargets : prerequisites [TAB] recipe[TAB] recipe
• prerequisites are also called “sources”
• Simple exampleclobber.o : clobber.cpp clobber.h config.h[TAB] g++ -c –o clobber.o clobber.cppclob.ex : clobber.o killerApp.o [TAB] g++ -o clob.ex cobber.o killerApp.o
18
08/25/2011 CS294-73 – Lecture 4
More powerful rules• Pattern Rules
%.o : %.cpp$(CC) -c $(CFLAGS) $(CPPFLAGS) $< -o $@#Gives a pattern that can turn a .cpp file into a .o file
• Multitarget Rules%.f %.H : %.ChF
• Suffix Rules- .c.o:
- $(CC) -c $(CFLAGS) $(CPPFLAGS) -o $@ $<
19
08/25/2011 CS294-73 – Lecture 4
Other Makefile commands• include• $(MAKE)
- calling a makefile from inside a recipe- $(MAKELEVEL) can be looked at to see how deep the call stack is
• export- send variables from this level of make to lower makelevels
• subst- CFLAGs:= $(CFLAGS) $(subst FALSE,,$(subst TRUE,-DCH_MPI $(mpicppflags),$(MPI)))
• foreach- libincludes = $(foreach i,$(LibNames),-I$(CHOMBO_HOME)/src/$i)
20
08/25/2011 CS294-73 – Lecture 4
What the “make” program does• Much mental confusion about make comes from thinking
that the Makefile is the make program- Easy to see. It looks like a shell script.- Remember: Makefile is only Variables & Rules
• make:- parses all of your Makefile- builds up variable chains (overriding variables defined on command
line)- builds up rules database- Then looks at what target the user has specified- make then attempts to create a chain of rules from the files that exist to
the targets specified. - recursive “=“ variables in source-target expressions are evaluated
- Using the date stamp on files discovered in the chain make executes recipes to deliver the target.
- “=“ variables are evaluated in recipes.
21
08/25/2011 CS294-73 – Lecture 4
Demonstration of the pervasive Make ‘error’FooBar = trendyF:= fashionvars:
@echo $(FooBar) $(F)
ifeq ($(F),fashion) FooBar=tragicendifF:= comedy>make varstragic comedy>
22
08/25/2011 CS294-73 – Lecture 4
Chombo Makefiles Variables of interest• DIM?=2• DEBUG?=TRUE• # OPT=FALSE• PRECISION?=DOUBLE• MPI?=FALSE• CXX?=g++• FC?=g77• MPICXX?=mpiCC
23
• USE_64?=TRUE #64bit pointers
• USE_MT?=TRUE #memory tracking
• USE_COMPLEX=TRUE #type for complexnumbers
• USE_TIMER?=TRUE #Chombo::Timer
• USE_HDF?=TRUE #HDF5 file i/o
?= special assignment, only if not already set.
08/25/2011 CS294-73 – Lecture 4
HDF5http://www.hdfgroup.org/ftp/HDF5/current/src/
building from the source code is the best option. download the source file
hdf5-1.8.7.tar.gz
cd /usr/localsudo tar xzf hdf5-1.8.7.tar.gzcp -r hdf5-1.8.7 hdf5-1.8.7.parallelcd hdf5-1.8.7sudo mkdir buildcd buildsudo ../configure --enable-production --prefix=/usr/local/hdf5-1.8.7 --with-default-api-version=v16sudo make allsudo make testsudo make installcd /usr/local/hdf5-1.8.7.parallelsudo mkdir buildcd buildexport CC=mpic++sudo ../configure --enable-parallel --enable-production --prefix=/usr/local/hdf5-1.8.7.parallel --with-default-api-version=v16sudo make allsudo make testsudo make install
your mpi compiler might have a different name on your system.
24
08/25/2011 CS294-73 – Lecture 4
Writing out an HDF5 file from Chombo• Chombo/lib/src/AMRIO.H
voidWriteAMRHierarchyHDF5(const string& filename, const Vector<DisjointBoxLayout>& a_vectGrids, const Vector<LevelData<FArrayBox>* > & a_vectData, const Vector<string>& a_vectNames, const Box& a_domain, const Real& a_dx, const Real& a_dt, const Real& a_time, const Vector<int>& a_vectRatio, const int& a_numLevels);
• There are several variations on this function for special uses.• Do not use the write[FAB|Level|etc] functions in your code.
- Those are for debugging. To be discussed later.
25
08/25/2011 CS294-73 – Lecture 4
Command Line Make>cvs checkout Chombo>cd Chombo/lib/mk>ln –s local/Make.defs.leopard Make.defs.local>cd ..>make –j4 all DIM=3 MPI=TRUE OPT=HIGH>ls
libboxtools3d.Darwin.64.mpic++.gfortran.DEBUG.OPTHIGH.MPI.alibbasetools3d.Darwin.64.mpic++.gfortran.DEBUG.OPTHIGH.MPI.a.
26