Smoking docker
-
Upload
workhorse-computing -
Category
Technology
-
view
115 -
download
0
Transcript of Smoking docker
What is “Smoking”?● High-level sanity check, not detailed analysis.
Does it catch fire when we power it on?
● CPAN runs tests as part of installation.
We are unusual that way.
● CPAN::Reporter reports results of tests.● CPAN::Reporter::Smoker builds with reporting.
What do you smoke?● Whatever is in your sandbox.● A project and all of its supporting code.● All of CPAN, or the RECENTS file.
How do you smoke?● “make -wk -C myproject all test” in your sandbox.● Nightly build, git hooks, Travis CI.● CPAN or CPANM smoke testers.
Risks of Smoking● Pull in arbitrary code.
Maybe broken.
Possibly malicious.
More likely just stupid
system( rm -rf / *.foobar );
Second-hand Smoke● Permanent side effects.● Left over test files.● Wasted disk space.● Bandwidth.● Still possible with personal tests:
Dependent modules.
One approach: Virtual Machines● Heavy footprint.● Fully configure a running machine to run a test.● Inefficient use of hardware.
Lightweigit: “chroot”● Filesystem isolation only.● Tests can interfere with network, processes.● Files pollute permenant storage.● No resource limits.
Another approach: Process isolation.● Keep processes from interfering with one another.● Lightweight: Share logger, daemons, filesytems...● More efficient use of CPU.● Less work to set up.
● Not always easy to do right.
Jails & Zones
Better way: lxc● Approach via IBM.
Mainframe LPAR ported to linux.● Uses process groups.● Excellent control.
● One problem: People used them like VM's.● Two problem: lxc has hundreds of switches.
lxc for the Masses
● Docker: 80/20 of lxc.
Medium isolation.● Generate an “image”.
Read-only stack of AUFS mounts.● Run a “container”.
Mountpoint + resources + isolation.● Decent manglement tools.
AUFS with lxc
● Images are layered.
Read-only.
● "Container layer"
updated.
discarded.
Good and bad:
No permenant updates.
Not so nice● Images look good for smoking:
Read-only underneath: minimal harm.
Re-cycle O/S layer with Perly build.
Allows mix'n match testing.
● Catch:
No fixes or upgrades to smokebox.
Changes require complete rebuild.
One fix: Volumes● Described as “data volumes”.● Can store anything.● Like Perl, build directory, CPAN mirror.
● Mountpoints and symlinks.
Dir's or files from O/S into container.
● Mount RO or RW.
Maintain Perl & Smoker.
Hybrid approach● O/S is an image.
● Perl, CPAN on a volume.
O/S tools build and maintain perl.
perl maintains /var/lib/CPAN
perly volumes● Tempting: perlbrew dir's as volumes.
Problems with storage in /home.
Hard to share across users.
Storage depends on user account.
Multiple accounts duplicate space.
● Alternate: Let Docker manage the volumes
Simpler sharing.
/var/lib/docker [hopefully] has more space.
Manage via docker.
Perl Testers Wiki● Good instructions for CPAN::Reporter::Smoker.● Start with an O/S image and get a working smoker.● CPANM has its own version.
"minismokebox"
Grab a distro● Docker Hub has hundreds of distro images.● I'll use Gentoo here: it includes build environment.
No need to install gcc, make...
“stage3” is the starting point.
Installing Stage3● docker import $tarball into gentoo-stage3:YYYYMMDD.● Layer custom files into smoker-gentoo-stage3:lYYYYMMDD.
root's .exrc .bashrc● Used to bulid perl, run smoke.
Building perl, smoker● Volumes for /opt, /var/tmp & script files.
Standard paths simplify build scripts.● “docker run” with command of build script.
build perl.
install & run minicpan.
configure reporting & smoker.
CPAN update to test reporting.
Aside: --tmpfs● docker run –tmpfs has a problem:
The volume has execution turned off.
Fine for data.
Useless for build.● Avoid the issue with a tmpfs volume.
Volume must be private to build.
Creating a tmpfs
● Private tmp volume via PID.
● tmpfs usually cleaned up automatically on the way out.
● Use volume rm to be sure.
tmp=”var-tmp-$BASHPID”;$docker volume create
--driver=local--opt type=tmpfs--opt device=tmpfs--opt o=size=320M$tmp
&&docker run
...-v $tmp:/var/tmp...
;# may fail due to non-existant volume.docker volume rm $tmp;
Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz:/var/tmp/perl.tar.gz‑
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN
Start the ball rolling
● Assemble a container.● Dispatch the build script.
● “-t” avoids issues with core test that uses column/row counts.
● “-a” gets output from container into log.
$docker volume create \ --driver=local \ --opt type=tmpfs \ --opt device=tmpfs \ --opt o=size=320M \ $tmp &&$docker volume create \ --driver=local \ $perl &&$docker run --rm \ -t \
--name=”build-$perl”\ -a stdout \ -a stderr \ "${vols[@]}" \ $image \ /var/tmp/build-perl \2>&1 | tee build.out;
docker volume rm $tmp;
Building Perl● The fun starts in /var/tmp/build-perl● Mostly here-scripts.● Install & configure smoketest engine.● Includes common non-smoke modules.
Building PerlHOME='/var/lib/CPAN';
( cd /var/tmp; gzip -dc < perl.tar.gz | tar xf -; cd perl*;
jobs=$(grep '^processor' /proc/cpuinfo | wc -l);export TEST_JOBS=$jobs;
./Configure \ -de \ -Dprefix=/opt/perl5 \ -Dman1dir='none' \ -Dman3dir='none' \ -Doptimize='-O2 -pipe -march=native' \ && /usr/bin/make -j$jobs all test_harness install;
) ||exit -1;
rm -rf /var/tmp/perl*;
● Generic “perl.tar.gz” via docker volume mapping.
● /opt/perl5 is in external volume.
● Don't knead man pages.
Building PerlHOME='/var/lib/CPAN';
● /home is in the O/S image!
Changes in container layer are discarded!!
● Persistent files go in /var/lib/CPAN or /opt.
Building Perl
TEST_JOBS=$jobs ... /usr/bin/make -j$jobs all test_harness install;
Files=2410, Tests=851044, 223 wallclock secs(172.97 usr 22.15 sys + 862.57 cusr 57.18 csys = 1114.87 CPU)
Aside: Testing perl builds● Use tmpfs for /opt instead of disk volume.
Fast, disposable storage for validating your build scripts.
● git clone <whatever> /scratch;
cd /scratch/whatever;
● docker run … -v /scratch/whatever:/var/tmp/whatever … ;
● build script does cd /var/tmp/whatever; make all test install;
Testing bleeding edge perl● Replace tar xvf with volume:
-v /gitclone/perl-5.25.X:/var/tmp/perl
● Rest of it stays the same.
Configure and make run quickly.
Minimal overhead if core modules don't change.
Lots “up to date” messages from CPAN.
Minor setups
● “h2ph” validates $PATH.
● Pick your remote server.
export PATH=/opt/perl5/bin:$PATH;
h2ph -r -l /usr/include || exit -1;
# at this point install seems usable
lib=$(ls -d /opt/perl5/lib/5* | tail -1);site=$(ls -d /opt/perl5/lib/site_perl/5* | tail -1);
cpan='/var/lib/CPAN';build='/var/tmp/CPAN';
remote='http://mirror.uic.edu/CPAN/';local="$cpan/sources/";
CPAN uses /var/lib, /var/tmp
● Push CPAN config onto stable storage.
● CPAN config is specific to perl version.
perl -MCPAN -e 'shell'<<CPAN;yeso conf cpan_home $cpano conf histfile $cpan/histfileo conf prefs_dir $cpan/prefso conf build_dir $buildo conf keep_source_where $localo conf auto_commit yeso conf build_dir_reuse yeso conf check_sigs yeso conf inhibit_startup_message yeso conf halt_on_failure noo conf build_cache 1o conf index_expire 1o conf commitCPAN
TLC
● Update CPAN.● Deal with modules
that require user input, special choices.
(CPAN_RUN_SHELL_TEST_WITHOUT_EXPECT=1;
perl -MCPAN -e 'install CPAN';
perl -MCPAN -e shell <<CPAN;install Module::Signature2CPAN
AUTOMATED_TESTING=1; # avoid interactive prompt
perl -MCPAN -e 'shell <<CPAN;install Term::Readline::Perl
CPAN)
● Common.● Smoke modules.
● Iterate them with ${modules[@]}.
modules=( YAML::XS JSON::XS CPAN::SQLite IO::Socket::SSL Net::SSL Net::SSLeay HTTP::Date Test::Most Metabase::Resource Module::Version Module::Version Bundle::CPAN Log::Log4perl Bundle::CPAN CPAN::Mini CPAN::Reporter CPAN::Testers Test::RequiresInternet WWW::RobotRules LWP::UserAgent WWW::Mechanize CPAN::Reporter::Smoker Bundle::CPAN::Reporter::Smoker::Tests Bundle::CPANReporter2 App::SmokeBox::Mini App::cpanminus::reporter);
Other modules
Generate ID file for Reporter
● ~ is /var/lib/CPAN
Not /home!
( mkdir ~/.cpantesters; cd ~/.cpantesters; metabase-profile <<END;Steven [email protected]/secretEND chmod 0400 metabase_id.json;)
Configure CPAN::Reporter
# Generate test reports if CPAN::Reporter is installed (yes/no)? [no] yes# Would you like me configure CPAN::Reporter now? [yes] <enter># email_from? [] [email protected]# edit_report? [default:ask/no pass/na:no] <enter># send_report? [default:ask/yes pass/na:yes] <enter># transport? [Metabase uri https://metabase.cpantesters.org/api/v1/ id_file metabase_id.json] <enter># Would you like to run 'metabase-profile' now to create '/root/.cpanreporter/metabase_id.json'? [y] <enter>
perl -MCPAN -e shell <<CPAN;o conf init [email protected]
CPAN
Voyeurism● Fixing the build uses “docker exec”.● Attach to a running container.● Does not require ssh.● Allows fixing, re-running the build.
Voyeurism● Fixing the build uses “docker exec”.● Attach to a running container.● Does not require ssh.● Allows fixing, re-running the build.
● Or just watching it:
docker exec -it build-perl-5.24.1 'top';
Access a running container● Say you get something like:
t/07_plugins.t ........... ok t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .) at t/08_since_epoch.t line 12.BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.# Looks like your test exited with 2 before it could output anything.t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)Failed 11/11 subtests
Access a running container● Say you get something like:
● Q: How do you fix it?
t/07_plugins.t ........... ok t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib /var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .) at t/08_since_epoch.t line 12.BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.# Looks like your test exited with 2 before it could output anything.t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)Failed 11/11 subtests
Access a running container● Exec into the container.● Use CPAN to install the module.● Which leaves you with:
docker exec -it build-perl-5.24.1 bash –-login;
Aside: Container ID● docker ps shows what is running.
Includes the container id & name:
$ docker ps;CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES493d9f285faf jekyll:5000/workhorse/gentoo-stage3-amd64 "/bin/bash -c /var..." 28 minutes ago Up 28 minutes build-perl-5.24.1
# same result:
$ docker exec -it 493d9f285faf 'bash login';‑‑root@493d9f285faf ~ #
$ docker exec -it build-perl-5.4.21 'bash login';‑‑root@493d9f285faf ~ #
Through the looking glass● Inside the container you are root.
Kernel's proc struct has unchanged EUID.
● Limited view of the system.
Only container proc's in “ps” or “top”.
Nice on a crowded server.
Runlevel restrictions are ineffective!
● Run “perl -MCPAN” or “make install”.
Helpful if a non-essential test fails.
Through the Looking Glass● Outside docker you have a specific PID & EUID.● Inside docker the initial process has EUID 0 and PID 1.● EUID 0 is manageable.● PID 1 requires that the smoketest proc reap children.
Otherwise you end up with defunct proc's.
Will eventually eat all proc slots with long-running container.
One fix is Docker::Reaper.
Remember:● Changes to the image go to a separate AUFS layer.● They are discarded.● Fixing files in /etc/ will rescue this build, not the next one.
● Changes to /opt & /var/lib/CPAN are persistent.
TMTOWTDI: minicpan● Outside of docker.
Maintain CPAN cache in /var/lib.
● Inside docker.
Use /var/lib.
Running repeated jobs● Docker can automatically restart running jobs.● docker run -d restart=always ...‑‑
Downside: Lots of proc's started up at once.
Issues with tmpfs on system restarts.● --restart=unless-stopped
Allows stopping volumes on the way down.
Start singly on way back up.
Smoketest lifecyle● Run minicpan once on /var/lib/CPAN.
Global cache of modules.
● Run smoker with O/S images & perly vol's.
● This will only test *one* kernel!● Use VM's running docker for multi-kernel smoke tests.
Zombie apocolypse● Container startup command is process #1.● When it dies the container exits.● It has to reap all children.
Not a big issue for small, single-purpose containers.
Big problem for smoketests.
Fork & Reap
● Parent propagates signals.● Exits with $? from child process.● Loop reaps zombies.
Without this process slots will be exhausted.
If that happens: log on as SU and stop, rm the container.
for(;;) { my $pid = wait;
$pid != -1 or die "Lost child pid ($child)";
$pid != $child and next;
return $? }
Smoking
● Volumes for
perl
CPAN
/var/tmp● Run smoker.
Never exits.
vols=( "-v $perl:/opt:ro" "-v $cpan:$cpan" "-v $tmp:/var/tmp");( $docker volume create \ ... $tmp \ && exec $run \ -t \ --name="$cname" \ ${vols[@]} \ $image \ "/var/lib/CPAN/.cpantesters/docker-smoker") 2>&1 |tee $log;
Propsed change● Smoker processes recents and exit.
Simplifies iterating multiple perl verisons.
● Goal:
Product of images, perl vol's.
Automate bleed test.
Summary● Smoking can be hazardous.● LXC provides lightweight solution.● Mix O/S image, perl volume.
O/S is fixed.
Volume updated to maintain smoker.
● [email protected]:lembark/docker-smoker.git
Mostly shell kwikhaks.
Working to make it cleaner, perly.