Smolder And Buildbot

Michael Peters, Plus Three LP (http://plusthree.com)

What is Smolder?What is Smolder?

● Will it automate my build/test cycle?– No.

● Will it run my test suite?– No.

● Will it checkout my code from svn, run my build and then run my test suite?– Hello!

Smolder takes the information about a test run and makes it pretty.

http://plusthree.com/


History of SmolderHistory of Smolder● Plus Three, LP● Long running test suites for multiple projects

– 5,460 tests in 28 minutes (510K HTML report)– 18,000+ tests in 50 minutes

● Stole lots of ideas from the Pugs smoke server.● Added lot

– Comments, tags, invalidation– Pretty graphs– Multiple projects (public and private)– Atom data feeds



TAPTAP

● Test Anything Protocol● Created by Test::Building, Test::More, etc● Consumed by Test::Harness, prove, etc

1..5ok 1 – foo is fineok 2 – testing bar valuenot ok 3 – The object isa Frobnicate# this is some useful# comment thingnot ok 4 # SKIP Foo::Bar is not installednot ok 5 # TODO future feature



TAPTAP

● How do you create it?use Test::More (tests => 5)ok($foo, 'foo is fine');is($bar, 'right', 'testing bar value');isa_ok($bar, 'Frobnicate');diag(“this is some useful\ncomment thing”);SKIP: { skip('Foo::Bar is not installed' 1); ok($foo>bar, 'has a bar');}TODO: { local $TODO = 'future feature'; ok($foo>feature, 'some feature');}



TAPTAP

● Started in Perl-Land● We have since created colonies

– C– PHP– Python– Java– JavaScript– Parrot– Postgres– .... more



TAP ArchivesTAP Archives

● A single file (.tar or .tar.gz)● Multiple recorded TAP streams (files)● Some meta information (META.yml)

– Stream (test) order– Start/Stop times– Future expansion (server info, log files, etc)

● Easy to create with TAP::Harness::Archive– prove -a



Continuous Integration CycleContinuous Integration Cycle

● Automated builds– Simple bash script#!/bin/bashif svn update | grep 'Updated to' then make build && make testelse echo "No updates"fi

● Get fancier– SmokeRunner::Multi– Custom harness written with Test::Harness– BuildBot



BuildbotBuildbot

● Most projects don't have multi-platform, multi-architecture needs– Anything written in pure Perl– Most $work applications

● Some projects do– Languages (Perl, Python)– VMs (Parrot)– System Tools (SCM, etc)



What is Buildbot?What is Buildbot?

● Will it automate my build/test cycle?– Yes.

● Will it run my test suite?– Yes.

● Will it checkout my code from svn, run my build and then run my test suite?– Absolutely!



Other Buildbot SpiffynessOther Buildbot Spiffyness

● Master configuration, multiple slaves do the work– Lots of architectures and platforms

● Arbitrary build commands– Not a language specific system– Doesn't care if you use make, Build.PL, ant, maven

● Portable– It's Python and works in lots of places despite all

the whitespace● Very configurable



So Why Do I Need Smolder?So Why Do I Need Smolder?

● Buildbot gives just a single pass/fail for each run

● All other details are just dumped to the screen● No good way to organize your test runs● ....





● It's UGLY!





● Oooh-la-la!




So Why Do I Need Smolder?So Why Do I Need Smolder?So Why Do I Need Smolder?So Why Do I Need Smolder?

Buildbot Smolder=>=>



So Why Do I Need Smolder?So Why Do I Need Smolder?So Why Do I Need Smolder?So Why Do I Need Smolder?

Buildbot

=>=>

Smolder=>=>




● Information is more organized– Access to the raw TAP Archives– Search interface and Graphs

● Let's you answer questions like– When was the last time our test suite passed on

platform X?– Who broke the test suite?– How many tests have we added for this release?– How did checkin X affect the speed of the test

suite?



Buildbot Talking to SmolderBuildbot Talking to Smolder

● Remember Buildbot allows arbitrary build commands

● So your project just needs a command that can run the suite, create a TAP archive and send it to a Smolder server– prove + smolder_smoke_signal– make smoke_tests

● Buildbot doesn't know that it's being cheated on!



Scary Live Demos!Scary Live Demos!● Parrot● Smolder



Smolder's FutureSmolder's Future

● Test Coverage● Tracking individual test assertions● Better visual display of diagnostics (new TAP

extensions)● More and better ways to search old reports


Smolder And Buildbot

Technology

Transcript of Smolder And Buildbot