Michael Peters, Plus Three LP (http://plusthree.com)

What is Smolder?What is Smolder?

● Will it automate my build/test cycle?– No.

● Will it run my test suite?– No.

● Will it checkout my code from svn, run my build and then run my test suite?– Hello!

Smolder takes the information about a test run and makes it pretty.

Michael Peters, Plus Three LP (http://plusthree.com)

History of SmolderHistory of Smolder● Plus Three, LP● Long running test suites for multiple projects

– 5,460 tests in 28 minutes (510K HTML report)– 18,000+ tests in 50 minutes

● Stole lots of ideas from the Pugs smoke server.● Added lot

– Comments, tags, invalidation– Pretty graphs– Multiple projects (public and private)– Atom data feeds

Michael Peters, Plus Three LP (http://plusthree.com)

TAPTAP

● Test Anything Protocol● Created by Test::Building, Test::More, etc● Consumed by Test::Harness, prove, etc

1..5ok 1 – foo is fineok 2 – testing bar valuenot ok 3 – The object isa Frobnicate# this is some useful# comment thingnot ok 4 # SKIP Foo::Bar is not installednot ok 5 # TODO future feature

Michael Peters, Plus Three LP (http://plusthree.com)

TAPTAP

● How do you create it?use Test::More (tests => 5)ok($foo, 'foo is fine');is($bar, 'right', 'testing bar value');isa_ok($bar, 'Frobnicate');diag(“this is some useful\ncomment thing”);SKIP: {  skip('Foo::Bar is not installed' 1);  ok($foo­>bar, 'has a bar');}TODO: {  local $TODO = 'future feature';  ok($foo­>feature, 'some feature');}

Michael Peters, Plus Three LP (http://plusthree.com)

TAPTAP

● Started in Perl-Land● We have since created colonies

– C– PHP– Python– Java– JavaScript– Parrot– Postgres– .... more

Michael Peters, Plus Three LP (http://plusthree.com)

TAP ArchivesTAP Archives

● A single file (.tar or .tar.gz)● Multiple recorded TAP streams (files)● Some meta information (META.yml)

– Stream (test) order– Start/Stop times– Future expansion (server info, log files, etc)

● Easy to create with TAP::Harness::Archive– prove -a

Michael Peters, Plus Three LP (http://plusthree.com)

Continuous Integration CycleContinuous Integration Cycle

● Automated builds– Simple bash script#!/bin/bashif svn update | grep 'Updated to' then make build && make testelse echo "No updates"fi

● Get fancier– SmokeRunner::Multi– Custom harness written with Test::Harness– BuildBot

Michael Peters, Plus Three LP (http://plusthree.com)

BuildbotBuildbot

● Most projects don't have multi-platform, multi-architecture needs– Anything written in pure Perl– Most $work applications

● Some projects do– Languages (Perl, Python)– VMs (Parrot)– System Tools (SCM, etc)

Michael Peters, Plus Three LP (http://plusthree.com)

What is Buildbot?What is Buildbot?

● Will it automate my build/test cycle?– Yes.

● Will it run my test suite?– Yes.

● Will it checkout my code from svn, run my build and then run my test suite?– Absolutely!

Michael Peters, Plus Three LP (http://plusthree.com)

Other Buildbot SpiffynessOther Buildbot Spiffyness

● Master configuration, multiple slaves do the work– Lots of architectures and platforms

● Arbitrary build commands– Not a language specific system– Doesn't care if you use make, Build.PL, ant, maven

● Portable– It's Python and works in lots of places despite all

the whitespace● Very configurable

Michael Peters, Plus Three LP (http://plusthree.com)

So Why Do I Need Smolder?So Why Do I Need Smolder?

● Buildbot gives just a single pass/fail for each run

● All other details are just dumped to the screen● No good way to organize your test runs● ....

So Why Do I Need Smolder?So Why Do I Need Smolder?

Michael Peters, Plus Three LP (http://plusthree.com)

So Why Do I Need Smolder?So Why Do I Need Smolder?

● It's UGLY!

So Why Do I Need Smolder?So Why Do I Need Smolder?

Michael Peters, Plus Three LP (http://plusthree.com)

So Why Do I Need Smolder?So Why Do I Need Smolder?

● Oooh-la-la!

So Why Do I Need Smolder?So Why Do I Need Smolder?

Michael Peters, Plus Three LP (http://plusthree.com)

So Why Do I Need Smolder?So Why Do I Need Smolder?So Why Do I Need Smolder?So Why Do I Need Smolder?

Buildbot Smolder=>=>

Michael Peters, Plus Three LP (http://plusthree.com)

So Why Do I Need Smolder?So Why Do I Need Smolder?So Why Do I Need Smolder?So Why Do I Need Smolder?

Buildbot

=>=>

Smolder=>=>

Michael Peters, Plus Three LP (http://plusthree.com)

So Why Do I Need Smolder?So Why Do I Need Smolder?

● Information is more organized– Access to the raw TAP Archives– Search interface and Graphs

● Let's you answer questions like– When was the last time our test suite passed on

platform X?– Who broke the test suite?– How many tests have we added for this release?– How did checkin X affect the speed of the test

suite?

Michael Peters, Plus Three LP (http://plusthree.com)

Buildbot Talking to SmolderBuildbot Talking to Smolder

● Remember Buildbot allows arbitrary build commands

● So your project just needs a command that can run the suite, create a TAP archive and send it to a Smolder server– prove + smolder_smoke_signal– make smoke_tests

● Buildbot doesn't know that it's being cheated on!

Michael Peters, Plus Three LP (http://plusthree.com)

Scary Live Demos!Scary Live Demos!● Parrot● Smolder

Michael Peters, Plus Three LP (http://plusthree.com)

Smolder's FutureSmolder's Future

● Test Coverage● Tracking individual test assertions● Better visual display of diagnostics (new TAP

extensions)● More and better ways to search old reports