The Challenge of Continuously Delivering

Julian Fitzell, CincomESUG, EdinburghAugust 24, 2011

Image © Petr Vins: http://www.sxc.hu/photo/1054507

ChannelStream

Document Output Management

Control

Enrichment

OMR

Marketing

Barcode Datamatrix

Output

Why?

• Postal contract savings

• Saves licking envelopes

• Personalization and cross-marketing

• Avoid IT/supplier for customization

• Enhance legacy output

ChallengesImage © Tim Regan: http://www.flickr.com/photos/dumbledad/3225255407/

Slow development

Manual process

Test environment not

easily reproducible

External dependencies

fragile

Test data unversioned

Configuration files unversioned

False positives

Unclear separation between

unit/functionalTests not always run

Tests not always updated

Building images slow +

error-prone

Tests slow/hard to run

Infrequent integration

One developer integrating

Reduced confidence from automated tests

STRESS!

+Frequent releases

Integration is risky/hard

Difficulty tracking versions

So...

• Make building images easier

• Make running tests easier

• Make integration easier

Jez Humble, http://continuousdelivery.com/2010/08/continuous-delivery-vs-continuous-deployment/

con·tin·u·ous de·liv·er·y• putting the release schedule in the

hands of the business• ensuring your software is always

production ready throughout its entire lifecycle

• making any build potentially releasable to users at the touch of a button

Goals

• Reduce cycle time between an idea and its availability to users

• Deliver high-quality software reliably and efficiently

Frequent, automated releases:

• Decrease stress• Eliminate errors and improve

auditability• Make feedback more immediate• Protect against reliance on individual

expertise• Reduce wasted time on repetitive tasks

Make releases boring

• Automate (almost) everything

• Version everything

• Deploy everywhere the same way

• If it hurts, do it more often

• Everyone is responsible for delivery

Development+

Integration

Problems

• Contention over “head”

• One person doing most of the integration

• Delayed integration => harder integration

• Poor visibility of changes

• Hard to predict/analyze failure

Integration

Issue tracker + emailIntegration

Task/merge toolsIntegration

Integrate often

• Hide functionality (feature toggles)

• Incremental, releasable changes

• “Branch by abstraction” for large-scale changes

• Use components to decouple parts

Integration

Image © Lars Sundstrom: http://www.sxc.hu/photo/1082530

Remaining Issues

• Restructurings can still be tricky when branches are outstanding

• “Integration token” still just informal conversation

Integration

Building

Problems

• Manual builds on developer machines

• Wasted developer time

• “Here, take my image”

• Taking shortcuts

• “Works on my machine”

Building

Helper scripts + docsBuilding

Fully automated scripts

Builder

-filein foo-wrapper.ws

foo.ws

foo-wrapper.ws

Target

Building

Builds run (at least)dailyBuilding

Archive built imagesBuilding

Image © 2009 by Urbancode

Run automatically on ALL code changes

Avoid versioning split between SVN + Store

Rationalize build/version numbering

!"#$%&'()*+,%&-+.+/011+23&45(+67)%3)+8359)+:00+.+;4)7)4$%(*+<=5'+11>>?+.+@A+/>:-B?B-C000++.+DDD-3"#$%&'()-&'E+.+F+/00C+#G+!"#$%&'()+

&'EE'%H4$&)+)%'3I=+9=$9+"'445%I+59+'39+%'+4'%I)"+")J35")K+9=)+)LL'"9+9'+J3$45LG+59+$K+!"#$"%-+M$K5&+;,+5K+%'D+)$KG+)%'3I=+9'+#)+$%+!"&%'(%)*$&%+4)7)4+$&95759G-+

!"#$%#&'()*+,$-.#&'()/+01#&')2&%)3+,-41#&')N59=+'3"+E'()4+'L+E$93"59G+5%+H4$&)+D)+$")+")$(G+9'+$((")KK+'3"+9'H5&K+5%+2%9)"H"5K)+;'%95%3'3K+,%9)I"$95'%A+M354(5%I*+O)H4'G5%I*+P)K95%I+$%(+Q)H'"95%I-+P=)K)+9'H5&K+&'7)"+9=)+)KK)%95$4+)4)E)%9K+'L+9=)+)%(R9'R)%(+M354(+S5L)&G&4)*+9=)+T'3"%)G+L"'E+K'3"&)+&'()+9'+K'L9D$")+5%+H"'(3&95'%-+

!"#$%#&')U$"()%R7$"5)9G*+()7)4'H)"R&)%9"5&+;,+5K+$#'39+L$K9+L))(#$&V+L"'E+#354(5%I+9=)+K'L9D$")-+N=)%+;,+E))9K+)%9)"H"5K)+&'%&)"%K*+#354(+E$%$I)E)%9*+()H)%()%&5)K+#)9D))%+H"'T)&9K*+$%(+$+&'%9"'4+'7)"+9=)+#354(+H"'&)KK+#)&'E)+&"595&$4+)4)E)%9K-+

W'K9+%)D+H"'T)&9K+K9$"9+'39+D59=+#354(K+H)"L'"E)(+'%+()7)4'H)"+E$&=5%)K+$%(+D59='39+$+K9$%($"(+H"'&)KK-+<%)+()7)4'H)"+#354(K+5%+9=)5"+,O2+D=54)+$%'9=)"+3K)K+$+#354(+K&"5H9-+P=)+4)$K9+E$93")+9)$EK+3K)+9=)K)+#354(K+L'"+9)K95%I+'"+)7)%+L'"+")4)$K)K+9'+H"'(3&95'%-+P=)+H"'#4)EK+")K3495%I+L"'E+9=5K+4$&V+'L+&'%9"'4+$")+J35&V4G+$HH$")%9+9'+E'K9+9)$EK*+$%(+9=3K+K9$"9+$+K)$"&=+L'"+#)99)"+$49)"%$957)K-+

+

+

P=)+L5"K9+K9)HK+L'"+E$93"5%I+9=)+#354(+$")+K9$%($"(5X5%I+9=)+#354(+H"'&)KK+$%(+H)"L'"E5%I+'LL5&5$4+#354(K+R()7)4'H)"+E$&=5%)+L'"+9=)+#354(+E)$%K+&=$%I)K+5%+'%)+

5K+")9"5)7)(+L"'E+K'3"&)+&'%9"'4+9'+3K)+5%+#354(K-+P=5K+E$G+#)+9=)+4$9)K9+L"'E+9=)+95H+'L+$+#"$%&=*+'"+4$#)4)(+K'3"&)+&'()*+'"+K'E)9=5%I+)4K)-+P=)+5EH'"9$%9+H$"9+5K+9=$9+9=)+&='K)%+&'%7)%95'%+#)+$HH45)(+&'%K5K9)%94G-+N59=+9=)K)+H"$&95&)K+5%+H4$&)*+9=)+9)$E+=$K+")$&=)(+$%+!"&'+),-&+'.+4)7)4+'L+#354(+&'EH)9)%&)-+

6%G+9)$E+3K5%I+K9$%($"(+;'%95%3'3K+,%9)I"$95'%+D544+9$V)+9=5K+$+K9)H+L3"9=)"+$%(+$39'E$9)+)Y)&395'%+'L+9=)+#354(+K9)HK-+P=)+#354(+K)"7)"+D544+(5&9$9)+E$&=5%)K*+K'3"&)R&'()+"34)K*+$%(+"3%+9='K)+#354(+K9)HK*+

Building

http://www.urbancode.com/html/resources/white-papers/

Testing

Problems• Tests took up to 16 hours to run

• Test environment hard to create

• Failing tests sometimes pass when re-run

• Not everyone could run tests or be sure of results

• High cost of test maintenance

• Sometimes quicker to test manually than fix the test environment

Testing

Done

• Optimize tests (now run in ~ 2 hours)

• Separate unit/regression from functional/integration tests

• Store test data in a central location

Testing

Run tests automaticallyTesting

Image © 2009 by Urbancode

Run regression and smoke tests as part of build and notify of errors by email

Run functional tests after build

Version control test data

Automate creation of testing environment!"#$%&'()*+,%&-+.+/011+23&45(+67)%3)+8359)+:00+.+;4)7)4$%(*+<=5'+11>>?+.+@A+/>:-B?B-C000++.+DDD-3"#$%&'()-&'E+.+F+/00C+#G+!"#$%&'()+

H)$EI+$9+9=)+!"#$"%+4)7)4+J"$&95&)+&'%95%3'3I+()J4'GE)%9*+$39'E$95&$44G+()J4'G5%K+9'+J"'(3&95'%+D59='39+$%G+E$%3$4+5%9)"7)%95'%A+6+#354(+5I+&")$9)(+9=)%+$39'E$95&$44G+()J4'G)(+9="'3K=+9=)+9)I95%K+)%75"'%E)%9I-+6I+59+E'7)I+9="'3K=+9=)+J5J)45%)*+9=)+#354(+5I+I3#L)&9)(+9'+$39'E$9)(+9)I95%K+$9+)7)"G+I9$K)*+$%(+'%&)+59+=$I+J$II)(+$44+9=)+$JJ"'J"5$9)+M3$459G+K$9)I*+59+5I+5EE)(5$9)4G+()J4'G)(+9'+J"'(3&95'%-+8'E)+I)"5'3I+('9N&'E+$JJ45&$95'%I+")4)$I)+&=$%K)I+D59=5%+$%+='3"+'O+59+")$&=5%K+I'3"&)+&'%9"'4-+<#75'3I4G*+9=)+$39'E$9)(+9)I95%K+E3I9+#)+7)"G+E$93")*+$4'%K+D59=+"'44N#$&P+$%(+$JJ45&$95'%+E'%59'"5%K-++Q39+5%+$+O$I9NJ$&)(+)%75"'%E)%9*+)R9")E)4G+O$I9+()J4'GE)%9I+'O+%)D+O3%&95'%$459G+$")+$+P)G+&'EJ)95957)+$(7$%9$K)*+$%(+E595K$9)+9=)+"5IP+'O+#5KN#$%K+O3%&95'%$459G+&=$%K)I-+

!"#$%&'(;'%95%3'3I+,%9)K"$95'%+=$I+4'%K+#))%+$II'&5$9)(+D59=+I'E)+4)7)4+'O+$39'E$9)(+9)I95%K-+H=5I+5I+9"3)+#'9=+5%+9=)+I)E5%$4+$"95&4)+#G+S$"95%+T'D4)"+'"+5%+9=)+'4()"+J"$&95&)+()I&"5#)(+#G+89)7)+S&;'%%)44+'O+9=)+U$54G+Q354(+$%(+8E'P)+H)I9-+V59=5%+9=)+I&'J)+'O+2%9)"J"5I)+;'%95%3'3I+,%9)K"$95'%*+E3495J4)+9GJ)I+'O+$39'E$9)(+9)I9I+$%(+E$%3$4+9)I95%K+$")+&'%I5()")(-++

U)IJ59)+9=5I*+E$%G+9)$EI+$")+D)$P+$9+9)I95%K-+H=)G+J)"O'"E+$+#354(*+J39+59+9="'3K=+I'E)+#$I5&+)R)"&5I)I+E$%3$44G+$%(+9=)%+")4)$I)-+H=)+I$E)+J$"9I+'O+9=)+$JJ45&$95'%+#")$P+")J)$9)(4G+$%(+%)D+O3%&95'%$459G+5I+45K=94G+9)I9)(-+6I+9)$EI+E$93")+5%+9)I95%K+9=)G+()9)&9+J"'#4)EI+I''%)"*+5%&")$I5%K+#'9=+J"'(3&95759G+$%(+&'%O5()%&)-+

+

+

S'I9+9)$EI+=$7)+I'E)+O'"E+'O+$39'E$9)(+9)I95%K+5%+J4$&)-+@)"=$JI+$+IE$44+3%59+9)I9+I359)+'"+I'E)+I&"5J9)(+9)I9I+)%I3")+9=$9+9=)+#$I5&+O3%&95'%$459G+'O+9=)+$JJ45&$95'%+D'"PI-+H=)I)+#$I5&+$39'E$9)(+")K")II5'%+9)I9I+J"'75()+)$IG*+)$"4G+()9)&95'%+'O+O3%($E)%9$4+J"'#4)EI-+H)$EI+$9+9=)+!"&'()*+&(',+I9$K)I+'O+2%9)"J"5I)+;'%95%3'3I+,%9)K"$95'%+$")+K)%)"$44G+L3I9+#)&'E5%K+$&&45E$9)(+9'+$39'E$9)(+9)I95%K-+

H'+")$&=+9=)+-(./+%+4)7)4+'O+E$93"59G*+9=)")+I='34(+#)+$+I)9+'O+O$I9+9)I9I+9=$9+$")+"3%+D59=+)7)"G+#354(-+H=)I)+9)I9I+J"'75()+9=)+9)$E+&'%O5()%&)+9=$9+9=)+$JJ45&$95'%+D544+#$I5&$44G+D'"P+75"93$44G+$44+'O+9=)+95E)-+

Image © 2009 by Urbancode

Testing

Releasing

Problems

• One person is:

• best person for build

• best person for integration

(so, one can delay the other)

• Forced to branch when close to release

Releasing

Done

• Helper scripts

Releasing

Image © 2009 by Urbancode

Fully automate runtime image build

Automate installer creation

Automate installation into test environments

Changelog generation

!"#$%&'()*+,%&-+.+/011+23&45(+67)%3)+8359)+:00+.+;4)7)4$%(*+<=5'+11>>?+.+@A+/>:-B?B-C000++.+DDD-3"#$%&'()-&'E+.+F+/00C+#G+!"#$%&'()+

H"'75(5%I+45994)+$JJ3"$%&)+9=$9+$+J3&&)JJK34+()H4'GE)%9+5%+'%)+)%75"'%E)%9+5%(5&$9)J+9=$9+9=)+()H4'GE)%9+D544+#)+J3&&)JJK34+5%+9=)+%)L9+)%75"'%E)%9-+

+

+

6+9)$E+9=$9+=$J+E'7)(+$D$G+K"'E+$+K344G+E$%3$4+H"'&)JJ+9'+3J5%I+$+%3E#)"+'K+=)4H)"+J&"5H9J*+'"+$+K344G+J&"5H9)(+H"'&)JJ*+=$J+E$()+5EH'"9$%9+5EH"'7)E)%9J-+6&"'JJ+9=)+5%(3J9"G*+E'J9+9)$EJ+=$7)+J'E)+=)4H)"+J&"5H9J+#39+K$44+J='"9+'K+K344G+J&"5H9)(+()H4'GE)%9J+ +)JH)&5$44G+9'+&'%9"'44)(+)%75"'%E)%9J-+

!"#$%&$'()#$+9)$EJ+$")+I''(+$9+()H4'G5%I+K'"+9)J95%I+H3"H'J)J-+M=)G+D544+=$7)+H39+9=)5"+K344G+J&"5H9)(+()H4'GE)%9J+#)=5%(+9''45%I+9=$9+$44'DJ+K'"+H3J=N#399'%+()H4'GE)%9J+9'+J'E)+'"+$44+'K+9=)5"+9)J9+)%75"'%E)%9J-+M=5J+9$O)J+$+I")$9+()$4+'K+4'$(+'KK+'K+9=)+()H4'GE)%9+)%I5%))"J+D=54)+")(3&5%I+9=)+('D%95E)+'K+9)J9+9)$EJ+D$595%I+K'"+$+()H4'GE)%9-+P3J9+$J+&'%95%3'3J+#354(J+$")+$+&=$"$&9)"5J95&+'K+!"#$%&$'()#$+#354(+9)$EJ*+$39'E$95&+()H4'GE)%9+9'+9=)+K5"J9+9)J9+)%75"'%E)%9+5J+$+=$44E$"O+'K+!"#$%&$'()#$+E$93"59G+5%+()H4'GE)%9-++Q)H)%(5%I+'%+9)$E+(G%$E5&J*+9=5J+J='34(+=$HH)%+$9+9=)+)%(+'K+$%G+J3&&)JJK34+&'%95%3'3J+#354(+'"+$9+")I34$"+5%9)"7$4J+9="'3I=+9=)+($G+D=)%+9)J9)"J+$")+3%45O)4G+9'+#)+5%9)""3H9)(-+

M=)+4$J9+E$"O)"+'K+$%+!"#$%&$'()#$+9)$E+5J+9=$9+9=)G+=$7)+E$()+I''(+H"'I")JJ+5%+J9$%($"(5R5%I+9=)5"+()H4'GE)%9J+$&"'JJ+)%75"'%E)%9J-+M=)")+E$G+J9544+#)+J'E)+)%75"'%E)%9$4+7$"5$95'%*+'"+9D'+9GH)J+'K+

4$9)"+()H4'GE)%9J-+S)$&=5%I+9=5J+4)7)4+'K+E$93"59G+5J+$+I''(+9$"I)9+K'"+E$%G+9)$EJ-+

*'+)",$'+9)$EJ+93"%+9=)5"+$99)%95'%+9'+&'%9"'44)(+'"+H"'(3&95'%+)%75"'%E)%9J-+Q)H4'GE)%9J+9'+H"'(3&95'%+T")4)$J)JU+$")+H3J=N#399'%+$%(+J3&&)JJK34+H"'(3&95'%+")4)$J)J+$39'E$95&$44G+9"5II)"+E$9&=5%I+")4)$J)J+9'+(5J$J9)"+")&'7)"G-+6%G+9)$E+9=$9+()H4'GJ+9'+H"'(3&95'%+'%+5%9)"%$4+5%K"$J9"3&93")+J='34(+&'%J5()"+9$"I)95%I+9=)+*'+)",$'+4)7)4A+=$75%I+$+&'%J5J9)%9+()H4'GE)%9+H"'&)JJ+$&"'JJ+$44+)%75"'%E)%9J+("$E$95&$44G+")(3&)J+9=)+45O)45=''(+'K+4$J9NE5%39)+K$543")J+D=)%+E'75%I+9'+H"'(3&95'%-+*'+)",$'+9)$EJ+$")+$4J'+&=$"$&9)"5R)(+#G+K344G+$39'E$9)(+()H4'GE)%9J+9'+J'E)+9)J95%I+)%75"'%E)%9J+#$J)(+H$JJ5%I+V3$459G+I$9)J-+W'"+)L$EH4)*+$%+$HH"'7$4+K"'E+$+9)J9+E$%$I)"+J9$I)J+$+#354(+9'+#)+$39'E$95&$44G+5%J9$44)(+5%+9=)+J9")JJN9)J9+)%75"'%E)%9-+

Releasing

CINCOM and the Quadrant Logo are registered trademarks of Cincom Systems, Inc.All other trademarks belong to their respective companies.

© 2011 Cincom Systems, Inc.All rights reserved

Julian Fitzelljfitzell@cincom.com

Twitter: @jfitzell