Jenkins entwined with deployment and pragmatism
-
Upload
eric-ritchie -
Category
Technology
-
view
671 -
download
3
description
Transcript of Jenkins entwined with deployment and pragmatism
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Jenkins entwined with deployment and pragma>sm
Eric Ritchie DevOps Engineer Novadex +49 (0) 7142 / 788905-‐0 [email protected]
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Eric Ritchie bids you welcome!
20/11/13 2
§ currently working as DevOps Engineer and System Administrator at Novadex GmbH
§ a Zend Framework and PHP ZCE & MySQL DBA
§ a Linux System Administrator with 18 years experience
§ a PHP developer since version 3.0.18 (or so)
§ known to speak at a conference or two
Never heard of him? Aside from looking like this, he is:
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
LeaerMaschine: Custom-‐tailored Communica>on
Personalized data stream Cross-‐Channel Output
Collabora>on
Portal Op>on Campaign management
CRM integra>on
Content management E-‐commerce integra>on
Data protec>on Security
20/11/13 3
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
What I read in a book once…
20/11/13 4
„DevOPs should aim for 10 deployments per day“
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Novadex’s key tool: Jenkins
20/11/13 5
§ Great for Con?nuous Integra?on
§ Can hide complex processes behind a nice buDon
§ Deployment remains one of those complex processes § Requires a lot of scrip>ng/manual work § This risks our deployment goal
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
An example of that scrip>ng
20/11/13 6
<target name="package"> <echo message="...using UPSTREAM_BUILD_NUMBER received from triggering job for packagenaming. UPSTREAM_BUILD_NUMBER is [${UPSTREAM_BUILD_NUMBER}]"/> <!-‐-‐ leaermaschine-‐complete packaging -‐-‐> <!-‐-‐ wipe the dir from possible previous copy opera>on -‐-‐> <delete includeemptydirs="true"> <fileset dir="${package.contents.basedir}" includes="**/*"/> <fileset dir="${package.basedir}/DEBIAN" includes="**/*"/> </delete> <!-‐-‐ prepare debian control file -‐-‐> <copy todir="${package.basedir}/DEBIAN" overwrite="true"> <fileset file="resource/package/test/control"/> <filterset begintoken="[[" endtoken="]]"> <filter token="version" value="${UPSTREAM_BUILD_NUMBER}-‐1"/> <filter token="name" value="leaermaschine-‐complete"/> <filter token="depends" value=""/> <filter token="descrip>on" value="Complete Leaermaschine applica>on"/> </filterset> </copy> <!-‐-‐ copy pos>nstalla>on script -‐-‐> <copy file="${basedir}/resource/package/test/pos>nst_lm" tofile="${package.basedir}/DEBIAN/pos>nst"/> <chmod file="${package.basedir}/DEBIAN/pos>nst" perm="775"/> <copy todir="${package.contents.basedir}"> <fileset dir="${export.basedir}"> <snip> </fileset> </copy> <exec executable="sudo" dir="${working.basedir}" failonerror="true"> <arg line="sudo dpkg -‐b ${package.basedir} ${dist.basedir}/leaermaschine-‐complete_${UPSTREAM_BUILD_NUMBER}-‐1_all.deb"/> </exec> </target>
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Deployment v1.x?
20/11/13 7
§ Applica?on bundled into a .deb package § As shown on the previous slide (yeah, that’s what it did) § Shell scrip>ng available for in deployment manipula>ons
§ SCP’d to target machine, installed via SSH exec § Need to set up SSL cer>ficates to allow password free access
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Problems?
20/11/13 8
§ Who is comfortable with bash shell scrip?ng?
§ What happens when something goes wrong? § Ever tried to roll back a copied .deb package in a hurry?
§ Not really all that cluster friendly § Ok, we have Chef/Puppet if we like overkill
§ Disk cluDer § Each version of the .deb package is cached
§ Need to grant unlimited access to live servers, argh! § Remember that SSL cer>ficate we installed?
§ What if we need to support Red Hat also? § Hypothe>cal in the case of Novadex, but s>ll a valid problem.
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Enter Zend Server deployment (and me)
20/11/13 9
Cluster/Cloud ready, plaxorm agnos>c, PHP scrip>ng and version handling with integrated roll back.
Yes, I used to work for Zend ;)
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Tale of the liale known tool: zs-‐manage
20/11/13 10
§ We all have heard about the Web API, right? § You may be forgiven for thinking it is a new thing
§ We have had command line deployment tools since day one § But nobody knew about them, which is a pity
§ zs-‐manage has many advantages § Can leverage all deployment features § Does not rely on SSL cer>ficates for security § Same commands for local/cluster/cloud deployment
§ But beware, there are some caveats § Pragma>sm to follow...
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Let’s empower Jenkins…
20/11/13 11
<!-‐-‐ retrieve the app.id for the applica>on, specified by app.name. may be empty if app hasn't already deployed -‐-‐> <exec executable="bash" dir="${basedir}" outputproperty="app.id" errorproperty="err" failonerror="true"> <arg value="-‐c" /> <arg value="/usr/local/zend/bin/zs-‐manage app-‐get-‐status -‐N ${zs.api.name} -‐U ${zs.url} -‐K ${zs.api.key} 2>&1 | grep "APPINFO\\s.*${app.name}.*" | awk '{print $2}'" /> </exec> <if> <equals arg1="${app.id}" arg2="" /> <then> <echo message="Couldn't determine app.id for app.name '${app.name}' -‐ preparing ini>al deployment of applica>on ..." /> <!-‐-‐ deploy applica>on via zend server command -‐-‐> <exec executable="/usr/local/zend/bin/zs-‐manage" dir="/tmp" failonerror="true"> <arg line="app-‐deploy -‐u "NDX_ENV=${zs.ndx.env}" -‐c -‐b hap://${app.name}/ -‐a ${app.name} -‐p ${zs.package.des>na>ondir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${zs.url}" /> </exec> </then> <else> <echo message="app.name '${app.name}', app.id is ${app.id} -‐ preparing the update of applica>on ..." /> <!-‐-‐ update applica>on via zend server command -‐-‐> <exec executable="/usr/local/zend/bin/zs-‐manage" dir="/tmp" failonerror="true"> <arg line="app-‐update -‐u "NDX_ENV=${zs.ndx.env}" -‐i ${app.id} -‐p ${zs.package.des>na>ondir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${zs.url}" /> </exec> </else> </if>
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Crea>ng the package to be deployed
20/11/13 12
§ We have a choice § Zend Studio -‐ a manual process § zdpack -‐ Zend standard § zip -‐ needs no introduc>on
§ Novadex have used zip at ?mes § But currently use the following code
<exec executable="/usr/local/zend/bin/zdpack" dir="${working.basedir}" failonerror="true"> <!-‐-‐ NOTE: name of the created package is the name specified in deployment.xml name-‐tag -‐-‐> <arg line="pack -‐-‐src-‐dir=${package.contents.basedir} -‐-‐scripts-‐dir=${basedir}/resource/zs_deploy_new/scripts -‐-‐package-‐descriptor=${working.basedir}/deployment.xml -‐-‐output-‐dir=${working.basedir}" /> </exec> <!-‐-‐ rename created package to reflect version of package contents -‐-‐> <move file="${working.basedir}/${app.name}.zpk" tofile="${working.basedir}/${zs.package.name}" overwrite="true"/> <!-‐-‐ copy the new package from temp loca>on to package directory -‐-‐> <copy file="${working.basedir}/${zs.package.name}" todir="${zs.package.des>na>ondir}" overwrite="true"/>
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Sadly, size does maaer
20/11/13 13
§ You don’t leave your doors unlocked when you leave home § Most admins are equally careful with their servers
§ Web API could be an aDack vector § Which is why Zend take steps to secure it § Web API calls must be signed with, amongst other informa>on, a >mestamp
§ To help prevent replay aDacks, calls must complete within 300 seconds § This security feature can shoot us in the foot § In short, the network speed determines our maximum packet size
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Working around the size issue
20/11/13 14
§ No perfect solu?ons, yet § I am working on Zend ;)
§ But we can apply some pragma?sm § Package can be broken into sub-‐packages § Zend Deployment has a new Library feature, use it § Pull instead of push § Upgrade the pipe § Time travel § Or, emulate the GUI
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Our solu>on
20/11/13 15
<echo message="copying ${zs.package.name} to ${remote.hostname}" /> <scp file="${zs.package.des>na>ondir}/${zs.package.name}" todir="${remote.user}@${remote.hostname}:${remote.dir}" keyfile="${remote.keyfile}" /> <echo message="app.name '${app.name}', app.id is '${app.id}' -‐ preparing the update of applica>on ..." /> <!-‐-‐ execute zs-‐manage locally on host where package should be deployed! -‐-‐> <sshexec host="${remote.hostname}" username="${remote.user}" keyfile="${remote.keyfile}" command="/usr/local/zend/bin/zs-‐manage app-‐update -‐u "NDX_ENV=${zs.ndx.env}" -‐i ${app.id} -‐p ${remote.dir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${remote.webapi.url}" />
GUI emula?on: It works for us
Just remember: SCP/SSH may not be allowed by your opera>ons team. You may need another solu>on ! !
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Memory and lots of it
20/11/13 16
§ Ques?on: Where does all this deployment data go? § Hint: The WebAPI is part of the GUI
§ The GUI allocates memory equivalent to three ?mes the packet size § Since the data is loaded, copied and finally unpacked (don’t ask)
§ Where is the GUI’s memory limit configured? § Not in the GUI, that would spoil the fun ;) § Can you guess?
Answer: The memory_limit variable in: /usr/local/zend/gui/lighapd/etc/php-‐fcgi.ini
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Aaack of the cluster
20/11/13 17
§ Zend deployment can be hard to synchronise § In other words, you cannot guarantee when all nodes will run the
same version of the applica>on
§ An extreme case is the “run once” node § Database migra>ons are usually performed here, so deployment is slower
§ This caused problems with our “In maintenance” page § Time for some extreme pragma>sm…
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
I am a Job Queue fan… so I wrote this
20/11/13 18
// Sanity check if (!class_exists('ZendJobQueue')){ logger("Job Queue seems to be disabled!"); exit(1); } if (getenv("ZS_RUN_ONCE_NODE") == 1) { $name = DEPLOYMENT_JOB_BASENAME.$appCurrentVersion; $queue = new ZendJobQueue(); // Before scheduling our job, just make sure that it has not been scheduled before $total = null; $jobArray = $queue-‐>getJobsList(array('name' => $name), $total); if ($total != 0) { foreach($jobArray AS $job) { $queue-‐>removeJob($job['id']); } } // Schedule the job script to run a long >me in the future (since we don't really need it to run) $runAt = date('Y-‐m-‐d H:i:s', strto>me('+7 days')); $semaphoreJobId = $queue-‐>createHapJob('hap://www.google.de/search?q=dummy', array(), array('name' => $name, 'schedule_?me' => $runAt)); if (!$semaphoreJobId) { logger("Semphore job was not accepted in the queue!"); exit(1); } }
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
…and this at the end of the script
20/11/13 19
if (getenv("ZS_RUN_ONCE_NODE") == 1) { // We are the RUN ONCE script so clear the semaphore $queue-‐>removeJob($semaphoreJobId); } else { // We are not the RUN ONCE script so wait for it to clear the semaphore $queue = new ZendJobQueue(); $total = 1; $status = ZendJobQueue::JOB_STATUS_PENDING | ZendJobQueue::JOB_STATUS_SCHEDULED; while ($total != 0) { $jobArray = $queue-‐>getJobsList(array('name' => $name, 'status' => $status), $total); sleep(1); } }
We now plan to use an Nginx, that spontaneously appeared in our architecture, to set our maintenance page. Nevertheless, it was s>ll a cool idea ;)
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
Finding out what is happening
20/11/13 20
§ So far so good, but we s?ll don’t have true integra?on § Jenkins has no liale* visibility over the deployment process
§ We can log the output of our deployment scripts § But these are not available to Jenkins § We can make them available to the GUI, at least, by adding the paths to the
GUI_AVAILABLE_LOGS table in the Zend Server database § More convenient, but we s>ll need to go check the GUI a�er each deployment
* Last minute slide hack
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
WebAPI to the rescue… again
20/11/13 21
§ Features added to the WebAPI allow registered logs to be remotely read § logsReadLines § logsGetLogfile
§ We can finally integrate log output into our Jenkins reports
§ Just a few problems § Finding the “run once” node § These methods are not implemented in zs-‐manage (yet) § Wri>ng the code (I haven’t had the >me yet)
Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.
www.novadex.com
www.leaermaschine.com
Please scan this …and tell me where it goes
Honestly, marke>ng wouldn’t tell me!
Thank you very much for your aaen>on
20/11/13 22
Eric Ritchie DevOps Engineer hDps://joind.in/9926
+49 (0) 7142 / 788905-‐0 [email protected]
Novadex GmbH Stuagarter Straße 59 74321 Bie>gheim-‐Bissingen Germany