1 Page

A framework to manage multiple

releases using Agile Scrum and Kanban

Venkatraman L – Head of Project Management, InMobi

3 Page

Contents 1. Abstract ......................................................................................................................................... 4

2. How to read this paper ................................................................................................................ 4

3. Terminologies to get started ....................................................................................................... 4

4. How it all began ............................................................................................................................ 5

5. The Framework ............................................................................................................................ 6

Step 1: Gap Analysis ................................................................................................................... 6

Step 2: Plan the Phase 1 ............................................................................................................ 7

The Execution Framework - Sprints, Stand-ups... .................................................................. 9

The Simple 3-Step Framework (push & pull) ......................................................................... 10

The Pull – Kanban ...................................................................................................................... 10

Arriving at Release Dates ......................................................................................................... 11

Managing Release-Level Visibility – The Key to Success! .................................................. 11

The Leap into Phase 2 .............................................................................................................. 11

6. Key Benefits of the framework ................................................................................................. 12

7. Key Challenges Overcome ....................................................................................................... 13

8. Conclusion .................................................................................................................................. 13

9. References .................................................................................................................................. 13

10. Author’s Profile: ........................................................................................................................ 14

4 Page

1. Abstract

The key to any startup is speed against any process / framework, to begin with. However, as time

progresses and the startup suddenly scales leaps and bounds in terms of people and business, the

need for a structured approach starts to emerge. The situation can be rightly called “The Controlled

Chaos”. The paper describes a startup with multiple stack teams (teams are grouped by architectural

components forming a clean workflow) aiming to deliver multiple simultaneous releases at a

sustainable pace. Almost a significant number of releases cut across multiple stack teams and that

posed a huge challenge as teams were used to working in their own styles and priorities. The

framework that was designed to manage these releases focused on release dependency matrix, inter

and intra team dependencies, release planning, sprint planning, without compromising on the velocity

of the teams. One thing to call out was that, there was no luxury of picking a “pilot” team as in other

Agile rollouts as there was no single team who could be picked. In short, the mandate was to fix the

engine mid-flight. Release lifecycle was just broken down into steps of 3 to make the entire process

swift and lean. A mix of push (to stack team backlogs) and a pull (Kanban mode during

execution) adds flavor to the whole framework.

2. How to read this paper

The paper begins with outlining few terminologies that are specific to the framework. The next section

details the scenario that needed a robust framework to manage releases (30+ in parallel). The section

post that details how the framework evolved and where it stands now. The final section talks about

key challenges and the volume of releases that the teams are handling.

3. Terminologies to get started

All the terminologies described below, though they are generic, have some nuances when it comes to

the context of this paper.

Core Team

Core team is a key team during the release ideation and discovery process. This team is responsible

to define the business specs for each release, elaborate the architectural requirements, and suggest

any changes to the designs, identify and call out the technical dependencies between the stack

teams and also with the platform teams. The participants are the Product Owner (biz specs),

Architect (technical dependencies) and UX (as needed by the release)

Scrum Team

By Scrum team, we mean the Product Owner (PO), Scrum Master, Team (Development, Technical

Operations and Test Engineering) who would spend substantial amount of time in the 2-week sprints.

This team (w/ the support of the architect) is responsible for building the product right.

5 Page

Role of Architects in the Scrum Team

Architects would also be an integral part of the planning meetings and will be available during the

entire sprint for answering queries and modifying or making changes to the architecture and design.

Architects would be part of multiple projects (shared across the projects) but their time availability is

critical to the success of each of the teams they interface with.

Portfolio Items (PI) & Themes

Each Portfolio Item is equivalent of a release in the system. The reason to move into a portfolio view

of releases is due to the sheer volume of releases that were in execution at any given point in time. A

group of PI constitutes the release backlog of the organization. These are prioritized every fortnight.

sThe entire set of releases was aligned to few Business Themes (Marketplace, Sustenance, System

health, Improve Operational Capabilities, to name a few) so that the executive team can take an

informed decision when the releases are approved.

Release v/s Deployment

Release is a time-boxed (feature rich) deliverable that works end-to-end in the system. Each release

derives out of a business need. Deployment is termed as a push to production by individual stack

teams. Multiple deployments put together constitute a release in the system.

Backlog Grooming

The process of (but not limited to) elaborating, clarifying, adding acceptance criteria and planning

long-range solutions could be called as the Backlog Grooming. This is ideally done during the stage

when the team starts looking at the Release Backlog and tries to provide feedback to the PO by

asking the right questions and identifying if the product envisioned is feasible.

Sprint / Iteration

Sprint is a time-boxed iteration (recommended is 2 weeks but could potentially be of any longer

duration too). During this period, the team works uninterrupted (by anybody outside the Scrum) with

the help of the Scrum Master who ensures that the team focuses on what has been committed to (the

sprint backlog) be delivered. Each sprint is expected to deliver a potentially shippable product

increment. However, it is the decision of the PO that decides if the product is shippable.

4. How it all began

Planning & Execution Status

During Q2 of 2011, though there was enough momentum in the system (small teams) and so was a

lot of churn in the backlog due to business priorities. Many stack teams were not having a focused

planning and execution cycles. They were either going by the flavor of the month (driven by

competition, state of union) or the sudden priorities that crept up. The iteration and deployment

rhythm was broken and they were just executing on a daily / weekly priorities. There were instances

where the teams dropped what they were working and moved on to an item with higher priority as

business was suffering. Amidst all these, there were some teams who were diligently doing weekly

6 Page

releases. All teams however wanted a clear priority to execute so that all teams can align and deliver

timely releases. In short, the X (read 20+) number of releases that were signed up for the quarter was

heading in multiple directions.

What were the problems?

1. Clarity on the teams contributed to the release. This was a key gap b/w the PO and

Engineering due to handoffs, discussions that were not followed up etc.

2. Flesh the details and own the technical dependencies (between stack teams) – Which team

owned it?

3. Status of each of the stack team deliverable and in turn the entire release?

4. How to breakdown of the business ask (of the release) into smaller requirements that each

stack team can independently work on?

5. What will the backlog for the stack teams to pick during the sprints? – Who defines their

priorities? Do they follow stack prioritization or release level prioritization? What if one of the

items was a pre-requisite for the release that needed prioritization?

6. Who and how will we tie up the dependencies to form a big-bang release?

7. How do we bring in the visibility into the upcoming sprints (specifically for resource planning

than just for technical dependencies)

8. How do we communicate the status of all releases that were being executed in parallel

9. How to run the stack sprints with production fixes in parallel?

10. How do we manage Code merge and branch out when parallel product initiatives are being

run.

a. When and how to manage the code merge?

b. Who manages the code merges?

c. Who manages the merge down and merge up issues?

d. What if there has been an update to the production by the time the merge happens?

5. The Framework

As in Agile, it’s all about “experimenting”. There is no one silver bullet framework that can apply to

these situations. Few team members from Product, Engineering, QA and Project Management

debated, deliberated and devised the framework that works on delivering multiple releases in parallel

with the finite set of resources. The input from the Senior Management / Executive Management was

the priority list of releases (with no specific absolute priority as the company had to go in for flank

attack)

Step 1: Gap Analysis The key thing was to identify the gaps in the system before the teams could adopt the new way of

release planning and execution. To that effect, a quick retrospective was put between the mid and

senior leadership team to analyze what were the core areas that needed attention. The focus was

deliberately not on tools or techniques but to address those bottlenecks that would help the teams to

plan and execute faster and better. The Pareto principle of identifying those 20% things that would

give 80% lift was the focus of the team.

7 Page

Few top-level things that were identified were:

Release Dependencies that currently go unnoticed and also unattended mid-flight (once the

releases were in execution). It was a given that 90% of releases had dependencies.

Inter team sequencing of the dependencies and deployments

Intra-team sequencing of code merges, release signoffs by QA, regressing merged builds

Constant and consistent visibility to teams and upper management related to state of

releases

Coordination with technical and business operations in communicating the release calendars

Step 2: Plan the Phase 1 There was a concise quarterly plan to attack all of the problems above. The first quarter was totally

dedicated towards ensuring

Develop a plan of record for tracking all releases (fix the communication issue)

Analysis of the current state of the release (many were struck at 10%, some at 80% for

weeks) – find out the dependencies

Identify the hardware requirements delaying releases and address them on war footing basis

Framework of release thrashing where all stakeholders across various stacks contributed

Production push process that outlined basic requirements that should be met (outlining the

code changes, install steps, pre-requisites, notification and approvals)

Step 3: The push / pull

Releases were tracked as “Epics” (Parent User Stories in Agile terminology). Product Owners own the

Epics and would work with Architects (forming the core team) to break it down into child user stories

that carried the technical details related to how the business requirement can be met. These “child

user stories” would reside in the team backlogs, get executed and deployed in the sequence that

would be defined by the lead architect of the release. So, as you can imagine, stack team A would

have an a-la-carte of requirements from various releases.

The focus during the first quarter was not on any agile tool but a simple Google Doc that was shared

with the stack team members. The Google Docs used to have the child user stories (the sprint

backlog), the capacity allocation per stack team, the vacation plans of team members, customized

burn down charts using the planned v/s actual values. The Scrum masters were the glue between the

teams and used to manage the coordination between the various teams (using scrum of scrums as a

forum)

Tracking releases in execution

The key thing by which releases could be tracked was to park them in the right state. Each release traversed through:

Pipelined > Approved > In Ideation > In Execution > Deployed > Under Review > Closed

8 Page

Each of these release states had a gating criterion through which the states moved by the release managers.

States Explained - Pipelined, Approved & In Ideation

Each release gets pipelined for approval to the executive team where the releases are scrutinized

for completeness, business need and ROI and then approved. The approvals are obtained in a larger

forum where all other releases compete in the portfolio and stakeholders can substantiate their case.

This used to be the global prioritization meeting chaired by the executive management (Chief Product

Officer and Chief Technology Officer)

Process: (Primarily driven by core team with help of Scrum Master) - Iterative Process

1. Ideation between Product Owner, UX and Architect (Core team) to arrive at the first level

Release Backlog (Owner: PO)

2. Iterate on the low-fidelity mocks by the UX team, if applicable. (Owner: PO and UX)

3. Lead Architects discusses with their peers (within other stack teams) to arrive at the

dependencies between stack teams and document them (Owner: Architect)

4. Generate artifacts on the system level design changes (Owner: Architect + Lead Engineers)

5. Discuss with Engineering managers, Leads, Technology operations and other stakeholders

(Owner: Architect) (Iterative Process)

6. Discuss with Biz Ops, Business, Sales (Owner: PO) (Iterative Process)

a. Does sales have a need for this release? Do they have a potential customer base?

b. Does this require a PR? Does this require an organization wide training?

7. Discuss with PO continuously to identify if the system that is being developed is what is really

being envisioned, tying everything together (Owner: Architect)

Output : Release moves into Ideation

1. Child User Stories under the PI (Owner: Architect)

2. High-level Technical Specifications - Architectural / Design Documents (Owner: Architect +

required members of technical team)

3. Low Fidelity Mocks (Owner: UX)

States Explained - In Execution

This is the phase where the team can begin the sprints using the release backlog. This is essentially

the sprint planning and execution cycle where there is a constant collaboration between Product,

Scrum Team, Core Team and other stakeholders with the help of daily standup meetings, status

reporting, retrospectives and demos. The key to this phase is that the teams self-organize, work

together, inspect and adapt using all the agile tools (planning, daily reporting, retrospectives). Each

stack team would use the pull method to move the requirements from phases (dev > qa >

deployment)

Input:

1. Child User Stories updated from the previous stage

2. Architectural Artifacts (Design WIKI, diagrams)

3. Sign-offs from the dependent stakeholders (Ops, Sales)

9 Page

Process: (Primarily moderated / driven by Scrum Master)

1. Release Planning (with few stakeholders) - To identify the tentative TTM / number of sprints

(Owner: Architect + PO + Scrum Master + Few key stakeholders from the scrum team + Tech

and Biz Ops)

2. Capacity Planning - To identify the availability of time of each team member (Owner:

Engineering Manager(s))

3. Sprint Planning and Sprint Execution

Output:

1. Deployments based on Release Dependency Matrix (Owner: Scrum Team)

2. Updated Communication Plan (blocking calendars for demos, look ahead planning for the

future sprints etc) (Owner: Scrum Master)

States Explained - Deployed / Under Review

Each release post-deployment (of all stacks) moves and stays in Deployed till reviewed by the

Technology QA team. Under Review is the phase where the release is reviewed for its adoption

metrics in the field.

The Execution Framework - Sprints, Stand-ups... Typically 50% of releases would span across multiple sprints across multiple teams. The core and

scrum teams perform a high level release planning (and breaks it to sprints) to identify the sequence

of execution, the dependencies etc. Each sprint typically would require engineers across the stacks to

contribute. In order to ensure the technical dependencies were managed well, each contributing stack

team has to agree to a “contract” that gets defined between the teams that would define how a

release would cut across the entire system. The lead architect who is part of the core team would

define this contract and pass it on to the teams to elaborate and suggest alternatives or changes.

Also, the expectation is that technical bottleneck of the management of code branches and merge

would need to be handled within the stack teams.

10 Page

The Simple 3-Step Framework (push & pull)

Figure 1 – A 3 Step Process of Release (push to backlog)

Each team used to run their sprint planning and sprints and the Scrum masters used to align the

deployment sequencing with the help of key members in the teams. There were trials done in terms of

making all releases begin and end on the same dates but realized that the key stakeholders who

need to be involved in sprint planning (architects, engineering owners, line managers) were stretched

thin between meetings. Hence we realized that we needed to have the teams run at their own pace

and it was left to the Scrum masters who doubled up as release managers to tie up the deployment

sequencing.

The Pull – Kanban Within the backlog, the entire stack workflow uses the Kanban mode of “PULL”-ing the user stories

from one state to another. Once US is ready for QA, the QA team will pull it into QA based on the exit

criteria specified as a team.

11 Page

Arriving at Release Dates Once the dates are arrived at each of the stacks (Code Complete, QA signoff, Deployment), the

longest pole in the entire release including integration testing timelines was taken as the final

“Targeted Release Date” and communicated to the field teams. The sprints continued to move and

the dates for the releases are arrived at as and when the deployment dates are available. Scrum

Masters owned the diligence in tying up these deployment dates to deliver releases, consistently.

Figure 2 – Arriving at Release Date based on Deployment Sequencing

Managing Release-Level Visibility – The Key to Success! Though the engineers from the stack teams worked within the stack, the daily stand-up meetings,

dependencies and the status would be looked at also from the release level. The standup meetings

proved extremely important to discuss if there were any slippages, contract issues and blockers. A

focused release-level daily stand-up meetings (also called the Scrum of Scrums) had key

representatives from each of the stacks coming together in analyzing how the releases were

progressing and how the teams could help each other move forward towards the release. These

meetings helped in reporting the release level status, which became the consistent focus within the

organization.

The Leap into Phase 2 Taking the learning from Phase 1 using retrospectives with teams and executive sponsors, the teams

moved into Phase 2 around end of Q4. By then, the teams had multiplied thrice in size and the

number of releases had doubled (posed a big challenge). The teams moved into managing

releases as individual Portfolio Items but the core push/pull framework remained intact (just the

terminologies were changed). The reason to move into Portfolio Management was to provide the

12 Page

visibility to the executive team on where the team and their capacities were being utilized (bucketing

investments)

Figure 3 – The current framework that is based on PI and US

(PIxyz denotes the epic and USxyz denotes the child user stories below the release)

6. Key Benefits of the framework

There was couple of benefits (both qualitative and quantitative):

1. The framework has been time tested and proved that teams could still get releases done

even if there were multiple dependencies.

2. Teams not just in Bangalore but also across the globe could adopt to the framework and had

clear visibility into where their dependencies were (which stack, what sprint, when it will be

deployed, to name a few)

3. The framework enabled us to scale to around 50 parallel releases during Q2 of 2012

4. In the last 4 quarters, the number of releases that were done was close to 120+ (with the last

2 quarters contributing to 60%). Even when the initiatives were brought in to accelerate the

execution, the framework stayed put and guided the releases to closure.

13 Page

7. Key Challenges Overcome

Though the framework sounds very straightforward and commonplace, the following were the

challenges that had to be mitigated

Operate from week 1 in a no-pilot mode (implement the framework across the teams at one

go) without slowing down the existing releases.

Experiment, learn, adapt, revise and re-implement in fast turnaround cycles (across 12+

teams)

Sudden surge in the team sizes due to business needs (adoption of framework, adapting to

new business asks and timelines)

o Pressure on teams to deliver more due to the surge in team sizes and in turn the

capacity

Executive team willing to provide leeway from the existing framework to see if that would

speed up execution (this put things out of gear which had to be brought back in)

Stack teams having sudden priority changes (tech debts and scalability issues) thereby

jeopardizing the entire release timelines

Adoption issues due to business pressures

Frequent shift in business priorities (which is a given in any industry) leading to randomization

of releases

o All of a sudden towards end of Q1 2012, the organization had thrust a mandate of 30

releases in 30 days and teams had to huddle to accomplish them. The framework still

came in handy to lay out the dependencies, t-shirt sizing and release level tracking

o The teams are in the 3rd

such initiative post that

8. Conclusion

It is strongly recommended that the team who defines the framework needs to start off with an open

mind for feedback and failures. The first step to defining any framework is a complete analysis of the

existing bottlenecks than to go with a silver bullet of Scrum, XP or mix of other methodologies. The

second significant step is to work with key stakeholders to see if all their demands have been met

(else the adoption becomes bleak). It is extremely important to set expectations with senior and

executive management on what milestones will be achieved in the adoption cycle. This framework

above helps teams who have dependencies in managing releases. In a nutshell, the chain is as

strong as the weakest link. Hence, it is imperative that there is a unified process across the entire

organization so that everybody moves at a sustainable pace. It is equally important that all

stakeholders play their part well so that things do not break down.

9. References

1. Scrum Guide: http://www.scrum.org/storage/scrumguides/Scrum%20Guide.pdf

2. Scrum Primer:

http://assets.scrumfoundation.com/downloads/1/scrumprimer121.pdf?1294640838

14 Page

10. Author’s Profile:

13 years of IT experience with excellent exposure to various Project Management Methodologies - Agile Scrum (Iterative and Incremental) with XP, Lean, Kanban and Traditional. Has strong experience and exposure in both product and service companies. Currently heading the Technology Project Management Office at InMobi responsible for driving the portfolio of releases across 18+ engineering teams. In the last two engagements, has owned the responsibility for coaching, mentoring teams to adopt Agile Scum as the delivery framework. Leading a team of 4 Project Managers who are playing both the scrum master and release manager roles.

Email: venkat@inmobi.com