Zero-Downtime Gerrit Code Review Upgrade

#GerritUserSummit

Zero Downtime GerritHow to upgrade your Gerrit master while keeping the balls rolling.

Luca [email protected]

@gitenterprise @gerritreview #GerritUserSummit

• Founded in the UK• HQ in London, Offices in the UK and Italy• Committed to OpenSource

About GerritForge


• Co-founder and Director of GerritForge • over 20 years in Agile

Development and ALM• Contributor to many

OpenSource projects• Mind and hands of

@gitenterprise and @gerritreview Twitter feeds

About Luca Milanesio


Life outside Google Data-Centers

•Gerrit is quite stable

•GerritHub uptime to date: 226 days

• Extensive unit & integration tests suite



BUT



"Sometimes" we need to upgrade

– Feb 2 gerrit-2.11.6– Feb 12 gerrit-2.11.7– Mar 3 gerrit-2.12.1– Mar 9 gerrit-2.11.8– Mar 11 gerrit-2.12.2– May 17 gerrit-2.11.9

– Jul 7 gerrit-2.12.3– Aug 24 gerrit-2.12.4– Aug 26 gerrit-2.11.10– Sep 24 gerrit-2.12.5– Oct 21 gerrit-2.13– Oct 21 gerrit-2.13.1– Oct 21 gerrit-2.13.2



How others are managing upgrades



Three flavors up upgrades1. Code-only : Typically patch-releases2. Code + DB Schema upgrade: Major releases3. Code + DB Schema upgrade + Repos upgrade: Painful

Major release



How Google is managing upgrades



Level of pain in the OpenSource Gerrit implementations1. Code-only

5 mins of fear2. Code + DB Schema upgrade

1/2 days of planning + 15/20' outage3. Code + DB Schema upgrade + Repos upgrade:

1/2 months of planning + leap of faith


Levels of Zero-Downtime

0. Outage1. Server always-on and reachable2. Gerrit UX visible and responsive3. Git/HTTP and Git/SSH accessible4. Most of the functions operational5. All functions fully operational


Levels of Zero-Downtime: where are we?

0. Outage1. Server always-on and reachable2. Gerrit UX visible and responsive3. Git/HTTP and Git/SSH accessible4. Most of the functions operational5. All functions fully operational


HA Proxy

Zero-Downtime Upgrade – Stage 1

Client traffic

Ver. X (empty)RSync

DB Export / Import


HA Proxy

Zero-Downtime Upgrade – Stage 2

Client traffic

Ver. X Ver. XGerrit replication


HA Proxy

Zero-Downtime Upgrade – Stage 3 upgrade

Client traffic

Ver. X Ver. X + N

Upgrade Gerrit war to Ver. X+NInit + Off-line reindex


HA Proxy

Zero-Downtime Upgrade – Stage 4 – read-only

Client traffic

Ver. X Ver. X + N

Install Read-only plugin


HA Proxy

Zero-Downtime Upgrade – Stage 5 – delta replication

Client traffic

Ver. X Ver. X + N

Install Read-only plugin

Delta replication

DB Export / Re-Import


HA Proxy

Zero-Downtime Upgrade – Stage 6 – Reinit

Client traffic

Ver. X Ver. X + N

READ-ONLY Init + Startup


HA Proxy

Zero-Downtime Upgrade – Stage 7 – delta reindex

Client traffic

Ver. X Ver. X + N

Delta reindexREAD-ONLY


HA Proxy

Zero-Downtime Upgrade – Stage 8 – traffic redirect

Client traffic

Ver. X Ver. X + N

Cache warm-up


Timing and service level

1Stages 2 3 4 5 6 7

Times: days hours 10s 5' 5' 1s

ServiceLevel 100% up on Ver. X Read-only Ver. X 100% up on Ver. X

+ N

8


DEMO


Wants to know more?

www.gerritforge.com/contact

Zero-Downtime Gerrit Code Review Upgrade

Technology

Transcript of Zero-Downtime Gerrit Code Review Upgrade