Distributed development with Bazaarnano-bio.ehu.es/files/lecture-pouillon-bazaar.pdfOverview An...

Overview An extremely flexible tool Example: the management of ABINIT The end

Distributed development with Bazaar

Yann Pouillon

1. Facultad de Químicas, Universidad del País Vasco UPV/EHU,Donostia-San Sebastián, Spain.

2. European Theoretical Spectroscopy Facility (ETSF), Spain.

CECAM Tutorial — Zaragoza, Spain2010/06/21

Outline

1 OverviewAn evolutionary processQuick tour of Bazaar 2.1

2 An extremely flexible toolKeeping up with an evolving projectPopular Bazaar workflows

3 Example: the management of ABINITSustaining the growth of the codeThe ABINIT Forge

Outline

Different times, different challenges, different models

1970’s: first steps (SCCS times)VCS = tool to keep history of single filesweave method: uniform data retrieval timeoutcome: SCCS adopted by Unix

1980’s: from solo to team work (RCS times)independent streams of developmentproject management & VCS decoupledoutcome: data retrieval efficiency & reliability

1990’s: organizing collaboration (CVS times)golden age of client-server modeluprising of the centralized modeloutcome: atomicity, file management, deployment

2000’s: going back to freedomemergence of the distributed modelexplosion of "ecological diversity"outcome: better flexibility, usability & scaling

The centralized model

Strength: control

Individualistic (driving force = project leader)

Requires a server

Requires permanent connection to the server

Snapshotting & publishing tightly bound

Associated social model: benevolent dictatorship

Subversion: "subvert the beast"

Since 2000: overcome CVS limits while keeping compatibilityToday: probably the most popular for the centralized modelReason: smooth transition from CVS

Strength: control

Requires a server

Strength: control

Requires a server

The distributed model

Strength: freedom & autonomy

Team-oriented (driving force = community)

Exploration of new concepts made easy (no loss of history)

Network access: only to publish or access remote changes

Associated social model: doocracythe actual developers take the decisionsdynamical reorganization, delegation of powersmostly bottom-up structure

Bazaar: "VCS for Human Beings"

"Just a tool" philosophy + bindings to other VCS’s

Centralized vs. distributed: a false debate

Former bone of contention

Patch bomb = unmanageable contribution

Distributed model: no protection

Centralized model: just more difficult

Patch bomb = social failure =⇒ no technical solution

Importance of fluid communication

Interplay between social & technical aspects

Centralized model popular for historical reasons

Balance between control & freedom: YOU choose !

Reminder: CVS = beast of SVN−→ SVN = beast of other tools

Centralized vs. distributed: a false debate

Former bone of contention

Patch bomb = unmanageable contribution

Distributed model: no protection

Centralized model: just more difficult

Patch bomb = social failure =⇒ no technical solution

Importance of fluid communication

Interplay between social & technical aspects

Centralized model popular for historical reasons

Balance between control & freedom: YOU choose !

Reminder: CVS = beast of SVN−→ SVN = beast of other tools

In conclusion ...

Do not feed the troll!

Outline

Main objectives of Bazaar

Usability: "DVCS for Human Beings" (Ubuntu philosophy)

Portability: only requires Python ≥ 2.4Efficiency:

smart support for renaming files & directoriesoptimal use of disk spaceminimal overhead (fast branching & merging)

Productivity: "just a tool" (read the documentation)

Extensibility: Python plugins & well-documented API

Pluggability: suitable for embedded systemsSafety:

strong & active open-source communitytest suite with > 10,000 automatic tests

Freedom: Bazaar is Free Software (GPLv2+)

Bazaar core concepts

Revision: snapshot of the state of a file treeBranch: ordered series of revisionsRepository : store of revisions (branches sharing history)Working Tree: version-controlled directory holding sourceCheckout : source tree remotely connected to a branchStandalone Tree: independent branch with working treeBranching: action of splitting development linesMerging: action of joining development linesConflict : two branches contain different changes at thesame placeCriss-Cross: painfully intricate branches

For more informationhttp://doc.bazaar-vcs.org/bzr.dev/en/user-guide/

http://doc.bazaar-vcs.org/bzr.dev/en/user-reference/bzr_man.html

The Bazaar working tree

Directory structuremy_project/

+-.bzr/+-README+-branch/+-branch-format+-branch-lock/+-checkout/+-repository/

+-.bzrignore+-file1+-file2+-file3+-...

Bazaar main commands

bzr help: get generic help

bzr help commands: get help on commands

bzr [command] --help: get help on [command]

bzr init: create a standalone tree

bzr status: display current status of a branch

bzr add: tell Bazaar to version-control files & dirs

bzr remove: tell Bazaar to stop version-controlling files

bzr commit: take a snapshot of the working tree

bzr push: publish history to a remote branch

bzr pull: import history from a remote branch

bzr info: display branch information

Committing changes

Writing changelogsOne short summary line (no trailing dot)

* dir1/sub1/file1: Some changes. Makefull sentences.

* dir2/sub2/file2,dir3/sub3/file3: Someother changes.

* dir4/sub4/file4: Related changes (noblank line before).

* Additional notes and issues.

Recommended: GNU changelog format

All lines < 80 characters and start at first column

Reference document:http://www.gnu.org/prep/standards/html_node/Change-Logs.html

Merging branches

When joining development lines: conflicts may happenSolving conflicts: graphical tools might help, e.g.:

kdiff3xxdiff

bzr merge: merge a branch into another−→ criss-cross =⇒ bzr merge --weave

bzr conflicts: display remaining conflicts

bzr resolved: mark a conflict as resolved

bzr revert: rollback files/everything to the last snapshot

Merging branches

When joining development lines: conflicts may happenSolving conflicts: graphical tools might help, e.g.:

kdiff3xxdiff

bzr merge: merge a branch into another−→ criss-cross =⇒ bzr merge --weave

bzr conflicts: display remaining conflicts

bzr resolved: mark a conflict as resolved

bzr revert: rollback files/everything to the last snapshot

Good practice

Committing safely

ALWAYS issue a bzr status before committing

Merging safely

1 Commit first2 Merge the other branch3 Solve the conflicts4 Commit again5 Continue developing ...

Good practice

Committing safely

ALWAYS issue a bzr status before committing

Merging safely

1 Commit first2 Merge the other branch3 Solve the conflicts4 Commit again5 Continue developing ...

User interfaces

GUI: Bazaar Explorer

Web: Loggerhead

Outline

There is no "one true way"

Definition

Workflow: way a team chooses to work together

In a project: workflow changes may become necessary

With Bazaar: the tool adapts to your needs

Purpose: let you focus on what truly matters

Following workflow descriptions

Extracted from the Bazaar User Guidehttp://doc.bazaar-vcs.org/bzr.dev/en/user-guide/

There is no "one true way"

Definition

Workflow: way a team chooses to work together

In a project: workflow changes may become necessary

With Bazaar: the tool adapts to your needs

Purpose: let you focus on what truly matters

Following workflow descriptions

Extracted from the Bazaar User Guidehttp://doc.bazaar-vcs.org/bzr.dev/en/user-guide/

Outline

Solo / single

Best practice for programming / writing:backup of old versionsrollback to an earlier statetracking of historyfast and lightweight

Papers, posters, talks, reviews, ...

Partners / peer-to-peer

Advantages over solo:easier sharing of dataline-wise tracking of changes (who did what)no server requiredintelligent merging

Lock-step / centralized

Advantages over centralized VCS’s:better branching and mergingsimpler renaming support (files & directories)ability to change workflow whenever it makes sense

Centralized with local commits

Advantages over centralized:can work offline (e.g. during travel)less interference of bad commits

Distributed with shared mainline

Advantages over centralized with local commits:easier organization of workcross-merging possibleshared repositories and integrated merging

Distributed with human gatekeeper

Advantages over distributed with shared mainline:systematic review of the codetighter control over when changes go to mainline

Distributed with automatic gatekeeper

Advantages over distributed with human gatekeeper:systematic testing of the code (higher mainline integrity)better scaling with team growthdedicated tool: Patch Queue Manager (PQM)

Outline

The mutation of ABINIT

Challenges

Growing size of the code

Growing complexity of the code

Growing number of contributors

Growing size of the community

ABINIT developers: all around the world + heterogeneousUntil 2004

source code managed by CVSworkflow: mostly solo

2004 −→ 2008transition: CVS −→ GNU Arch −→ Bazaarexploration of workflows

Breaking free from the client-server model

GNU Arch (2001)

Originally: set of specifications to manage project history−→ generic protocol−→ overcome the drawbacks of centralized versioning−→ freedom regarding implementations

Specifically designed for distributed development

Work while disconnected from the network

Branching and mirroring do not require cooperationUses available infrastructure: no dedicated server

support for multiple network protocolspermissions managed by the underlying system

Implementations in C / C++ / Python

Breaking free from the client-server model

GNU Arch (2001)

Originally: set of specifications to manage project history−→ generic protocol−→ overcome the drawbacks of centralized versioning−→ freedom regarding implementations

Specifically designed for distributed development

Work while disconnected from the network

Branching and mirroring do not require cooperationUses available infrastructure: no dedicated server

support for multiple network protocolspermissions managed by the underlying system

Implementations in C / C++ / Python

From GNU Arch to Bazaar

TLA (Tom Lord’s Arch) [2001–2005]home page: http://wiki.gnuarch.org/FrontPageseveral forks: ArX, arch++, bazpart of the GNU Project in 2003basic maintenance since 2006

Baz (fork of TLA) [2004–2005]project initiated by Canonical Ltd. (Ubuntu)simpler UI, new archive format, better algorithmsmany issues: usability, lack of flexibility, bad scaling

Bazaar "New Generation" (bzr) [since 2005]home page: http://bazaar-vcs.org/not based on GNU Arch anymore, still using some conceptscompletely rewritten from scratch in Python 2.4key component of Ubuntu and Launchpad

Outline

Components of the Forge

File server available through SSHbackup facilitymerging made easy

Repositoriesone per developer / teamcontaining all necessary branchesstrict naming conventions for public branches

Maintenance utilitiescreation of repositories & branchesinformation about archive statusmanagement of permissions (Unix)

Test farm / nightly buildschecking branch status: buildability & mergeabilitymonitoring critical filesbuilding branches on various architectures−→ various hardware / OS / compiler combinationsreporting nicely & daily online

Different scales, different workflows

Top level: distributed with human gatekeeper

Geographical teams: peer-to-peer, lock-step, local commits

Topical teams: peer-to-peer, distributed + shared mainline

Maintainers: solo, using shared repositories

Example of shared repositoryABINIT/

+-6.3/ # <--- Repository+-6.3.0-pouillon/+-6.3.0-trunk/+-6.3.0-gmatteo-public/

Benefits to ABINIT from Bazaar

Central archive for data storage & sharing

Reduced overhead for merges

Easy propagation of partial merges

All developer teams may work as they prefer

Only initial support required (autonomy comes fast)

Services: test farm, branch browser, web reporting

Good practice

A technical tool never fulfills social dutiesTo ensure sustainability:

define & enforce clear rules for developers

define & enforce clear policies for the branches

make sure information flows

Acknowledgments

Funding by MICINN & ETORTEK

ABINIT Developers

Bazaar Developers

Wikimedia Commons & Wikipedia

art.gnome.org

Always remember ...

Golden rule of versioning

Commit early, commit often!

Thank you!

Always remember ...

Golden rule of versioning

Commit early, commit often!

Thank you!

Distributed development with Bazaarnano-bio.ehu.es/files/lecture-pouillon-bazaar.pdfOverview An...

Documents

Transcript of Distributed development with Bazaarnano-bio.ehu.es/files/lecture-pouillon-bazaar.pdfOverview An...