Post on 25-Jan-2017
Tips for Administering Complex Distributed Perforce EnvironmentsMichael MirmanPerforce Administrator
2
Overview
System Overview Shelved changes in Distributed Environment Using broker as a router: Routing challenges Some administration tasks• Expand capabilities of the –df option• Removing users• Regular database defragmentation• Controlled fail-over
3
System Overview
• ~1,800 users worldwide• ~15M archive files• ~90 hosts• Database ~0.6Tb on
each server• Single production
system
Commit server
Edge servers
perforce:1666P4broker (router)
perforce:1666P4broker (router)
perforce:1666P4broker (router)
perforce:1666P4broker (router)
4
System Overview (cont.)
Local Data CenterWorkspace servers
perforce:1666P4broker (router)perforce:1666
P4broker (router)
perforce:1666P4broker (router)
perforce:1666P4broker (router)
Commit server
Archive on NFS disk
Edge servers
Remote Data Center
Archive on a local disk
Archive on a local disk
WkspXXWkspXXStandbyWkspXXUtility
EdgeXXEdgeXXStandby
5
Shelved Changes in Distributed Environment
Users want transparency using shelves Limitation in Distributed Environment to the
originated server:p4 changes –s shelved //some/path/...
Work around:1. Promote all shelves2. Broker redirection for shelved changes query
6
Promoting Shelved Changes in Distributed Environment Edge server shares the archive with the commit server:• EdgeServer#lbr.replication=shared• EdgeServer#startup.1 = pull -i 1 -J /path/to/journal
Edge server does not share the archive with the commit server:• EdgeServer#lbr.replication=readonly• EdgeServer#startup.1 = pull -i 1 -J /path/to/journal• EdgeServer#startup.2 = pull -u -i 1• EdgeServer#startup.3 = pull -u -i 1• EdgeServer#dm.shelve.promote=1
7
Broker Redirection for Shelved Changes in Distributed Environment p4 changes –s shelved //some/path/... works fine when
routed to the commit servertarget = commitserver:1666command: ^changes${ flags = -s; args = shelved; action = redirect; destination target;}
It would be nicer if we didn’t have to reroute this command
8
Routing caveats
Any command has client context
p4 clients –a –e clientname
p4broker Commit server
9
Routing caveats (cont.)
Regular unloading unused clients to conserve db space Occasional rebalancing edge servers requires removal and/or
rebuild of the routing tables Users without loaded clients don’t get into the routing table Problems for the user to come back• -> p4 reload -c myclient• Client 'myclient' is restricted to use on server 'WkspXX', not on server
'WkspYY'. Workaround: p4 -Zroute=WkspXX reload -c myclient
10
Creating a New Client in Distributed Environment Scenario 1
mkdir newdirecho P4CLIENT=myclient > $P4CONFIGp4 client# edit client spec, save; exit
Usually works fine When it doesn’t:• Current user does not exist in the routing table (default routing)
11
Creating a New Client in Distributed Environment (cont.) Scenario 2
cd /myfriend/directoryp4 client myclient# edit client spec (possibly including Root); save; exit
May have bad consequences if:• /myfriend/directory is a valid workspace, defining routing• This routing is not what current user has as the default• Future new clients may go to a different edge server• Not all clients shown in P4V; users must have multiple sessions• Seeing pending changes between clients is difficult or impossible
12
Routing caveats - Solution
Maintain “anchor” clients, which never get unloaded• Client .mmirman.anchor 2016/02/03 root /tmp/fake 'Anchor client to
ensure proper routing. Please don't delete. ' Rebuild routing tables with anchor clients Downside: user can see this client and considers it noise What we would love to see: ability to manage routing table
without rebuilding from scratch
Some Administration Tasks
14
Using –df option
Task: Allow usage of –df in “p4 change”, “p4 shelve”, “p4 client”, to the owners of corresponding items, as well as admins
Broker configuration: command: ^change$ { action = filter; flags = -d; execute = "/path/to/change.wrap"; }
15
Extensions for the –df option
Deleting client• Delete shelves or not?• We don’t like the -Fs option (save shelves): it creates clutter• Shelf cannot be deleted if its client does not exist anymore• Our implementation is a broker script with -Ds option
Deleting stream• p4 stream –df• Delete all clients, including unloaded• Allow only admins to do this• Still under discussion (big implications from client –df)
16
Removing user
Someone left company a reclaim Perforce license Remove user from Perforce groups Remove user spec ... Wait!• Remove user’s clients (including unloaded and locked)
- Remove shelved changes (shelve -d or -df); may require reverting files not resolved in other users’ clients
- Revert open files (p4 revert -k)- Disassociate changes from jobs- Remove pending changes (change -d or -df)
17
Regular database defragmentation
Have a regular schedule to maintain good performance
Example:• Rebuild all standbys at the beginning of the month• Controlled fail-over every second Sunday• Rebuild standbys again later in the month
18
Rebuild Standby servers
Task: Regularly rebuild standby servers to make them available for controlled fail-overs-> rebuild_scheduleSchedule to rebuild servers:Today (16) we rebuild these servers: Edge01Standby CommitStandbyDay 1: Edge01Standby CommitStandbyDay 2: Wksp00xxStandby Edge02StandbyDay 3: Wksp01xxStandby Wksp00yyStandbyDay 4: Wksp02xxStandby Wksp03xxStandbyDay 5: Wksp04xxStandby Wksp00xxUtilityDay 6: Wksp00yyUtility Wksp01xxUtilityDay 7: Wksp02xxUtility Wksp03xxUtilityDay 8: Wksp04xxUtilityDay 16: Edge01Standby CommitStandbyDay 17: Wksp00xxStandby Edge02StandbyDay 18: Wksp01xxStandby Wksp00yyStandbyDay 19: Wksp02xxStandby Wksp03xxStandbyDay 20: Wksp04xxStandby Wksp00xxUtilityDay 21: Wksp00yyUtility Wksp01xxUtilityDay 22: Wksp02xxUtility Wksp03xxUtilityDay 23: Wksp04xxUtility
19
Controlled Fail Over in Distributed Environment Shut down replication to the server to be failed over (p4 monitor
terminate) Shut down server and its standby (p4 admin stop) Change server.id in the root directories Remount the archive (r/o on the standby; r/w on the new server) Remove the state file (it will be automatically recreated later) Bring up the new server Bring up the new standby Restart other replications (if any) of the server
Q & AThank you!
Michael.Mirman@mathworks.com