Perforce Performance

Perforce Performance

Michael ShieldsPerformance Lab Manager

Perforce Software

(in the spirit of www.snopes.com)

© 2007 Perforce Software 3

www.snopes.com

• Research validity of outrageous claims• for those emails that "just don't seem right"

• Example:• Claim: {eTailer} suspending account until updated• Status: False

• Urban legends, common fallacies, etc.• Advertisements aplenty


Objectives

• Explore:• metadata locking• filesystem cache• exclusionary entries• embedded wildcards

• Expose misconceptions• Reveal facts not (yet) common knowledge• Discuss some Perforce Server internals


Perforce Metadata

• Claim: Perforce Server uses metadatastored in the db.* files

• Status: ?


Perforce Metadata


• Status: True


Perforce Metadata


• Status: True• For example:

• db.rev, db.change, db.integed, db.have, etc.• Only used by server, not clients• Database schema located at:

www.perforce.com/perforce/doc.current/schema/index.html

• also applicable to checkpoint and journal entries

http://www.perforce.com/perforce/doc.current/schema/index.html


Metadata and Concurrency

• Multiple commands processed simultaneously• Each command must use:

• physically consistent metadata• B-trees in each db.* file must be structurally correct

• e.g. writer not splitting pages a reader is traversing

• logically consistent metadata• db.* files correctly contain related data

• e.g. db.rev* entries exist for submitted changelist

• Synchronization required for metadata access


Metadata Locking

• Claim: There is a single global lock controllingall metadata access

• Status: ?


Metadata Locking


• Status: ?• Claim: The single global lock is an exclusive lock,

serializing all metadata access• Status: ?


Metadata Locking


• Status: False• Claim: The single global lock is an exclusive lock,

serializing all metadata access• Status: False


Metadata Locking


• Status: False• Claim: The single global lock is an exclusive lock,

serializing all metadata access• Status: False• If true, these would severely limit concurrency


Metadata Locking

• Each command locks only the db.* files it needs• A lock on a db.* file is either shared or exclusive

• shared lock• allows access by others using a shared lock• used when might read but won’t write to the db.* file

• exclusive lock• denies access by others• used when might write to the db.* file


Observe Metadata Locking

• Use OS utilities to observe locking behavior• e.g. strace on Linux, truss on FreeBSD and

Solaris, and FileMon on Windows• For example (on Linux):

$ strace -etrace=open,flock,close p4d -f -p 50620(system calls for server startup)Perforce Server starting...

• -f (single-threaded server) to simplify tracing• trace only open(), flock(), and close() calls


Observe Metadata Locking

• From another session:$ p4 -p 50620 sync //depot/main/... //depot/branch1/...//depot/main/file1#1 - added as /p4client/depot/main/file1//depot/main/file2#1 - added as /p4client/depot/main/file2//depot/main/file3#1 - added as /p4client/depot/main/file3//depot/branch1/file1#1 - added as /p4client/depot/branch1/file1//depot/branch1/file2#1 - added as /p4client/depot/branch1/file2//depot/branch1/file3#1 - added as /p4client/depot/branch1/file3

• two arguments• head revisions implied

• six files synced


Command Startup Phaseopen("journal", O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE, 0666) = 5open("db.depot", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 6flock(6, LOCK_SH) = 0flock(6, LOCK_UN) = 0open("license", O_RDONLY|O_LARGEFILE) = 7close(7) = 0open("db.group", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 7flock(7, LOCK_SH) = 0flock(7, LOCK_UN) = 0open("db.protect", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 8flock(8, LOCK_SH) = 0flock(8, LOCK_UN) = 0open("db.user", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 9flock(9, LOCK_EX) (LOCK_SH in 2007.2!) = 0flock(9, LOCK_UN) = 0open("db.domain", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 10flock(10, LOCK_EX) (LOCK_SH in 2007.2!) = 0flock(10, LOCK_UN) = 0open("db.view", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 11flock(11, LOCK_SH) = 0flock(11, LOCK_UN) = 0


Compute Phase for First Argument

open("db.have", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 12flock(12, LOCK_SH) = 0open("db.resolve", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 13flock(13, LOCK_SH) = 0open("db.revdx", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 14flock(14, LOCK_SH) = 0open("db.revhx", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 15flock(15, LOCK_SH) = 0open("db.working", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 16flock(16, LOCK_SH) = 0flock(16, LOCK_UN) = 0flock(15, LOCK_UN) = 0flock(14, LOCK_UN) = 0flock(12, LOCK_UN) = 0flock(13, LOCK_UN) = 0

• computes dataset passed on to next phase• db.revhx and db.revdx locked instead of db.rev

• head revisions implied by argument


Execution Phase for First Argument

open("depot/main/file1,v", O_RDONLY|O_LARGEFILE) = 17close(17) = 0open("depot/main/file2,v", O_RDONLY|O_LARGEFILE) = 17close(17) = 0open("depot/main/file3,v", O_RDONLY|O_LARGEFILE) = 17close(17) = 0

• file content retrieved and sent to client• no db.* file locks held

• for this example, db.have is not updatedduring this execution phase


Compute Phase for Second Argument

flock(12, LOCK_SH) = 0flock(13, LOCK_SH) = 0flock(14, LOCK_SH) = 0flock(15, LOCK_SH) = 0flock(16, LOCK_SH) = 0flock(16, LOCK_UN) = 0flock(15, LOCK_UN) = 0flock(14, LOCK_UN) = 0flock(12, LOCK_UN) = 0flock(13, LOCK_UN) = 0

• for this example, compute phases are similar• no additional db.* files used


Execution Phase for Second Argument

open("depot/main/file1,v", O_RDONLY|O_LARGEFILE) = 17close(17) = 0open("depot/main/file2,v", O_RDONLY|O_LARGEFILE) = 17close(17) = 0open("depot/main/file3,v", O_RDONLY|O_LARGEFILE) = 17close(17) = 0flock(12, LOCK_EX) = 0flock(5, LOCK_EX) = 0flock(5, LOCK_UN) = 0flock(12, LOCK_UN) = 0

• second argument was //depot/branch1/...• lazy copies of //depot/main/...

• db.have (fd 12) updated for all six files syncedduring a single exclusive lock (journal is fd 5)

• batch updates introduced in 2003.2• extended to integrate in 2005.2


Command Cleanup Phase

close(4) = 0close(5) = 0close(9) = 0close(7) = 0close(6) = 0close(10) = 0close(11) = 0close(12) = 0close(13) = 0close(14) = 0close(15) = 0close(16) = 0close(8) = 0

• other system calls not traced, e.g. fsync()


Other Observations

• Locks taken and released as needed• not held over the entire command

• Type of db.* file lock varies throughout command• shared or exclusive• e.g. db.have

• Locks always taken in the same order• eliminates possibility of deadlock

• simpler and faster server code


Review?

• Claim: A command holding only shared locks will not block another from acquiring a shared lock

• Status: ?


Review?


• Status: True


Review?


• Status: True (directly) and false (indirectly)


"Wedge" Scenario


• Status: True (directly) and false (indirectly)• A writer can "wedge" between readers

• writer acquires at least one exclusive lock• writer stalls acquiring an exclusive lock on a db.*

file for which a shared lock is held by a reader• other readers stall acquiring shared locks on db.*

files for which exclusive locks are held by the writer• Not a deadlock


Physical Memory

• Claim: If active server processes are swapped out, then additional physical memory is needed

• Status: ?


Physical Memory


• Status: True


Physical Memory


• Status: True• Claim: If active server processes are not

swapped out, then physical memory is adequate• Status: ?


Physical Memory



swapped out, then physical memory is adequate• Status: Not necessarily true


Physical Memory



swapped out, then physical memory is adequate• Status: Not necessarily true• For best performance, additional

physical memory required for filesystem cache


Filesystem Cache

• Satisfy I/O requests without physical I/O• logical I/O operations complete much faster, which• generally speeds various server phases, which• decreases time db.* file locks are held, which• leads to better concurrency, which• allows more commands processed faster


Filesystem Cache Content

• Consists of recently used pages from disk files• significant portion will be pages from db.* files

• Metadata accessed by traversing B-tree• only portions of the B-tree are read• whole db.* files are generally not cached

• Portions of db.* files not cached• unused client workspaces (portions of db.have)• retired release branches (portions of db.rev*)• etc.


Perforce and Filesystem Cache

• Server relies heavily on filesystem cache• no global cache within server

• less complex and more stable• similar functionality offered by filesystem cache

• Server does cache within each command• each db.* file opened has a private cache

• see "db.* File Generation Numbers" in2006 European User Conference

• additional caching within server data structures


Example of Filesystem Cache Benefit

• Benchmark using Solaris 10 server on Opteron• "forcedirectio" mount option simulates no cache

• Four concurrent "browse" clients on Linux 2.6• each "browse" issues dirs and fstat commands

from the top to the bottom of the repository• typical of drilling down using P4V

• each client does 64 "browses" without delay• random; same seeds used for each run• dedicated subnet


Total Elapsed Time for 256 Browses


Filesystem Cache Effectiveness

• Goal is to minimize physical I/O for db.* files• high level of physical I/O might signal

the need for more physical memory• Monitor physical I/O with OS utilities

• iostat on Linux, FreeBSD, and Solaris• perfmon on Windows


Exclusionary Entries

• Claim: Exclusionary entries in protections and client views always improve performance

• Status: ?




• Status: False




• Status: False• Exclusionary entries: "-" starts depot path

• for example:list group dev * -//depot/product/main/doc/...

• Limiting files is recommended in thePerforce System Administrator's Guide• type of entry has performance implications



• Can significantly impact dirs command• more files scanned in the metadata

• General dirs behavior:• position using non-exclusionary entries• as directory is collected, skip to next directory

• Exclusionary entries necessitatescanning just to collect a directory• must scan to find a directory that's not excluded

• Wider exclusionary scope yields more scanning• subdirectories (recursively) included in scope


dirs Examplep4 dirs //depot/product/main/*

write group eng * //depot/product/main/...

//depot/product/main/art/file1.jpgfile2.jpgfile3.jpg

//depot/product/main/doc/file1.fmfile2.fmfile3.fm

//depot/product/main/src/file1.ccfile2.ccfile3.cc




//depot/product/main/art//depot/product/main/art/file1.jpgfile2.jpgfile3.jpg









//depot/product/main/doc







//depot/product/main/src


With Protection Exclusionsp4 dirs //depot/product/main/*

write group eng * //depot/product/main/...list group dev * -//depot/product/main/doc/...







//depot/product/main/art//depot/product/main/art/file1.jpgfile2.jpgfile3.jpg




With Protection and View Exclusionsp4 dirs -C //depot/product/main/*

//depot/product/main/... //dev-client/depot/product/main/...-//depot/product/main/art/... //dev-client/depot/product/main/art/...






Enumerated Inclusionary Protectionsp4 dirs //depot/product/main/*

write group dev * //depot/product/main/art/...write group dev * //depot/product/main/src/...write group doc * //depot/product/main/...write group qa * //depot/product/main/...write group rel * //depot/product/main/...










//depot/product/main/art


Enumerated Inclusionary Protections and Viewsp4 dirs -C //depot/product/main/*

//depot/product/main/doc/... //dev-client/depot/product/main/doc/...//depot/product/main/src/... //dev-client/depot/product/main/src/...






Embedded Wildcards

• Claim: For best performance, protections and client views must be concise

• Status: ?


Embedded Wildcards


• Status: False


Embedded Wildcards


• Status: False• Example usage:

write group doc * //depot/.../doc/...

• Results are typically more concise• might be easier to understand

• Can require additional CPU and I/O• when protections table joined with client view


Minimize Embedded Wildcards

• Repository structure not consistent with usage• examples:

• documentation scattered within source• platform-specific scattered within generic

• Restructuring might be better solution, or ...


Enumerating Entries

• Replace with explicitly enumerated entries• provides same (dataset-specific) functionality• allows for "fine-tuning"

• Typically increases size• increased protection table size tolerated well• increased client view size perhaps not so well

• Performance improvement can be significant• following example for sync compute phase

• 7000 files returned


CPU Time


db.revhx Reads


Example Protection Table Entries

• Using embedded wildcards:write group dev * //depot/main/.../00/...write group dev * //depot/main/.../01/......write group dev * //depot/main/.../08/...write group dev * //depot/main/.../09/...

• Using enumerated entries:write group dev * //depot/main/00/00/...write group dev * //depot/main/00/01/......write group dev * //depot/main/00/08/...write group dev * //depot/main/00/09/...write group dev * //depot/main/01/00/...write group dev * //depot/main/01/01/......write group dev * //depot/main/99/08/...write group dev * //depot/main/99/09/...


Example Client View Entries

• Using embedded wildcards://depot/main/.../00/... //dev-client/depot/main/.../00/...

• Using enumerated entries://depot/main/00/00/... //dev-client/depot/main/00/00/...//depot/main/01/00/... //dev-client/depot/main/01/00/......//depot/main/98/00/... //dev-client/depot/main/98/00/...//depot/main/99/00/... //dev-client/depot/main/99/00/...


Summary


• Status: True


• Status: False


Summary

• Claim: The single global lock is an exclusive lock,serializing all metadata access

• Status: False


• Status: True (directly) and false (indirectly)


Summary


• Status: True

• Claim: If active server processes are not swapped out, then physical memory is adequate

• Status: Not necessarily true


Summary

• Claim: Exclusionary entries in protections and client views always improves performance

• Status: False


• Status: False

Questions?

Perforce Performance

Documents

Transcript of Perforce Performance