Post on 17-Jul-2015
FreeBSD package management system
Vsevolod Stakhovvsevolod@FreeBSD.org
BSDCan May 17, 2014
What is pkg
Pkg (previously pkgng) is the binary package managementsystem written for FreeBSD.
I Binary packages management
I Replaces old pkg_* tools
I Uses central sqlite3 based storage
I Provides the comprehensive toolset for binary packagesmanagement
Vsevolod Stakhov vsevolod@FreeBSD.org 2 of 26
Pkg development goals
The main goal of pkg is to simplify systemmanagement tasks.
I Easy install, remove and upgrade of binary packages
I Integration with the ports
I Automatic resolving of dependencies and conflicts
I Provide secure package management tool
I Encourage users to install software from binary packages
I . . . but do not prevent users from building custompackages using the ports
Vsevolod Stakhov vsevolod@FreeBSD.org 3 of 26
Pkg development goals
The main goal of pkg is to simplify systemmanagement tasks.
I Easy install, remove and upgrade of binary packages
I Integration with the ports
I Automatic resolving of dependencies and conflicts
I Provide secure package management tool
I Encourage users to install software from binary packages
I . . . but do not prevent users from building custompackages using the ports
Vsevolod Stakhov vsevolod@FreeBSD.org 3 of 26
Planned ports and pkg interaction
Build STAGE Packages Pkg Instal
Build Depends
Run Depends
Vsevolod Stakhov vsevolod@FreeBSD.org 4 of 26
What is new in pkg 1.3
I New solver that can automatically resolve complexupgrade or install scenarios
I Improved security by sandboxing untrusted operations:
Pkg environment Sandbox
Unsafe operation
Result
Sandboxing:I archives extractingI vulnxml parsingI repositories signatures checking and public keys
extracting
I Concurrent locking system
Vsevolod Stakhov vsevolod@FreeBSD.org 5 of 26
What is new in pkg 1.3
I New solver that can automatically resolve complexupgrade or install scenarios
I Improved security by sandboxing untrusted operations:
Pkg environment Sandbox
Unsafe operation
Result
Sandboxing:I archives extractingI vulnxml parsingI repositories signatures checking and public keys
extracting
I Concurrent locking system
Vsevolod Stakhov vsevolod@FreeBSD.org 5 of 26
What is new in pkg 1.3
I New solver that can automatically resolve complexupgrade or install scenarios
I Improved security by sandboxing untrusted operations:
Pkg environment Sandbox
Unsafe operation
Result
Sandboxing:I archives extractingI vulnxml parsingI repositories signatures checking and public keys
extracting
I Concurrent locking system
Vsevolod Stakhov vsevolod@FreeBSD.org 5 of 26
Pkg architecture
Mirrors Jobsfetch
Solver
Checksum
+ Manifest
Files...
Package Install
FS
PkgDB
User
Request
Remote
Repos
Vsevolod Stakhov vsevolod@FreeBSD.org 6 of 26
The problems of the old solver in pkg
I Absence of conflicts resolving
I No alternatives support (plain dependencies only)
I Can perform merely a single task: either install orupgrade or remove
Vsevolod Stakhov vsevolod@FreeBSD.org 7 of 26
Tasks to solve
I Ports renaming:I simple: racket-textual→ racket-minimalI splitting/merging:
A
A B C A B C
A
I Ports reorganising:I files movingI dependencies changeI adding or removing new conflicts
Vsevolod Stakhov vsevolod@FreeBSD.org 8 of 26
Tasks to solve
There are another issues to be resolved:
I Find conflicts using files listI Set jobs priorities using the following rules:
I install dependencies firstI check for reverse dependencies and increase priorityI deal with conflicts using the same priorityI packages removing reverses the priority order
Vsevolod Stakhov vsevolod@FreeBSD.org 9 of 26
Existing systems
There are many examples of solvers used in different packagemanagement systems, for example:
I Zypper/SUSE - uses libsolv as the base
I Yum/RedHat - migrating to libsolv
I OpenBSD/pkg_add - uses internal solver
I Apt/Debian - uses internal solver
I Pacman/Archlinux - uses internal solver
Vsevolod Stakhov vsevolod@FreeBSD.org 10 of 26
External solvers
To interact with an external solver we have chosen the CUDFformat used in the Mancoosi research projecthttp://mancoosi.org:
package: devel/libblahversion: 1depends: x11/libfoo
package: security/blahversion: 2depends: devel/libblahconflicts: security/blah-devel
Vsevolod Stakhov vsevolod@FreeBSD.org 11 of 26
Interaction with external solver
There are some limitations and incompatibilities with CUDF.
I CUDF supports plain integers as versions and we need toconvert versions twice
I There is no support of options in CUDF packages formulas
I External solvers are often too complicated and large
I CUDF transformation is expensive in terms ofperformance
Vsevolod Stakhov vsevolod@FreeBSD.org 12 of 26
We need an internal solver!
Alternatives:
I Write own logic of dependencies and conflicts resolution?
I Use some existing solution?
I Use some known algorithm?
Use SAT solver for packages management
SAT expression︷ ︸︸ ︷(x1‖¬x2‖x3)︸ ︷︷ ︸
Clause
&(x3‖¬x1)&(x2)
Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
We need an internal solver!
Alternatives:
I Write own logic of dependencies and conflicts resolution?
I Use some existing solution?
I Use some known algorithm?
Use SAT solver for packages management
SAT expression︷ ︸︸ ︷(x1‖¬x2‖x3)︸ ︷︷ ︸
Clause
&(x3‖¬x1)&(x2)
Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
We need an internal solver!
Alternatives:
I Write own logic of dependencies and conflicts resolution?
I Use some existing solution?
I Use some known algorithm?
Use SAT solver for packages management
SAT expression︷ ︸︸ ︷(x1‖¬x2‖x3)︸ ︷︷ ︸
Clause
&(x3‖¬x1)&(x2)
Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
We need an internal solver!
Alternatives:
I Write own logic of dependencies and conflicts resolution?
I Use some existing solution?
I Use some known algorithm?
Use SAT solver for packages management
SAT expression︷ ︸︸ ︷(x1‖¬x2‖x3)︸ ︷︷ ︸
Clause
&(x3‖¬x1)&(x2)
Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
Packages universeWe convert all packages involved to a packages universe ofthe following structure:
Name
Version
Con�icts Chain
Depend
Con�ict
N1 N2 N3 Nn
V1 V1 V1 V1
V2 V2 V2
V3
Vsevolod Stakhov vsevolod@FreeBSD.org 14 of 26
Making a SAT problem
I Assign a variable to each package: package A→ a1,package B→ b1
I Interpret a request as a set of unary clauses:I Install/Upgrade package A→ (a1)I Delete package B→ (¬b1)
I Convert dependencies and conflicts to disjuncted clauses
Vsevolod Stakhov vsevolod@FreeBSD.org 15 of 26
Converting dependencies and conflicts
I If package A depends on package B (versions B1 and B2),then we can either have package A not installed or any ofB installed:
(¬A‖B1‖B2)
I If we have a conflict between versions of B (B1, B2 and B3)then we ensure that merely one version is installed:
(¬B1‖¬B2)&(¬B1‖¬B3)&(¬B2‖¬B3)︸ ︷︷ ︸Conflicts chain
Vsevolod Stakhov vsevolod@FreeBSD.org 16 of 26
Converting dependencies and conflicts
I If package A depends on package B (versions B1 and B2),then we can either have package A not installed or any ofB installed:
(¬A‖B1‖B2)
I If we have a conflict between versions of B (B1, B2 and B3)then we ensure that merely one version is installed:
(¬B1‖¬B2)&(¬B1‖¬B3)&(¬B2‖¬B3)︸ ︷︷ ︸Conflicts chain
Vsevolod Stakhov vsevolod@FreeBSD.org 16 of 26
The solving of SAT problem
Some rules to follow to speed up SAT problem solving.
I Trivial propagation - solve unary clauses
I Unit propagation - solve clauses with only a singleunsolved variable
I DPLL algorithm backtracking.
I Package specific assumptions.
Vsevolod Stakhov vsevolod@FreeBSD.org 17 of 26
SAT problem propagation
I Trivial propagation - direct install or delete rules
(¬A‖B)& (A)︸︷︷︸true
&(¬C)︸ ︷︷ ︸false
&(¬A‖¬D)
I Unit propagation - simple depends and conflicts
Dependency
(¬A‖B)︸ ︷︷ ︸B→true
&
true︷︸︸︷(A) &
false︷ ︸︸ ︷(¬C)&
Conflict
(¬A‖¬D)︸ ︷︷ ︸D→false
Vsevolod Stakhov vsevolod@FreeBSD.org 18 of 26
SAT problem propagation
I Trivial propagation - direct install or delete rules
(¬A‖B)& (A)︸︷︷︸true
&(¬C)︸ ︷︷ ︸false
&(¬A‖¬D)
I Unit propagation - simple depends and conflicts
Dependency
(¬A‖B)︸ ︷︷ ︸B→true
&
true︷︸︸︷(A) &
false︷ ︸︸ ︷(¬C)&
Conflict
(¬A‖¬D)︸ ︷︷ ︸D→false
Vsevolod Stakhov vsevolod@FreeBSD.org 18 of 26
DPLL algorithm
DPLL is proved to be one of the efficient algorithms to solveSAT problem (not the fastest but more simple thanalternatives).
x1
x2
x3
1
0
1
Vsevolod Stakhov vsevolod@FreeBSD.org 19 of 26
DPLL algorithm
DPLL is proved to be one of the efficient algorithms to solveSAT problem (not the fastest but more simple thanalternatives).
x1
x2
x3
1
0
1
0
Vsevolod Stakhov vsevolod@FreeBSD.org 19 of 26
DPLL algorithm
DPLL is proved to be one of the efficient algorithms to solveSAT problem (not the fastest but more simple thanalternatives).
x1
x2
x3
1
0
1
0
1
0 0
1
Vsevolod Stakhov vsevolod@FreeBSD.org 19 of 26
Package specific assumptions
Pure SAT solvers cannot deal with package management asthey do not consider several packages peculiarities:
I try to keep installed packages (if no direct conflicts)
I do not install packages if they are not needed (but try toupgrade if a user has requested upgrade)
These options also improve SAT performance providing agood initial assignment.
Vsevolod Stakhov vsevolod@FreeBSD.org 20 of 26
Solvers and Pkg
I Pkg may pass the formed universe to an external CUDFsolver:
I convert versionsI format requestI parse output
I Alternatively the internal SAT solver may be used:I convert the universe to SAT problemI formulate requestI ???I PROFIT
Vsevolod Stakhov vsevolod@FreeBSD.org 21 of 26
Perspectives
I Using pkg solver for ports management
I Better support of multiple repositories
I Test different solvers algorithms using CUDF
I New dependencies and conflicts format
I Provides and alternatives
Vsevolod Stakhov vsevolod@FreeBSD.org 22 of 26
New dependencies format
libblah >= 1.0+ option1,+option2‖libfoo! = 1.1
I Can depend on normal packages and virtualpackages (provides)
I Easy to define the concrete dependency versionsI Alternative dependencies
P1Depends
P2 Vx>
Vy Vz !
P3
Or
Con ict
Vsevolod Stakhov vsevolod@FreeBSD.org 23 of 26
Alternatives
I Used to organize packages with the same functionality(e.g. web-browser)
I May be used to implement virtual dependencies(provides/requires)
Alternatives
Text EditorDepends
Package
Vsevolod Stakhov vsevolod@FreeBSD.org 24 of 26
Existing issues to be solved prior to 1.3
I A solver cannot find install candidates for non-automatictop level packages (those without reverse depends)
I Package upgrade is performed improperly (need torename, install and unlink)
I Minor issues and crashes
Vsevolod Stakhov vsevolod@FreeBSD.org 25 of 26
Questions?
vsevolod@FreeBSD.org
Vsevolod Stakhov vsevolod@FreeBSD.org 26 of 26