Leveraging Solver Preferences to Tame your Package Manager
-
Upload
roberto-di-cosmo -
Category
Technology
-
view
377 -
download
3
description
Transcript of Leveraging Solver Preferences to Tame your Package Manager
Leveraging solver preferencesto tame your package managers
Roberto Di Cosmohttp://www.dicosmo.org
Universite Paris Diderot and INRIA, Irillhttp://www.irill.org
July 8th, 2014OUPSParis
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 1 / 28
Package managers are all around us
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 2 / 28
Package managers : the basics
Definition of Package Manager (Wikipedia)
... tools to automate ... installing, upgrading, configuring, and removingsoftware packages ... in a consistent manner. It typically maintains adatabase of software dependencies and version information ...
Some package managers
Binary distributions apt, aptitude, yum,...
Source distributions portage, *BSD ports, homebrew, opam...
Language specific PyPI, Eclipse P2, (opam), ...
Application specific steam, ...
Functional approach nixOS, disnixOS,...
... you name it
Do they perform well?
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 3 / 28
Installations may really go wrong
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 4 / 28
Installation woes: no solution
# sudo apt-get install baobab
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
gnome-settings-daemon: Breaks: gnome-screensaver (< 2.28.0)
but 2.26.1-1 is to be installed
E: Broken packages
(whereas a solution did exist)
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 5 / 28
Installation woes: solver timeouthttps://github.com/ocaml/opam/issues/685
(whereas a solution could be found in less than 1/10th of a second)Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 6 / 28
Managing packages is hard
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 7 / 28
How hard are package installation problems?
Theorem
The following problems are NP-complete:
installability of a single package
co-installability of a set of packages
Proof ideaBi-directional mapping between dependency resolution and booleansatisfiability. Details:
Di Cosmo, Leroy, Treinen, Vouillon et al. Managing the complexity oflarge free and open source package-based software distributions. ASE2006: Automated Software Engineering.
Alternative proof of NP-hardness (Daniel Burrows, 2008)
Encode Sudoku solving as a package installation problem, left as exercisefor the public.
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 8 / 28
Finding a needle in a haystack
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 9 / 28
An exponential number of solutions
Too many upgrade candidates
Suppose we have components qi , for 1 ≤ i ≤ n, available in versions 1 and2, all of which are installed in version 1 on the system.We want to install a component p in version 1 that depends on all ofq1, . . . , qn, any version.Any of the 2n configurations {(p, 1)} ∪ {(qi , i)|i ∈ 1 . . . n, 1 ≤ i ≤ 2} is asolution.
Which one do we choose?
paranoid only install p
trendy install p and step up all qi ’s to version 2
... and exponentially many in between!
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 10 / 28
An exponential number of solutions???
The sidestep approach
centralize and group : patch tuesday, service pack, ...
isolate on a monolith : AppStore(s), ...
coexist : NixOS, backward compatibility policy, no conflicts policy,...
The delegate to others approach
identify “suites” of components (e.g. stable, testing, unstable), let theuser order them, and then try to stick to this order
Advantage tries to align the system with a chosen suite
Disadvantage depends on the quality of the most recent state: notassured for development versions, or when mixingrepositoriesdifficult and cumbersome to obtain a different behaviourwhen things go wrong, we may really get lost
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 11 / 28
Installation woes: debatable solution
# sudo apt-get install debhelperReading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
armagetron armagetron-common autoconf bonobo-activation codebreaker debconf
debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig
frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data
libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0
libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0
...
The following packages will be REMOVED:
autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev
libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4
libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common
libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl
libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev
libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev
x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev
xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader
The following NEW packages will be installed:
armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian
libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl
libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6
libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl
libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1
libplrpc-perl libsdl-console ...
75 packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded.
Need to get 67.1MB of archives. After unpacking 26.9MB will be used.
Do you want to continue? [Y/n] Abort.
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 12 / 28
Separation of concerns is the key
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 13 / 28
Package management reloaded
There are several distinct tasks that need to be performed in a packagemanagement system:
Server side
Maintain a coherent set of packages when we add, remove, updatepackages that are made available for the users... left for another talk
Client side
fetch and autenticate metadata and packages
user preferences ⇒specify the kind of solution we want
dependency solver ⇒find a solution
deploy the chosen solution
Factoring out common work
User preferences and dependency solving are really common to all.
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 14 / 28
Separation of concerns: solverDependency solving is NP-hard: stop coding a petty solver for every newcomponent based system, and adopt a modular approach!
Common Upgrade Description Format
Modular component manager
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 15 / 28
An excerpt from a CUDF file
preamble:
property: opam-version: string, opam-name: string
package: herelib
version: 3
depends: ocamlfind
conflicts: herelib
opam-name: herelib
opam-version: 109.12.00
...
package: lwt
version: 6
depends: base-threads , base-unix , camlp4 , ocamlfind
conflicts: react >= 3 , lwt
opam-name: lwt
opam-version: 2.4.4
...
request: opam
install: tyxml
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 16 / 28
Available external solvers
Three external CUDF solvers packaged in Debian
$apt-cache search cudf
aspcud - CUDF solver based on Answer Set Programming
mccs - multi-critera CUDF solver
packup - CUDF solver based on pseudo-Boolean constraints
There is also a nice solver for Java addicts
p2cudf, based on the Eclipse P2 plugin dependency resolver, availablefrom http://wiki.eclipse.org/Equinox/p2/CUDFResolver
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 17 / 28
Talking the external solver into doing what you want
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 18 / 28
Separation of concerns: user preferences
Combinators from the latest Mancoosi preference language1
count(X) : number of packages in X
sum(X,f) : sum of values of key f over packages in X
aligned(X,g1,g2) : number of packages aligned according to given criteria
notuptodate(X) : number of packages in X not current
where X can be solution, down, up, removed, new, changed,request,installrequest,upgraderequest
Combining preferences
We can ask for a solution that maximizes (+) or minimizes (-) each ofthese criteria, and combine them in lexicographic order, e.g.
-count(removed),-count(changed)
1For details, see see http://www.mancoosi.org/misc-2012/criteria/ andhttp://opam.ocaml.org/doc/Specifying_Solver_Preferences.html
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 19 / 28
Examples preferences when installing your packages
Simple
paranoid -count(removed),-count(changed)
trendy -count(removed),-notuptodate(solution),-count(new)
Gourmet
paranoid-count(removed),-notuptodate(request),-count(down),-count(changed)
trendy-notuptodate(request),-count(removed),-count(down),-count(changed)
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 20 / 28
More exotic examples
Building systems
Minimal system size -sum(solution,installedsize),-count(solution)
Noah’s ark +count(solution)
Noah’s ark, fresh -notuptodate(solution),+count(solution)
Performing upgrades
upgrade -count(down),-count(removed),-notuptodate(solution),-count(new)
priority -count(down),-count(removed),-notuptodate(solution),
+sum(solution,priority),-count(new)
Repairing a broken system configuration
Use an empty request with
fixup simple -count(changed)
fixup trendy -count(changed),-count(down),-notuptodate(solution)
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 21 / 28
Ok, how can I use all this in opam?
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 22 / 28
External solvers in opam
Thanks to the work done by the Mancoosi team at Irill and OCamlPro, inthe DORM project, aspcud is supported out of the box in opam since1.0,and all others in trunk: typing opam --help shows
...
OPTIONS
--criteria=CRITERIA
Specify user preferences for dependency solving for this run.
Overrides both $OPAMCRITERIA and $OPAMUPGRADECRITERIA.
For details on the supported language, see
http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html.
The default value is -removed,-notuptodate,-changed for upgrades,
and -removed,-changed,-notuptodate otherwise.
--cudf=FILENAME
Debug option: Save the CUDF requests sent to the solver to
FILENAME-<n>.cudf.
--solver=CMD
Specify the name of the external dependency solver. The default
value is aspcud
...
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 23 / 28
Examples (when aspcud is present)
Opam built-in defaultsopam install ...
--criteria="-count(removed),-count(changed),-notuptodate(solution)"
opam ugrade --criteria="-count(removed),-noutuptodate(solution),-count(changed)"
More advanced preferencesopam install ... --criteria="-count(removed),-count(down),
-notuptodate(request),-notuptodate(changed),-count(changed)"
opam upgrade --criteria="-count(removed),-count(down),-notuptodate(solution),
-count(new),+count(up)"
Repair (now in trunk as opam upgrade --fixup)opam upgrade --criteria="-count(changed)"
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 24 / 28
Getting your external solver
Debian/Ubuntu user?
Lucky guy! Just apt-get install aspcud, and you are done
No aspcud for you? Low-power arch (arduino, raspberri-pi)?
You are still safe!
go to http://cudf-solvers.irill.org
follow the instructions
access the Irill CUDF solver farm
get your solving done in the cloud
Big thanks to OCamlPro folks (Benjamin,Fabrice,Gregoire,Pierre), for helpsetting this up.
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 25 / 28
CreditsThe Mancoosi team in Paris Diderot and Irill
Pietro Abate Ralf Treinen
Jerome Vouillon Stefano Zacchiroli
libcudf CUDF manipulation library: opam show cudf
dose Mancoosi toolbox: opam show dose
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 26 / 28
ConclusionsPackage managers are complex: a very hard part is dependency solving!Modern package managers must share common components, in particular
dependency solvers user preference language
You can should use external solvers and preferences in opam today!
Learn more at www.mancoosi.org and www.irill.org
Di Cosmo, Leroy, Treinen, Vouillon et al Managing the complexity of large FOSS package-based software distributions.
ASE 2006
Abate, Treinen Mancoosi Deliverable D5.4: Report on the international competition
http://hal.inria.fr/hal-00698967/PDF/d5.4.pdf, 2011
Abate, Di Cosmo, Treinen, Zacchiroli Dependency solving: a separate concern in component evolution management.
Journal of Systems and Software, 2012.
Abate, Di Cosmo, Treinen, Zacchiroli A modular package manager architecture. Information and Software Technology,
2013.
Full details at http://www.dicosmo.org/Articles/usercriteria.pdf
Questions?Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 27 / 28
Abbreviations/old combinators
The combinators from the old preference language are still recognised byaspcud:
Old language New language
removed count(removed)new count(new)
changed count(changed)
nutuptodate notuptodate(solution)
Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 28 / 28