--- Begin Message ---
There've been many discussions of late about the challenges of scaling
app installation to a very large number of apps, including making app
packages much easier to automatically audit and sandbox, and making them
safer and quicker to install without the possibility of system-level
breakage. This is needed to make it quicker and easier for app
developers to get their software into Ubuntu. There are several pieces
to this (including a good deal of prior discussion in
https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that
one of those is some kind of simplified packaging format which we can
offer to third-party application developers.
This is not aimed at changing packages that are already part of the
Ubuntu archive; for the most part our existing system works well for
those, and they tend to have non-trivial dependency structures. We'll
continue to use dpkg and apt for building the Ubuntu operating system,
syncing with Debian, and so on. There's no point developing a packaging
system for apps and making it have the full panoply of features needed
for the Ubuntu archive: it'd just be second-system-effect on top of our
current packaging system. So the scope of what I've been considering is
purely leaf apps built on a fixed "base system", which in the case of
the initial target of the Ubuntu phone/tablet work would be the run-time
part of the Ubuntu SDK. The radically-reduced dependency structure
means that most apps will be Ubuntu-SDK-specific to start with, although
I'd like to make sure that the package format design includes enough
support up-front to allow this to be useful for other platforms that
define suitable base system profiles in future.
So, at Steve Langasek's request, I've been putting together a proof of
concept of a low-level app package installer and packaging format.
Highlights of what it can do so far are:
* no dependencies between apps; single implicit dependency on the base
system by way of a Click-Base-System field
* installs each app to an entirely separate directory
* entirely declarative: maintainer scripts are forbidden
* base package manager overhead, i.e. the time required to install a
trivial package containing a single small file, is about 0.15 seconds
on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's
with the current prototype implementation in Python; a later
implementation could be in C and would then be faster still)
* not limited to installing as root, although there may be similar
constraints elsewhere to ensure that apps can't edit their own code
at run-time
* packages built by feeding the intended output directory tree to a
simple Python tool, plus a manifest.json file
* building packages requires only the Python standard library, with the
intent that it should be possible to build these packages quite
easily on non-Ubuntu or even non-Linux systems
* binary packaging format sufficiently similar to existing one that we
could add support to higher-level tools with minimal effort
* strawman design for hooks into system packages, which will be
entirely declarative from the app's point of view
* unit-tested from the start
Obvious items I still need to work on:
* produce a strawman hooks implementation with some real worked
examples
* integrate (or demonstrate how to integrate) the container isolation
properties worked on elsewhere
* Click-Base-System field is skeletally simple right now, and may need
to be expanded to at least leave open the possibility of multiple
flavours of base system (see also GNOME's profiles idea)
* adjust unpack handling to avoid problems with project renames and
name clashes, and to unpack each version into its own directory and
flip symlinks to allow for multi-user independence
* integrate into the Ubuntu SDK, as well as providing examples of how
it can be integrated into other build systems too
To do, but not in this project:
* package acquisition
* indexing support and other frontend things (i.e. the app store)
* upload handling
So far I've demonstrated to my own satisfaction that we can get
acceptable performance and isolation characteristics with very little
work; the installer part on its own is under 300 lines of Python right
now, which I consider useful since bugginess is often proportional to
code size. I should have the prototype ready for people to look at in
time for UDS next week, and I'll ensure that there's a session scheduled
for this.
Is there anything else people can think of that a system like this needs
to consider?
Footnote: Why not use one of the existing systems out there?
We still might, but at present it's not clear that it would make a whole
lot of difference.
Many of the existing app packaging systems are more of a reflection of
the system they were built for than anything else. If you look at, say,
Android's APK format, it's essentially a zip file with a manifest and
some conventions about Dalvik class installation and the like. Most of
the other mobile app formats are similar. Things like Listaller might
be a reasonable fit, but I'm worried about importing things like a full
dependency solver into this: much though I love dependency-based
packaging systems, they necessarily involve scanning the system
packaging database at some point and I would much prefer app packages to
be as independent of that as possible, mainly for performance but also
to be as bulletproof as possible. GNOME app bundles appear to have a
rather different sandboxing model than we need (e.g. enforced
restrictions on dependencies), we probably want better system
integration, and perhaps other differences; but that said I like the
coarse-grained profile model of dependencies (and unknowingly imitated
it), and it might make sense to share some profile names. Barry spent
some time looking at 0install, and it wasn't too bad a fit but we would
still need to solve many of the same system integration problems.
The proof of concept I wrote also isn't entirely new code. It's tiny
due to using .deb as a container format (minus maintainer scripts, full
dependencies, etc.), so I get to save effort by using dpkg to unpack
things, which leaves us room to selectively use more of its features in
future if we want to. This is quite a compelling approach to me for
various reasons. I could be persuaded to look at something else, but I
don't think there's actually much code to share at the packaging format
level anyway, as it's so simple; quite possibly more at the sandboxing
level, but that's really a separate layer.
Furthermore, the priority for this system at present is for Ubuntu
phone/tablet app packages, which haven't actually been built yet. As a
result, I'm less worried about fragmentation than I might otherwise be.
I would like to make sure that it's usable elsewhere too, although
somebody would need to work out what that means in terms of base system
profiles.
Although I'm very familiar with dpkg and traditional package management,
I only started looking at this side of things a couple of weeks ago, so
I'm sure I'm behind on research. Pointers welcome.
Thanks,
--
Colin Watson [cjwatson@xxxxxxxxxx]
--
ubuntu-devel mailing list
ubuntu-devel@xxxxxxxxxxxxxxxx
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
--- End Message ---
