Archive management plans for phone RTM
Colin Watson
cjwatson at ubuntu.com
Wed Jun 4 20:20:50 UTC 2014
The Ubuntu phone images are due to hit the release-to-manufacturing
(RTM) point later this cycle. With the pace of the phone work, it
doesn't look practical to deal with this by SRUing all the required
changes into trusty - we're talking about substantial feature
development which probably wouldn't be manageable in trusty even with a
rather liberal SRU policy for phone-specific components.
I've therefore been thinking about how we might manage to do a stable
phone image somewhere in the middle of a development release cycle;
while the extensive work we've done on rolling quality helps a lot, it
probably won't quite be enough on its own, and we don't want to have the
phone work interfere too much with other Ubuntu developers, so we need
some reasonable way to create a stable branch for RTM, preferably
without just hiding everything off in some non-Launchpad repository that
would require lots of work in our CI tools to handle. I spent some time
in Malta this week discussing this with William Grant of Launchpad and
various phone managers and developers, and I think we now have a
solution that meets the requirements, is technically sound, and might
potentially be useful elsewhere in the future.
= PPA strategy (shelved) =
The first idea I had was to copy everything relevant to the phone image
(probably a full build-dependency closure) into a PPA; create a second
PPA which could act as an equivalent of -proposed; adjust Launchpad and
launchpad-buildd, including the new live filesystem building code, to
allow this PPA to be entirely self-contained, so that builds don't refer
to the primary archive; and run whatever syncing and merging tools we
need to propagate changes from the primary archive to the RTM PPAs as
required.
This very nearly works. There are some publisher performance problems
with putting a very large number of packages in a single PPA, but
William would have been happy to sort those out. However, a more
serious problem is that chroots are shared between utopic in the primary
archive and utopic PPAs, and there's currently no way to disconnect
those, so over time it would become more difficult to build updated
packages for this PPA. While we might be able to fix this, and doing so
would be a workable plan B, it points to PPAs being the wrong model.
= Derived distribution strategy (candidate) =
A feature called "derived distributions" was added to Launchpad a few
years ago, originally for use by Linaro: it allows branching an entire
distribution or part of it into a new distribution, with some tools to
help keep them in sync where appropriate. It has never actually been
used properly in production; it was tried once, ran into some small but
fatal problems, but staffing issues at the time meant that these were
never fixed. William has given it a fresh try on a non-production
Launchpad instance and it doesn't seem to be too far off working
reasonably, so we think that we can salvage this work.
The usual Launchpad copying tools will still work much as before, and
we'll be able to have PPAs based on the new derived distribution.
The current plan is therefore:
* CI and Foundations team members will spend the next two months making
sure that all the tools we need are in place. This includes fixing
up derived distributions, preparing an initialisation script, making
sure that the CI engine (train or airline, as appropriate) can handle
building for a derived distribution, and adding support for
ubuntu-touch image builds from derived distributions.
* Monitor progress of phone images closely in the early part of August.
We should have a beta image around this point.
* Rick has asked me to take ownership of deciding when to branch for
RTM; note that this decision can be taken retroactively by copying
older versions of packages, so we can say "the image from three days
ago looked good but things have been broken since then, so let's
branch from that". I intend to take this decision based on how
stable images are and how much tension there is between phone work
and development of other parts of Ubuntu; all other things being
equal I would prefer to keep the branch shorter, but we will need at
least a couple of weeks' clearance before RTM in order that we aren't
debugging our infrastructure right up to the wire. Since I have
pre-booked holiday in mid-August, William Grant will be my backup for
this.
* As with the shelved PPA plan, we'll only copy packages that are
needed to build the phone image. This makes it easier to track
things and keeps our use of resources down, which has benefits such
as faster publishing.
* We will then have an "ubuntu-phone" (name TBD) distribution in
Launchpad with a series named something like "rtm-14.09", and will
land changes for RTM there, allowing other parts of Ubuntu to move on
without risk of breaking phone images. We'll have a -proposed pocket
as usual and proposed-migration will operate on it. I'd like
permissions on this distribution to be identical to those in Ubuntu
where possible, but we can discuss details there. There'll be
system-image channels for RTM, probably per-device.
* We don't want to have to maintain RTM series for a long time. The
plan is to move devices to the next stable branch along as soon as
possible. To facilitate this, the usual rule should be that changes
should only go into this distribution after they're already on their
way into mainline Ubuntu (as for SRUs), to make sure that we don't
end up in a situation where developers land changes on the stable
branch but don't quite get round to landing them in mainline.
= FAQ =
Can we use this strategy for other Ubuntu flavours or derivatives?
In theory, maybe. It's very much easier to use this strategy for
phone images, though, because the standard distribution mechanism is
via system-image channels so we don't need to worry about updating
sources.list or arranging for mirroring of the derived archive, etc.
There's also a potentially serious social cost to having lots of
archive branches, and I wouldn't want to do so except for short
durations under tight management; I don't want to see the
Ubuntu-and-flavours development community fracturing unnecessarily.
That said, the raw technology may prove to be useful in other similar
contexts in the future. Ask us after we've had some experience with
running it for RTM ...
What if derived distributions can't be made to work after all?
We fall back to PPAs and keep a really close eye on everything.
Fortunately this is an unlikely scenario, I think, as this will
probably end up being more work.
How will we manage syncing and merging?
The nature of the RTM branch is such that we probably won't need very
much in the way of keeping up to date with Ubuntu. Most changes will
land via the CI engine. That said, there will doubtless be some
things like security updates to core packages that we need to land, so
time permitting I'd like to set up something like merge-o-matic to
keep track of this. If possible, these will be copied via the CI
engine to minimise the risk of unexpected image testing failures.
Will non-Canonical developers be able to take part in this?
The current engine ("CI Train") is visible to non-Canonical
developers, but the use of a spreadsheet, a private Jenkins instance,
and semi-manual organisation of landings makes it difficult to use in
practice. The new engine (sometimes called "CI Airline") is due to
land this month and should be much easier to use without
company-specific privileges. I see no reason why it can't all be
open.
We'll be trying hard to have a common rootfs across customers, but
it's possible that deadlines will force there to be a couple of
patches which land separately for one or the other. Where possible,
though, this kind of thing will be in the device or custom tarball
rather than the main rootfs.
Any other questions?
--
Colin Watson [cjwatson at ubuntu.com]
More information about the Ubuntu-release
mailing list
