launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #04890
[Merge] lp:~jtv/launchpad/bug-832647 into lp:launchpad
Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/bug-832647 into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #832647 in Launchpad itself: "Dominate Debian"
https://bugs.launchpad.net/launchpad/+bug/832647
For more details, see:
https://code.launchpad.net/~jtv/launchpad/bug-832647/+merge/74378
= Summary =
We can't start delete Debian packages that have been removed from Debian's Sources list until Gina, the script we use to import Debian's source packages, learns to perform domination.
Domination is the process of figuring out which versions of a package have been superseded, and by which other version. The superseding version is called the dominant.
== Proposed fix ==
This involves a generalization of the domination process that people have been wanting for a long time. Let me explain.
Given a series of BinaryPackagePublicationHistory or SourcePackagePublicationHistory objects for a single package in a single distroseries, on the same archive and in the same pocket. We'll call them “pubs” for short, partly because it saves so much work and partly because there's something comfortable about working with pubs.
Traditional domination: find the pub for the latest version, using the pubs' creation dates as the tie-breaker. Mark all the rest as superseded by this latest version.
New domination: multiple versions can stay live. Mark the rest superseded by a newer version that stays live, or where no such version exists, mark as deleted instead.
Traditional domination re-implemented: find the pub for the latest version, using creation dates as the tie-breaker. This is the one version that stays live. Pass the pubs and this one version to the new domination code.
Domination in gina: Get a list of versions for the package, as found in the Sources list for the distroseries / archive / pocket. All these stay live. Pass the pubs and this list of current versions to the new domination code.
== Pre-implementation notes ==
Extensive — trust me on this, I still have the dents in my desk — discussions with both Julian and William.
Julian tells me there are no plans for the foreseeable future to use gina for importing any archives that may have binaries, but William says we may need to dust off that part of gina over the next year or so. I've made it as easy as possible to implement binary domination, but didn't do it myself (yet) as it's not essential to a vastly over-extended feature rotation. I left an XXX.
== Implementation details ==
In test_gina you may notice the test_suite boilerplate that's not usually needed any more. In this case it doesn't seem to run any doctests, but it does trigger a test for one of the handlers. I left it intact.
== Tests ==
{{{
./bin/test -vvc lp.soyuz -t gina
./bin/test -vvc archivepublisher -t dominat
}}}
== Demo and Q/A ==
We'll have to make sure that both gina and publish-ftpmaster still work, and that both dominate properly. The new feature is gina doing domination.
= Launchpad lint =
Checking for conflicts and issues in changed files.
Linting changed files:
lib/lp/archivepublisher/domination.py
lib/lp/soyuz/scripts/gina/retire.py
lib/lp/soyuz/model/publishing.py
lib/lp/soyuz/interfaces/publishing.py
scripts/gina.py
lib/lp/archivepublisher/tests/test_dominator.py
lib/lp/soyuz/scripts/tests/test_gina.py
./lib/lp/soyuz/interfaces/publishing.py
381: E261 at least two spaces before inline comment
478: E261 at least two spaces before inline comment
511: E261 at least two spaces before inline comment
681: E261 at least two spaces before inline comment
767: E261 at least two spaces before inline comment
./scripts/gina.py
26: '_pythonpath' imported but unused
--
https://code.launchpad.net/~jtv/launchpad/bug-832647/+merge/74378
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/bug-832647 into lp:launchpad.
=== modified file 'lib/lp/archivepublisher/domination.py'
--- lib/lp/archivepublisher/domination.py 2011-08-30 06:37:55 +0000
+++ lib/lp/archivepublisher/domination.py 2011-09-07 10:08:23 +0000
@@ -53,8 +53,6 @@
__all__ = ['Dominator']
from datetime import timedelta
-import functools
-import operator
import apt_pkg
from storm.expr import (
@@ -68,7 +66,7 @@
flush_database_updates,
sqlvalues,
)
-from canonical.launchpad.interfaces.lpstorm import IMasterStore
+from canonical.launchpad.interfaces.lpstorm import IStore
from lp.registry.model.sourcepackagename import SourcePackageName
from lp.soyuz.enums import (
BinaryPackageFormat,
@@ -87,17 +85,93 @@
apt_pkg.InitSystem()
-def _compare_packages_by_version_and_date(get_release, p1, p2):
- """Compare publications p1 and p2 by their version; using Debian rules.
-
- If the publications are for the same package, compare by datecreated
- instead. This lets newer records win.
- """
- if get_release(p1).id == get_release(p2).id:
- return cmp(p1.datecreated, p2.datecreated)
-
- return apt_pkg.VersionCompare(get_release(p1).version,
- get_release(p2).version)
+def join_spr_spn():
+ """Join condition: SourcePackageRelease/SourcePackageName."""
+ return (
+ SourcePackageName.id == SourcePackageRelease.sourcepackagenameID)
+
+
+def join_spph_spr():
+ """Join condition: SourcePackageRelease/SourcePackagePublishingHistory.
+ """
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return (
+ SourcePackageRelease.id ==
+ SourcePackagePublishingHistory.sourcepackagereleaseID)
+
+
+class SourcePublicationTraits:
+ """Basic generalized attributes for `SourcePackagePublishingHistory`.
+
+ Used by `GeneralizedPublication` to hide the differences from
+ `BinaryPackagePublishingHistory`.
+ """
+ @staticmethod
+ def getPackageName(spph):
+ """Return the name of this publication's source package."""
+ return spph.sourcepackagerelease.sourcepackagename.name
+
+ @staticmethod
+ def getPackageRelease(spph):
+ """Return this publication's `SourcePackageRelease`."""
+ return spph.sourcepackagerelease
+
+
+class BinaryPublicationTraits:
+ """Basic generalized attributes for `BinaryPackagePublishingHistory`.
+
+ Used by `GeneralizedPublication` to hide the differences from
+ `SourcePackagePublishingHistory`.
+ """
+ @staticmethod
+ def getPackageName(bpph):
+ """Return the name of this publication's binary package."""
+ return bpph.binarypackagerelease.binarypackagename.name
+
+ @staticmethod
+ def getPackageRelease(bpph):
+ """Return this publication's `BinaryPackageRelease`."""
+ return bpph.binarypackagerelease
+
+
+class GeneralizedPublication:
+ """Generalize handling of publication records.
+
+ This allows us to write code that can be dealing with either
+ `SourcePackagePublishingHistory`s or `BinaryPackagePublishingHistory`s
+ without caring which. Differences are abstracted away in a traits
+ class.
+ """
+ def __init__(self, is_source=True):
+ if is_source:
+ self.traits = SourcePublicationTraits
+ else:
+ self.traits = BinaryPublicationTraits
+
+ def getPackageName(self, pub):
+ """Get the package's name."""
+ return self.traits.getPackageName(pub)
+
+ def getPackageVersion(self, pub):
+ """Obtain the version string for a publicaiton record."""
+ return self.traits.getPackageRelease(pub).version
+
+ def compare(self, pub1, pub2):
+ """Compare publications by version.
+
+ If both publications are for the same version, their creation dates
+ break the tie.
+ """
+ version_comparison = apt_pkg.VersionCompare(
+ self.getPackageVersion(pub1), self.getPackageVersion(pub2))
+
+ if version_comparison == 0:
+ # Use dates as tie breaker.
+ return cmp(pub1.datecreated, pub2.datecreated)
+ else:
+ return version_comparison
class Dominator:
@@ -116,50 +190,79 @@
self.logger = logger
self.archive = archive
- def _dominatePublications(self, pubs):
+ def dominatePackage(self, publications, live_versions, generalization):
+ """Dominate publications for a single package.
+
+ :param publications: Iterable of publications for the same package,
+ in the same archive, series, and pocket, all with status
+ `PackagePublishingStatus.PUBLISHED`.
+ :param live_versions: Iterable of version strings that are still
+ considered live for this package. The given publications will
+ remain active insofar as they represent any of these versions;
+ the other publications will be marked superseded.
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
+ """
+ # Go through publications from latest version to oldest. This
+ # makes it easy to figure out which release superseded which:
+ # the dominant is always the oldest live release that is newer
+ # than the one being superseded.
+ publications = sorted(
+ publications, cmp=generalization.compare, reverse=True)
+
+ current_dominant = None
+ for pub in publications:
+ if generalization.getPackageVersion(pub) in live_versions:
+ # This publication stays active; if any publications
+ # that follow right after this are to be superseded,
+ # this is the release that they are superseded by.
+ current_dominant = pub
+ elif current_dominant is None:
+ # This publication is no longer live, but there is no
+ # newer version to supersede it either. Therefore it
+ # must be deleted.
+ pub.requestDeletion(None)
+ else:
+ # This publication is superseded. This is what we're
+ # here to do.
+ pub.supersede(current_dominant, logger=self.logger)
+
+ def _dominatePublications(self, pubs, generalization):
"""Perform dominations for the given publications.
:param pubs: A dict mapping names to a list of publications. Every
publication must be PUBLISHED or PENDING, and the first in each
list will be treated as dominant (so should be the latest).
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
"""
self.logger.debug("Dominating packages...")
- for name in pubs.keys():
- assert pubs[name], (
- "Empty list of publications for %s" % name)
- for pubrec in pubs[name][1:]:
- pubrec.supersede(pubs[name][0], logger=self.logger)
+ for name, publications in pubs.iteritems():
+ assert publications, "Empty list of publications for %s." % name
+ latest_version = generalization.getPackageVersion(publications[0])
+ self.dominatePackage(
+ publications, [latest_version], generalization)
- def _sortPackages(self, pkglist, is_source=True):
+ def _sortPackages(self, pkglist, generalization):
"""Map out packages by name, and sort by descending version.
:param pkglist: An iterable of `SourcePackagePublishingHistory` or
`BinaryPackagePublishingHistory`.
- :param is_source: Whether this call involves source package
- publications. If so, work with `SourcePackagePublishingHistory`.
- If not, work with `BinaryPackagepublishingHistory`.
- :return: A dict mapping each package name (as UTF-8 encoded string)
- to a list of publications from `pkglist`, newest first.
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
+ :return: A dict mapping each package name to a list of publications
+ from `pkglist`, newest first.
"""
self.logger.debug("Sorting packages...")
- if is_source:
- get_release = operator.attrgetter("sourcepackagerelease")
- get_name = operator.attrgetter("sourcepackagename")
- else:
- get_release = operator.attrgetter("binarypackagerelease")
- get_name = operator.attrgetter("binarypackagename")
-
outpkgs = {}
for inpkg in pkglist:
- key = get_name(get_release(inpkg)).name.encode('utf-8')
+ key = generalization.getPackageName(inpkg)
outpkgs.setdefault(key, []).append(inpkg)
- sort_order = functools.partial(
- _compare_packages_by_version_and_date, get_release)
for package_pubs in outpkgs.itervalues():
- package_pubs.sort(cmp=sort_order, reverse=True)
+ package_pubs.sort(cmp=generalization.compare, reverse=True)
return outpkgs
@@ -287,6 +390,8 @@
# Avoid circular imports.
from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
+ generalization = GeneralizedPublication(is_source=False)
+
for distroarchseries in distroseries.architectures:
self.logger.debug(
"Performing domination across %s/%s (%s)",
@@ -312,7 +417,7 @@
),
group_by=BinaryPackageName.id,
having=Count(BinaryPackagePublishingHistory.id) > 1)
- binaries = IMasterStore(BinaryPackagePublishingHistory).find(
+ binaries = IStore(BinaryPackagePublishingHistory).find(
BinaryPackagePublishingHistory,
BinaryPackageRelease.id ==
BinaryPackagePublishingHistory.binarypackagereleaseID,
@@ -322,7 +427,21 @@
BinaryPackageFormat.DDEB,
bpph_location_clauses)
self.logger.debug("Dominating binaries...")
- self._dominatePublications(self._sortPackages(binaries, False))
+ self._dominatePublications(
+ self._sortPackages(binaries, generalization), generalization)
+
+ def _composeActiveSourcePubsCondition(self, distroseries, pocket):
+ """Compose ORM condition for restricting relevant source pubs."""
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return And(
+ SourcePackagePublishingHistory.status ==
+ PackagePublishingStatus.PUBLISHED,
+ SourcePackagePublishingHistory.distroseries == distroseries,
+ SourcePackagePublishingHistory.archive == self.archive,
+ SourcePackagePublishingHistory.pocket == pocket,
+ )
def dominateSources(self, distroseries, pocket):
"""Perform domination on source package publications.
@@ -332,38 +451,65 @@
"""
# Avoid circular imports.
from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ generalization = GeneralizedPublication(is_source=True)
+
self.logger.debug(
"Performing domination across %s/%s (Source)",
distroseries.name, pocket.title)
- spph_location_clauses = And(
- SourcePackagePublishingHistory.status ==
- PackagePublishingStatus.PUBLISHED,
- SourcePackagePublishingHistory.distroseries == distroseries,
- SourcePackagePublishingHistory.archive == self.archive,
- SourcePackagePublishingHistory.pocket == pocket,
- )
+
+ spph_location_clauses = self._composeActiveSourcePubsCondition(
+ distroseries, pocket)
+ having_multiple_active_publications = (
+ Count(SourcePackagePublishingHistory.id) > 1)
candidate_source_names = Select(
SourcePackageName.id,
- And(
- SourcePackageRelease.sourcepackagenameID ==
- SourcePackageName.id,
- SourcePackagePublishingHistory.sourcepackagereleaseID ==
- SourcePackageRelease.id,
- spph_location_clauses,
- ),
+ And(join_spph_spr(), join_spr_spn(), spph_location_clauses),
group_by=SourcePackageName.id,
- having=Count(SourcePackagePublishingHistory.id) > 1)
- sources = IMasterStore(SourcePackagePublishingHistory).find(
+ having=having_multiple_active_publications)
+ sources = IStore(SourcePackagePublishingHistory).find(
SourcePackagePublishingHistory,
- SourcePackageRelease.id ==
- SourcePackagePublishingHistory.sourcepackagereleaseID,
+ join_spph_spr(),
SourcePackageRelease.sourcepackagenameID.is_in(
candidate_source_names),
spph_location_clauses)
+
self.logger.debug("Dominating sources...")
- self._dominatePublications(self._sortPackages(sources))
+ self._dominatePublications(
+ self._sortPackages(sources, generalization), generalization)
flush_database_updates()
+ def dominateRemovedSourceVersions(self, distroseries, pocket,
+ package_name, live_versions):
+ """Dominate source publications based on a set of "live" versions.
+
+ Active publications for the "live" versions will remain active. All
+ other active publications for the same package (and the same archive,
+ distroseries, and pocket) are marked superseded.
+
+ Unlike traditional domination, this allows multiple versions of a
+ package to stay active in the same distroseries, archive, and pocket.
+
+ :param distroseries: `DistroSeries` to dominate.
+ :param pocket: `PackagePublishingPocket` to dominate.
+ :param package_name: Source package name, as text.
+ :param live_versions: Iterable of all version strings that are to
+ remain active.
+ """
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ generalization = GeneralizedPublication(is_source=True)
+
+ package_pubs = IStore(SourcePackagePublishingHistory).find(
+ SourcePackagePublishingHistory,
+ join_spph_spr(),
+ join_spr_spn(),
+ SourcePackageName.name == package_name,
+ self._composeActiveSourcePubsCondition(distroseries, pocket))
+
+ self.dominatePackage(package_pubs, live_versions, generalization)
+
def judge(self, distroseries, pocket):
"""Judge superseded sources and binaries."""
# Avoid circular imports.
=== modified file 'lib/lp/archivepublisher/tests/test_dominator.py'
--- lib/lp/archivepublisher/tests/test_dominator.py 2011-02-04 05:11:00 +0000
+++ lib/lp/archivepublisher/tests/test_dominator.py 2011-09-07 10:08:23 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
# GNU Affero General Public License version 3 (see the file LICENSE).
"""Tests for domination.py."""
@@ -7,12 +7,24 @@
import datetime
+import apt_pkg
+from zope.security.proxy import removeSecurityProxy
+
from canonical.database.sqlbase import flush_database_updates
-from lp.archivepublisher.domination import Dominator, STAY_OF_EXECUTION
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.archivepublisher.domination import (
+ Dominator,
+ GeneralizedPublication,
+ STAY_OF_EXECUTION,
+ )
from lp.archivepublisher.publishing import Publisher
+from lp.registry.interfaces.pocket import PackagePublishingPocket
from lp.registry.interfaces.series import SeriesStatus
+from lp.services.log.logger import DevNullLogger
from lp.soyuz.enums import PackagePublishingStatus
+from lp.soyuz.interfaces.publishing import ISourcePackagePublishingHistory
from lp.soyuz.tests.test_publishing import TestNativePublishingBase
+from lp.testing import TestCaseWithFactory
class TestDominator(TestNativePublishingBase):
@@ -50,6 +62,8 @@
foo_10_source, foo_10_binaries[0])
def dominateAndCheck(self, dominant, dominated, supersededby):
+ generalization = GeneralizedPublication(
+ is_source=ISourcePackagePublishingHistory.providedBy(dominant))
dominator = Dominator(self.logger, self.ubuntutest.main_archive)
# The _dominate* test methods require a dictionary where the
@@ -58,7 +72,7 @@
# and dominated, the subsequents.
pubs = {'foo': [dominant, dominated]}
- dominator._dominatePublications(pubs)
+ dominator._dominatePublications(pubs, generalization)
flush_database_updates()
# The dominant version remains correctly published.
@@ -145,7 +159,9 @@
# This isn't a really good exception. It should probably be
# something more indicative of bad input.
self.assertRaises(
- AssertionError, dominator._dominatePublications, pubs)
+ AssertionError,
+ dominator._dominatePublications,
+ pubs, GeneralizedPublication(True))
class TestDomination(TestNativePublishingBase):
@@ -200,3 +216,256 @@
TestDomination.setUp(self)
self.ubuntutest['breezy-autotest'].status = (
SeriesStatus.OBSOLETE)
+
+
+def make_spphs_for_versions(factory, versions):
+ """Create publication records for each of `versions`.
+
+ They records are created in the same order in which they are specified.
+ Make the order irregular to prove that version ordering is not a
+ coincidence of object creation order etc.
+
+ Versions may also be identical; each publication record will still have
+ its own package release.
+ """
+ spn = factory.makeSourcePackageName()
+ distroseries = factory.makeDistroSeries()
+ pocket = factory.getAnyPocket()
+ sprs = [
+ factory.makeSourcePackageRelease(
+ sourcepackagename=spn, version=version)
+ for version in versions]
+ return [
+ factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=pocket,
+ sourcepackagerelease=spr,
+ status=PackagePublishingStatus.PUBLISHED)
+ for spr in sprs]
+
+
+def list_source_versions(spphs):
+ """Extract the versions from `spphs` as a list, in the same order."""
+ return [spph.sourcepackagerelease.version for spph in spphs]
+
+
+class TestGeneralizedPublication(TestCaseWithFactory):
+ """Test publication generalization helpers."""
+
+ layer = ZopelessDatabaseLayer
+
+ def alterCreationDates(self, spphs, ages):
+ """Set `datecreated` on each of `spphs` according to `ages`.
+
+ :param spphs: Iterable of `SourcePackagePublishingHistory`. Their
+ respective creation dates will be offset by the respective ages
+ found in `ages` (with the two being matched up in the same order).
+ :param ages: Iterable of ages. Must provide the same number of items
+ as `spphs`. Ages are `timedelta` objects that will be subtracted
+ from the creation dates on the respective records in `spph`.
+ """
+ for spph, age in zip(spphs, ages):
+ spph.datecreated -= age
+
+ def test_getPackageVersion_gets_source_version(self):
+ spph = self.factory.makeSourcePackagePublishingHistory()
+ self.assertEqual(
+ spph.sourcepackagerelease.version,
+ GeneralizedPublication(is_source=True).getPackageVersion(spph))
+
+ def test_getPackageVersion_gets_binary_version(self):
+ bpph = self.factory.makeBinaryPackagePublishingHistory()
+ self.assertEqual(
+ bpph.binarypackagerelease.version,
+ GeneralizedPublication(is_source=False).getPackageVersion(bpph))
+
+ def test_compare_sorts_versions(self):
+ versions = [
+ '1.1v2',
+ '1.1v1',
+ '1.1v3',
+ ]
+ spphs = make_spphs_for_versions(self.factory, versions)
+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+ self.assertEqual(
+ sorted(versions), list_source_versions(sorted_spphs))
+
+ def test_compare_orders_versions_by_debian_rules(self):
+ versions = [
+ '1.1.0',
+ '1.10',
+ '1.1',
+ '1.1ubuntu0',
+ ]
+ spphs = make_spphs_for_versions(self.factory, versions)
+
+ debian_sorted_versions = sorted(versions, cmp=apt_pkg.VersionCompare)
+
+ # Assumption: in this case, Debian version ordering is not the
+ # same as alphabetical version ordering.
+ self.assertNotEqual(sorted(versions), debian_sorted_versions)
+
+ # The compare method produces the Debian ordering.
+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+ self.assertEqual(
+ sorted(versions, cmp=apt_pkg.VersionCompare),
+ list_source_versions(sorted_spphs))
+
+ def test_compare_breaks_tie_with_creation_date(self):
+ # When two publications are tied for comparison because they are
+ # for the same package release, they are ordered by creation
+ # date.
+ distroseries = self.factory.makeDistroSeries()
+ pocket = self.factory.getAnyPocket()
+ spr = self.factory.makeSourcePackageRelease()
+ ages = [
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ datetime.timedelta(3),
+ ]
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ sourcepackagerelease=spr, distroseries=distroseries,
+ pocket=pocket)
+ for counter in xrange(len(ages))]
+ self.alterCreationDates(spphs, ages)
+
+ self.assertEqual(
+ [spphs[2], spphs[0], spphs[1]],
+ sorted(spphs, cmp=GeneralizedPublication().compare))
+
+ def test_compare_breaks_tie_for_releases_with_same_version(self):
+ # When two publications are tied for comparison because they
+ # belong to releases with the same version string, they are
+ # ordered by creation date.
+ version = "1.%d" % self.factory.getUniqueInteger()
+ ages = [
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ datetime.timedelta(3),
+ ]
+ distroseries = self.factory.makeDistroSeries()
+ pocket = self.factory.getAnyPocket()
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=pocket,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ version=version))
+ for counter in xrange(len(ages))]
+ self.alterCreationDates(spphs, ages)
+
+ self.assertEqual(
+ [spphs[2], spphs[0], spphs[1]],
+ sorted(spphs, cmp=GeneralizedPublication().compare))
+
+
+class TestDominatorMethods(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def makeDominator(self, publications):
+ if len(publications) == 0:
+ archive = self.factory.makeArchive()
+ else:
+ archive = publications[0].archive
+ return Dominator(DevNullLogger(), archive)
+
+ def test_dominatePackage_survives_empty_publications_list(self):
+ # Nothing explodes when dominatePackage is called with an empty
+ # packages list.
+ self.makeDominator([]).dominatePackage(
+ [], [], GeneralizedPublication(True))
+ # The test is that we get here without error.
+ pass
+
+ def test_dominatePackage_leaves_live_version_untouched(self):
+ # dominatePackage does not supersede live versions.
+ [pub] = make_spphs_for_versions(self.factory, ['3.1'])
+ self.makeDominator([pub]).dominatePackage(
+ [pub], ['3.1'], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pub.status)
+
+ def test_dominatePackage_deletes_dead_version_without_successor(self):
+ # dominatePackage marks non-live package versions without
+ # superseding versions as deleted.
+ [pub] = make_spphs_for_versions(self.factory, ['1.1'])
+ self.makeDominator([pub]).dominatePackage(
+ [pub], [], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+ def test_dominatePackage_supersedes_older_pub_with_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage
+ # designates a newer live version as the superseding version.
+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['1.1'], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.SUPERSEDED, pubs[0].status)
+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+ def test_dominatePackage_only_supersedes_with_live_pub(self):
+ # When marking a package as superseded, dominatePackage will
+ # only pick a live version as the superseding one.
+ pubs = make_spphs_for_versions(
+ self.factory, ['1.0', '2.0', '3.0', '4.0'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['3.0'], GeneralizedPublication(True))
+ self.assertEqual([
+ pubs[2].sourcepackagerelease,
+ pubs[2].sourcepackagerelease,
+ None,
+ None,
+ ],
+ [pub.supersededby for pub in pubs])
+
+ def test_dominatePackage_supersedes_with_oldest_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage picks
+ # the oldest of the newer, live versions as the superseding one.
+ pubs = make_spphs_for_versions(self.factory, ['2.7', '2.8', '2.9'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['2.8', '2.9'], GeneralizedPublication(True))
+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+
+ def test_dominatePackage_only_supersedes_with_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage only
+ # considers a newer version as the superseding one.
+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['0.1'], GeneralizedPublication(True))
+ self.assertEqual(None, pubs[1].supersededby)
+ self.assertEqual(PackagePublishingStatus.DELETED, pubs[1].status)
+
+ def test_dominateRemovedSourceVersions_dominates_publications(self):
+ # dominateRemovedSourceVersions finds the publications for a
+ # package and calls dominatePackage on them.
+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2', '0.3'])
+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, package_name, ['0.2'])
+ self.assertEqual([
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ PackagePublishingStatus.DELETED,
+ ],
+ [pub.status for pub in pubs])
+ self.assertEqual(
+ [pubs[1].sourcepackagerelease, None, None],
+ [pub.supersededby for pub in pubs])
+
+ def test_dominateRemovedSourceVersions_ignores_other_pockets(self):
+ # dominateRemovedSourceVersions ignores publications in other
+ # pockets than the one specified.
+ pubs = make_spphs_for_versions(self.factory, ['2.3', '2.4'])
+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+ removeSecurityProxy(pubs[0]).pocket = PackagePublishingPocket.UPDATES
+ removeSecurityProxy(pubs[1]).pocket = PackagePublishingPocket.PROPOSED
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, package_name, ['2.3'])
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+ def test_dominateRemovedSourceVersions_ignores_other_packages(self):
+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+ other_package_name = self.factory.makeSourcePackageName().name
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, other_package_name, ['1.1'])
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[0].status)
=== modified file 'lib/lp/soyuz/interfaces/publishing.py'
--- lib/lp/soyuz/interfaces/publishing.py 2011-09-02 04:51:25 +0000
+++ lib/lp/soyuz/interfaces/publishing.py 2011-09-07 10:08:23 +0000
@@ -195,9 +195,6 @@
the field name and value is the value string.
"""
- def supersede():
- """Supersede this publication."""
-
def requestObsolescence():
"""Make this publication obsolete.
=== modified file 'lib/lp/soyuz/model/publishing.py'
--- lib/lp/soyuz/model/publishing.py 2011-08-31 04:40:44 +0000
+++ lib/lp/soyuz/model/publishing.py 2011-09-07 10:08:23 +0000
@@ -327,8 +327,8 @@
fields = self.buildIndexStanzaFields()
return fields.makeOutput()
- def supersede(self):
- """See `IPublishing`."""
+ def setSuperseded(self):
+ """Set to SUPERSEDED status."""
self.status = PackagePublishingStatus.SUPERSEDED
self.datesuperseded = UTC_NOW
@@ -742,7 +742,7 @@
"Should not dominate unpublished source %s" %
self.sourcepackagerelease.title)
- super(SourcePackagePublishingHistory, self).supersede()
+ self.setSuperseded()
if dominant is not None:
if logger is not None:
@@ -1126,7 +1126,7 @@
self.distroarchseries.architecturetag))
return
- super(BinaryPackagePublishingHistory, self).supersede()
+ self.setSuperseded()
if dominant is not None:
# DDEBs cannot themselves be dominant; they are always dominated
=== added file 'lib/lp/soyuz/scripts/gina/retire.py'
--- lib/lp/soyuz/scripts/gina/retire.py 1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/scripts/gina/retire.py 2011-09-07 10:08:23 +0000
@@ -0,0 +1,27 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Retirement of packages that are removed upstream."""
+
+__metaclass__ = type
+__all__ = [
+ 'dominate_imported_source_packages',
+ ]
+
+from zope.component import getUtility
+
+from lp.archivepublisher.domination import Dominator
+from lp.registry.interfaces.distribution import IDistributionSet
+
+
+def dominate_imported_source_packages(logger, distro_name, series_name,
+ pocket, packages_map):
+ """Perform domination."""
+ series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
+ dominator = Dominator(logger, series.main_archive)
+ for package_name, entries in packages_map.src_map.iteritems():
+ live_versions = [
+ entry['Version']
+ for entry in entries if 'Version' in entry]
+ dominator.dominateRemovedSourceVersions(
+ series, pocket, package_name, live_versions)
=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
--- lib/lp/soyuz/scripts/tests/test_gina.py 2010-08-20 20:31:18 +0000
+++ lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-07 10:08:23 +0000
@@ -1,13 +1,41 @@
-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
# GNU Affero General Public License version 3 (see the file LICENSE).
from doctest import DocTestSuite
-import unittest
+from unittest import TestLoader
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.services.log.logger import DevNullLogger
+from lp.soyuz.enums import PackagePublishingStatus
import lp.soyuz.scripts.gina.handlers
+from lp.soyuz.scripts.gina.retire import dominate_imported_source_packages
+from lp.testing import TestCaseWithFactory
+
+
+class TestGina(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def test_dominate_imported_source_packages(self):
+
+ class SimpleFakePackagesMap:
+ def __init__(self, src_map):
+ self.src_map = src_map
+
+ logger = DevNullLogger()
+ pub = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ series = pub.distroseries
+ spr = pub.sourcepackagerelease
+ package = spr.sourcepackagename
+ packages_map = SimpleFakePackagesMap({package.name: []})
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, pub.pocket,
+ packages_map)
+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
def test_suite():
- suite = unittest.TestSuite()
+ suite = TestLoader().loadTestsFromName(__name__)
suite.addTest(DocTestSuite(lp.soyuz.scripts.gina.handlers))
return suite
=== modified file 'scripts/gina.py'
--- scripts/gina.py 2011-08-23 08:35:13 +0000
+++ scripts/gina.py 2011-09-07 10:08:23 +0000
@@ -53,6 +53,7 @@
PoolFileNotFound,
SourcePackageData,
)
+from lp.soyuz.scripts.gina.retire import dominate_imported_source_packages
# Set to non-zero if you'd like to be warned every so often
COUNTDOWN = 0
@@ -152,6 +153,10 @@
packages_map, kdb, package_root, keyrings, importer_handler)
importer_handler.commit()
+ # XXX JeroenVermeulen 2011-09-07 bug=843728: Dominate binaries as well.
+ dominate_imported_source_packages(
+ log, distro, distroseries, pocket, packages_map)
+
if source_only:
log.info('Source only mode... done')
return
@@ -209,9 +214,8 @@
npacks = len(packages_map.src_map)
log.info('%i Source Packages to be imported', npacks)
- for list_source in sorted(
- packages_map.src_map.values(), key=lambda x: x[0].get("Package")):
- for source in list_source:
+ for package in sorted(packages_map.src_map.iterkeys()):
+ for source in packages_map.src_map[package]:
count += 1
attempt_source_package_import(
source, kdb, package_root, keyrings, importer_handler)
@@ -244,10 +248,9 @@
log.info(
'%i Binary Packages to be imported for %s', npacks, archtag)
# Go over binarypackages importing them for this architecture
- for binary in sorted(packages_map.bin_map[archtag].values(),
- key=lambda x: x.get("Package")):
+ for package_name in sorted(packages_map.bin_map[archtag].iterkeys()):
+ binary = packages_map.bin_map[archtag][package_name]
count += 1
- package_name = binary.get("Package", "unknown")
try:
try:
do_one_binarypackage(binary, archtag, kdb, package_root,
