launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #04897
[Merge] lp:~jtv/launchpad/re-830890 into lp:launchpad
Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/re-830890 into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #830890 in Launchpad itself: "Gina should delete publications if removed from source archive"
https://bugs.launchpad.net/launchpad/+bug/830890
For more details, see:
https://code.launchpad.net/~jtv/launchpad/re-830890/+merge/74551
= Summary =
We want Gina to run full, proper, new-style domination on packages that are no longer found in the Sources lists it imports. This will cause any published versions of these packages in our archive to be marked Deleted.
There is a legacy problem, however.
Gina has never done any domination in the past. This results in lots and lots of Debian source package publications in our database being marked Pending. The domination we have in mind as the permanent mechanism would send all of those straight to Deleted, as if all those versions were kept in Debian Sources lists until they were all suddenly deleted one day.
== Proposed fix ==
Transitional domination: for packages that aren't in the Sources list at all, pretend that the latest version we have in the database is still in the Sources list. This will cause all its older releases to be marked as superseded by the newer one. The newer one will stay active. Then, once this is done for the legacy cases, we can switch the code over to the permanet, deleting style of domination.
== Pre-implementation notes ==
Discussed in detail with William. There are a few points to pay attention to:
* This deals with source publications only. It'll be easier to extend this to binary publications (if needed) once we have the legacy out of the way. We do not import binaries for Debian so the legacy issue does not pertain there.
* You won't see extensive superseded-by chains for deleted packages. The last known version will be considered to have superseded all previously existing versions of the deleted package.
* Only Published publications will be dominated. We still need to update Gina to produce Published publications rather than Pending ones (which was never very appropriate), and we still need to update the existing publications in the database.
== Implementation details ==
The loop that used to go over all packages in the Sources list (thus skipping the deleted packages) now iterates over all published packages in the relevant context. The special case for transitional domination is expressed as a special input to the regular domination method. Once we retire transitional domination, the special case can go and the regular domination method will do the right thing. The “entries = packages_map.src_map.get(package_name)” will become “entries = packages_map.src_map.get(package_name, [])” and removal of the transitional code follows from there.
== Tests ==
{{{
./bin/test -vvc lp.archivepublisher -t dominat
./bin/test -vvc lp.soyuz -t gina
}}}
== Demo and Q/A ==
The publish-ftpmaster script should still dominate packages as before.
Gina is harder. Update SourcePackagePublishingHistory statuses in a test version of Debian from Pending to Published. Gina should now dominate packages, meaning that all obsolete publications will be in Superseded status — except for the last known revision of any package that is no longer in the imported Sources lists.
You could also try creating a Published SPPH for Debian, with a fake and an unrealistically high version number. Gina should now mark it as Deleted.
= Launchpad lint =
Some pre-existing lint remains, but I didn't add any. Actually I didn't even touch the files it's in in this branch; that was all done in a branch that this one depends on (and again, all pre-existing, better state than I found it in etc.).
Checking for conflicts and issues in changed files.
Linting changed files:
lib/lp/archivepublisher/domination.py
lib/lp/soyuz/scripts/gina/retire.py
lib/lp/soyuz/model/publishing.py
lib/lp/soyuz/interfaces/publishing.py
scripts/gina.py
lib/lp/archivepublisher/tests/test_dominator.py
lib/lp/soyuz/scripts/tests/test_gina.py
./lib/lp/soyuz/interfaces/publishing.py
381: E261 at least two spaces before inline comment
478: E261 at least two spaces before inline comment
511: E261 at least two spaces before inline comment
681: E261 at least two spaces before inline comment
767: E261 at least two spaces before inline comment
./scripts/gina.py
26: '_pythonpath' imported but unused
--
https://code.launchpad.net/~jtv/launchpad/re-830890/+merge/74551
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/re-830890 into lp:launchpad.
=== modified file 'lib/lp/archivepublisher/domination.py'
--- lib/lp/archivepublisher/domination.py 2011-08-30 06:37:55 +0000
+++ lib/lp/archivepublisher/domination.py 2011-09-08 07:38:24 +0000
@@ -53,8 +53,6 @@
__all__ = ['Dominator']
from datetime import timedelta
-import functools
-import operator
import apt_pkg
from storm.expr import (
@@ -68,7 +66,7 @@
flush_database_updates,
sqlvalues,
)
-from canonical.launchpad.interfaces.lpstorm import IMasterStore
+from canonical.launchpad.interfaces.lpstorm import IStore
from lp.registry.model.sourcepackagename import SourcePackageName
from lp.soyuz.enums import (
BinaryPackageFormat,
@@ -87,17 +85,93 @@
apt_pkg.InitSystem()
-def _compare_packages_by_version_and_date(get_release, p1, p2):
- """Compare publications p1 and p2 by their version; using Debian rules.
-
- If the publications are for the same package, compare by datecreated
- instead. This lets newer records win.
- """
- if get_release(p1).id == get_release(p2).id:
- return cmp(p1.datecreated, p2.datecreated)
-
- return apt_pkg.VersionCompare(get_release(p1).version,
- get_release(p2).version)
+def join_spr_spn():
+ """Join condition: SourcePackageRelease/SourcePackageName."""
+ return (
+ SourcePackageName.id == SourcePackageRelease.sourcepackagenameID)
+
+
+def join_spph_spr():
+ """Join condition: SourcePackageRelease/SourcePackagePublishingHistory.
+ """
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return (
+ SourcePackageRelease.id ==
+ SourcePackagePublishingHistory.sourcepackagereleaseID)
+
+
+class SourcePublicationTraits:
+ """Basic generalized attributes for `SourcePackagePublishingHistory`.
+
+ Used by `GeneralizedPublication` to hide the differences from
+ `BinaryPackagePublishingHistory`.
+ """
+ @staticmethod
+ def getPackageName(spph):
+ """Return the name of this publication's source package."""
+ return spph.sourcepackagerelease.sourcepackagename.name
+
+ @staticmethod
+ def getPackageRelease(spph):
+ """Return this publication's `SourcePackageRelease`."""
+ return spph.sourcepackagerelease
+
+
+class BinaryPublicationTraits:
+ """Basic generalized attributes for `BinaryPackagePublishingHistory`.
+
+ Used by `GeneralizedPublication` to hide the differences from
+ `SourcePackagePublishingHistory`.
+ """
+ @staticmethod
+ def getPackageName(bpph):
+ """Return the name of this publication's binary package."""
+ return bpph.binarypackagerelease.binarypackagename.name
+
+ @staticmethod
+ def getPackageRelease(bpph):
+ """Return this publication's `BinaryPackageRelease`."""
+ return bpph.binarypackagerelease
+
+
+class GeneralizedPublication:
+ """Generalize handling of publication records.
+
+ This allows us to write code that can be dealing with either
+ `SourcePackagePublishingHistory`s or `BinaryPackagePublishingHistory`s
+ without caring which. Differences are abstracted away in a traits
+ class.
+ """
+ def __init__(self, is_source=True):
+ if is_source:
+ self.traits = SourcePublicationTraits
+ else:
+ self.traits = BinaryPublicationTraits
+
+ def getPackageName(self, pub):
+ """Get the package's name."""
+ return self.traits.getPackageName(pub)
+
+ def getPackageVersion(self, pub):
+ """Obtain the version string for a publicaiton record."""
+ return self.traits.getPackageRelease(pub).version
+
+ def compare(self, pub1, pub2):
+ """Compare publications by version.
+
+ If both publications are for the same version, their creation dates
+ break the tie.
+ """
+ version_comparison = apt_pkg.VersionCompare(
+ self.getPackageVersion(pub1), self.getPackageVersion(pub2))
+
+ if version_comparison == 0:
+ # Use dates as tie breaker.
+ return cmp(pub1.datecreated, pub2.datecreated)
+ else:
+ return version_comparison
class Dominator:
@@ -116,50 +190,95 @@
self.logger = logger
self.archive = archive
- def _dominatePublications(self, pubs):
+ def dominatePackage(self, publications, live_versions, generalization):
+ """Dominate publications for a single package.
+
+ Active publications for versions in `live_versions` stay active.
+ Any older versions are marked as superseded by the respective
+ oldest live versions that are newer than the superseded ones.
+
+ Any versions that are newer than anything in `live_versions` are
+ marked as deleted. This should not be possible in Soyuz-native
+ archives, but it can happen during archive imports when the
+ previous latest version of a package has disappeared from the Sources
+ list we import.
+
+ :param publications: Iterable of publications for the same package,
+ in the same archive, series, and pocket, all with status
+ `PackagePublishingStatus.PUBLISHED`.
+ :param live_versions: Iterable of version strings that are still
+ considered live for this package. The given publications will
+ remain active insofar as they represent any of these versions;
+ older publications will be marked as superseded and newer ones
+ as deleted.
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
+ """
+ # Go through publications from latest version to oldest. This
+ # makes it easy to figure out which release superseded which:
+ # the dominant is always the oldest live release that is newer
+ # than the one being superseded.
+ publications = sorted(
+ publications, cmp=generalization.compare, reverse=True)
+
+ current_dominant = None
+ for pub in publications:
+ if generalization.getPackageVersion(pub) in live_versions:
+ # This publication stays active; if any publications
+ # that follow right after this are to be superseded,
+ # this is the release that they are superseded by.
+ current_dominant = pub
+ elif current_dominant is None:
+ # This publication is no longer live, but there is no
+ # newer version to supersede it either. Therefore it
+ # must be deleted.
+ pub.requestDeletion(None)
+ else:
+ # This publication is superseded. This is what we're
+ # here to do.
+ pub.supersede(current_dominant, logger=self.logger)
+
+ def _dominatePublications(self, pubs, generalization):
"""Perform dominations for the given publications.
+ Keep the latest published version for each package active,
+ superseding older versions.
+
:param pubs: A dict mapping names to a list of publications. Every
publication must be PUBLISHED or PENDING, and the first in each
list will be treated as dominant (so should be the latest).
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
"""
self.logger.debug("Dominating packages...")
-
- for name in pubs.keys():
- assert pubs[name], (
- "Empty list of publications for %s" % name)
- for pubrec in pubs[name][1:]:
- pubrec.supersede(pubs[name][0], logger=self.logger)
-
- def _sortPackages(self, pkglist, is_source=True):
+ for name, publications in pubs.iteritems():
+ assert publications, "Empty list of publications for %s." % name
+ # Since this always picks the latest version as the live
+ # one, this dominatePackage call will never result in a
+ # deletion.
+ latest_version = generalization.getPackageVersion(publications[0])
+ self.dominatePackage(
+ publications, [latest_version], generalization)
+
+ def _sortPackages(self, pkglist, generalization):
"""Map out packages by name, and sort by descending version.
:param pkglist: An iterable of `SourcePackagePublishingHistory` or
`BinaryPackagePublishingHistory`.
- :param is_source: Whether this call involves source package
- publications. If so, work with `SourcePackagePublishingHistory`.
- If not, work with `BinaryPackagepublishingHistory`.
- :return: A dict mapping each package name (as UTF-8 encoded string)
- to a list of publications from `pkglist`, newest first.
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
+ :return: A dict mapping each package name to a list of publications
+ from `pkglist`, newest first.
"""
self.logger.debug("Sorting packages...")
- if is_source:
- get_release = operator.attrgetter("sourcepackagerelease")
- get_name = operator.attrgetter("sourcepackagename")
- else:
- get_release = operator.attrgetter("binarypackagerelease")
- get_name = operator.attrgetter("binarypackagename")
-
outpkgs = {}
for inpkg in pkglist:
- key = get_name(get_release(inpkg)).name.encode('utf-8')
+ key = generalization.getPackageName(inpkg)
outpkgs.setdefault(key, []).append(inpkg)
- sort_order = functools.partial(
- _compare_packages_by_version_and_date, get_release)
for package_pubs in outpkgs.itervalues():
- package_pubs.sort(cmp=sort_order, reverse=True)
+ package_pubs.sort(cmp=generalization.compare, reverse=True)
return outpkgs
@@ -287,6 +406,8 @@
# Avoid circular imports.
from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
+ generalization = GeneralizedPublication(is_source=False)
+
for distroarchseries in distroseries.architectures:
self.logger.debug(
"Performing domination across %s/%s (%s)",
@@ -312,7 +433,7 @@
),
group_by=BinaryPackageName.id,
having=Count(BinaryPackagePublishingHistory.id) > 1)
- binaries = IMasterStore(BinaryPackagePublishingHistory).find(
+ binaries = IStore(BinaryPackagePublishingHistory).find(
BinaryPackagePublishingHistory,
BinaryPackageRelease.id ==
BinaryPackagePublishingHistory.binarypackagereleaseID,
@@ -322,7 +443,21 @@
BinaryPackageFormat.DDEB,
bpph_location_clauses)
self.logger.debug("Dominating binaries...")
- self._dominatePublications(self._sortPackages(binaries, False))
+ self._dominatePublications(
+ self._sortPackages(binaries, generalization), generalization)
+
+ def _composeActiveSourcePubsCondition(self, distroseries, pocket):
+ """Compose ORM condition for restricting relevant source pubs."""
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return And(
+ SourcePackagePublishingHistory.status ==
+ PackagePublishingStatus.PUBLISHED,
+ SourcePackagePublishingHistory.distroseries == distroseries,
+ SourcePackagePublishingHistory.archive == self.archive,
+ SourcePackagePublishingHistory.pocket == pocket,
+ )
def dominateSources(self, distroseries, pocket):
"""Perform domination on source package publications.
@@ -332,38 +467,76 @@
"""
# Avoid circular imports.
from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ generalization = GeneralizedPublication(is_source=True)
+
self.logger.debug(
"Performing domination across %s/%s (Source)",
distroseries.name, pocket.title)
- spph_location_clauses = And(
- SourcePackagePublishingHistory.status ==
- PackagePublishingStatus.PUBLISHED,
- SourcePackagePublishingHistory.distroseries == distroseries,
- SourcePackagePublishingHistory.archive == self.archive,
- SourcePackagePublishingHistory.pocket == pocket,
- )
+
+ spph_location_clauses = self._composeActiveSourcePubsCondition(
+ distroseries, pocket)
+ having_multiple_active_publications = (
+ Count(SourcePackagePublishingHistory.id) > 1)
candidate_source_names = Select(
SourcePackageName.id,
- And(
- SourcePackageRelease.sourcepackagenameID ==
- SourcePackageName.id,
- SourcePackagePublishingHistory.sourcepackagereleaseID ==
- SourcePackageRelease.id,
- spph_location_clauses,
- ),
+ And(join_spph_spr(), join_spr_spn(), spph_location_clauses),
group_by=SourcePackageName.id,
- having=Count(SourcePackagePublishingHistory.id) > 1)
- sources = IMasterStore(SourcePackagePublishingHistory).find(
+ having=having_multiple_active_publications)
+ sources = IStore(SourcePackagePublishingHistory).find(
SourcePackagePublishingHistory,
- SourcePackageRelease.id ==
- SourcePackagePublishingHistory.sourcepackagereleaseID,
+ join_spph_spr(),
SourcePackageRelease.sourcepackagenameID.is_in(
candidate_source_names),
spph_location_clauses)
+
self.logger.debug("Dominating sources...")
- self._dominatePublications(self._sortPackages(sources))
+ self._dominatePublications(
+ self._sortPackages(sources, generalization), generalization)
flush_database_updates()
+ def findPublishedSourcePackageNames(self, distroseries, pocket):
+ """Find names of currently published source packages."""
+ result = IStore(SourcePackageName).find(
+ SourcePackageName.name,
+ join_spph_spr(),
+ join_spr_spn(),
+ self._composeActiveSourcePubsCondition(distroseries, pocket))
+ return result.config(distinct=True)
+
+ def findPublishedSPPHs(self, distroseries, pocket, package_name):
+ """Find currently published source publications for given package."""
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return IStore(SourcePackagePublishingHistory).find(
+ SourcePackagePublishingHistory,
+ join_spph_spr(),
+ join_spr_spn(),
+ SourcePackageName.name == package_name,
+ self._composeActiveSourcePubsCondition(distroseries, pocket))
+
+ def dominateRemovedSourceVersions(self, distroseries, pocket,
+ package_name, live_versions):
+ """Dominate source publications based on a set of "live" versions.
+
+ Active publications for the "live" versions will remain active. All
+ other active publications for the same package (and the same archive,
+ distroseries, and pocket) are marked superseded.
+
+ Unlike traditional domination, this allows multiple versions of a
+ package to stay active in the same distroseries, archive, and pocket.
+
+ :param distroseries: `DistroSeries` to dominate.
+ :param pocket: `PackagePublishingPocket` to dominate.
+ :param package_name: Source package name, as text.
+ :param live_versions: Iterable of all version strings that are to
+ remain active.
+ """
+ generalization = GeneralizedPublication(is_source=True)
+ pubs = self.findPublishedSPPHs(distroseries, pocket, package_name)
+ self.dominatePackage(pubs, live_versions, generalization)
+
def judge(self, distroseries, pocket):
"""Judge superseded sources and binaries."""
# Avoid circular imports.
=== modified file 'lib/lp/archivepublisher/tests/test_dominator.py'
--- lib/lp/archivepublisher/tests/test_dominator.py 2011-02-04 05:11:00 +0000
+++ lib/lp/archivepublisher/tests/test_dominator.py 2011-09-08 07:38:24 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
# GNU Affero General Public License version 3 (see the file LICENSE).
"""Tests for domination.py."""
@@ -7,12 +7,24 @@
import datetime
+import apt_pkg
+from zope.security.proxy import removeSecurityProxy
+
from canonical.database.sqlbase import flush_database_updates
-from lp.archivepublisher.domination import Dominator, STAY_OF_EXECUTION
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.archivepublisher.domination import (
+ Dominator,
+ GeneralizedPublication,
+ STAY_OF_EXECUTION,
+ )
from lp.archivepublisher.publishing import Publisher
+from lp.registry.interfaces.pocket import PackagePublishingPocket
from lp.registry.interfaces.series import SeriesStatus
+from lp.services.log.logger import DevNullLogger
from lp.soyuz.enums import PackagePublishingStatus
+from lp.soyuz.interfaces.publishing import ISourcePackagePublishingHistory
from lp.soyuz.tests.test_publishing import TestNativePublishingBase
+from lp.testing import TestCaseWithFactory
class TestDominator(TestNativePublishingBase):
@@ -50,6 +62,8 @@
foo_10_source, foo_10_binaries[0])
def dominateAndCheck(self, dominant, dominated, supersededby):
+ generalization = GeneralizedPublication(
+ is_source=ISourcePackagePublishingHistory.providedBy(dominant))
dominator = Dominator(self.logger, self.ubuntutest.main_archive)
# The _dominate* test methods require a dictionary where the
@@ -58,7 +72,7 @@
# and dominated, the subsequents.
pubs = {'foo': [dominant, dominated]}
- dominator._dominatePublications(pubs)
+ dominator._dominatePublications(pubs, generalization)
flush_database_updates()
# The dominant version remains correctly published.
@@ -145,7 +159,9 @@
# This isn't a really good exception. It should probably be
# something more indicative of bad input.
self.assertRaises(
- AssertionError, dominator._dominatePublications, pubs)
+ AssertionError,
+ dominator._dominatePublications,
+ pubs, GeneralizedPublication(True))
class TestDomination(TestNativePublishingBase):
@@ -200,3 +216,398 @@
TestDomination.setUp(self)
self.ubuntutest['breezy-autotest'].status = (
SeriesStatus.OBSOLETE)
+
+
+def make_spphs_for_versions(factory, versions):
+ """Create publication records for each of `versions`.
+
+ They records are created in the same order in which they are specified.
+ Make the order irregular to prove that version ordering is not a
+ coincidence of object creation order etc.
+
+ Versions may also be identical; each publication record will still have
+ its own package release.
+ """
+ spn = factory.makeSourcePackageName()
+ distroseries = factory.makeDistroSeries()
+ pocket = factory.getAnyPocket()
+ sprs = [
+ factory.makeSourcePackageRelease(
+ sourcepackagename=spn, version=version)
+ for version in versions]
+ return [
+ factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=pocket,
+ sourcepackagerelease=spr,
+ status=PackagePublishingStatus.PUBLISHED)
+ for spr in sprs]
+
+
+def list_source_versions(spphs):
+ """Extract the versions from `spphs` as a list, in the same order."""
+ return [spph.sourcepackagerelease.version for spph in spphs]
+
+
+class TestGeneralizedPublication(TestCaseWithFactory):
+ """Test publication generalization helpers."""
+
+ layer = ZopelessDatabaseLayer
+
+ def alterCreationDates(self, spphs, ages):
+ """Set `datecreated` on each of `spphs` according to `ages`.
+
+ :param spphs: Iterable of `SourcePackagePublishingHistory`. Their
+ respective creation dates will be offset by the respective ages
+ found in `ages` (with the two being matched up in the same order).
+ :param ages: Iterable of ages. Must provide the same number of items
+ as `spphs`. Ages are `timedelta` objects that will be subtracted
+ from the creation dates on the respective records in `spph`.
+ """
+ for spph, age in zip(spphs, ages):
+ spph.datecreated -= age
+
+ def test_getPackageVersion_gets_source_version(self):
+ spph = self.factory.makeSourcePackagePublishingHistory()
+ self.assertEqual(
+ spph.sourcepackagerelease.version,
+ GeneralizedPublication(is_source=True).getPackageVersion(spph))
+
+ def test_getPackageVersion_gets_binary_version(self):
+ bpph = self.factory.makeBinaryPackagePublishingHistory()
+ self.assertEqual(
+ bpph.binarypackagerelease.version,
+ GeneralizedPublication(is_source=False).getPackageVersion(bpph))
+
+ def test_compare_sorts_versions(self):
+ versions = [
+ '1.1v2',
+ '1.1v1',
+ '1.1v3',
+ ]
+ spphs = make_spphs_for_versions(self.factory, versions)
+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+ self.assertEqual(
+ sorted(versions), list_source_versions(sorted_spphs))
+
+ def test_compare_orders_versions_by_debian_rules(self):
+ versions = [
+ '1.1.0',
+ '1.10',
+ '1.1',
+ '1.1ubuntu0',
+ ]
+ spphs = make_spphs_for_versions(self.factory, versions)
+
+ debian_sorted_versions = sorted(versions, cmp=apt_pkg.VersionCompare)
+
+ # Assumption: in this case, Debian version ordering is not the
+ # same as alphabetical version ordering.
+ self.assertNotEqual(sorted(versions), debian_sorted_versions)
+
+ # The compare method produces the Debian ordering.
+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+ self.assertEqual(
+ sorted(versions, cmp=apt_pkg.VersionCompare),
+ list_source_versions(sorted_spphs))
+
+ def test_compare_breaks_tie_with_creation_date(self):
+ # When two publications are tied for comparison because they are
+ # for the same package release, they are ordered by creation
+ # date.
+ distroseries = self.factory.makeDistroSeries()
+ pocket = self.factory.getAnyPocket()
+ spr = self.factory.makeSourcePackageRelease()
+ ages = [
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ datetime.timedelta(3),
+ ]
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ sourcepackagerelease=spr, distroseries=distroseries,
+ pocket=pocket)
+ for counter in xrange(len(ages))]
+ self.alterCreationDates(spphs, ages)
+
+ self.assertEqual(
+ [spphs[2], spphs[0], spphs[1]],
+ sorted(spphs, cmp=GeneralizedPublication().compare))
+
+ def test_compare_breaks_tie_for_releases_with_same_version(self):
+ # When two publications are tied for comparison because they
+ # belong to releases with the same version string, they are
+ # ordered by creation date.
+ version = "1.%d" % self.factory.getUniqueInteger()
+ ages = [
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ datetime.timedelta(3),
+ ]
+ distroseries = self.factory.makeDistroSeries()
+ pocket = self.factory.getAnyPocket()
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=pocket,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ version=version))
+ for counter in xrange(len(ages))]
+ self.alterCreationDates(spphs, ages)
+
+ self.assertEqual(
+ [spphs[2], spphs[0], spphs[1]],
+ sorted(spphs, cmp=GeneralizedPublication().compare))
+
+
+class TestDominatorMethods(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def makeDominator(self, publications):
+ """Create a `Dominator` suitable for `publications`."""
+ if len(publications) == 0:
+ archive = self.factory.makeArchive()
+ else:
+ archive = publications[0].archive
+ return Dominator(DevNullLogger(), archive)
+
+ def test_dominatePackage_survives_empty_publications_list(self):
+ # Nothing explodes when dominatePackage is called with an empty
+ # packages list.
+ self.makeDominator([]).dominatePackage(
+ [], [], GeneralizedPublication(True))
+ # The test is that we get here without error.
+ pass
+
+ def test_dominatePackage_leaves_live_version_untouched(self):
+ # dominatePackage does not supersede live versions.
+ [pub] = make_spphs_for_versions(self.factory, ['3.1'])
+ self.makeDominator([pub]).dominatePackage(
+ [pub], ['3.1'], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pub.status)
+
+ def test_dominatePackage_deletes_dead_version_without_successor(self):
+ # dominatePackage marks non-live package versions without
+ # superseding versions as deleted.
+ [pub] = make_spphs_for_versions(self.factory, ['1.1'])
+ self.makeDominator([pub]).dominatePackage(
+ [pub], [], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+ def test_dominatePackage_supersedes_older_pub_with_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage
+ # designates a newer live version as the superseding version.
+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['1.1'], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.SUPERSEDED, pubs[0].status)
+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+ def test_dominatePackage_only_supersedes_with_live_pub(self):
+ # When marking a package as superseded, dominatePackage will
+ # only pick a live version as the superseding one.
+ pubs = make_spphs_for_versions(
+ self.factory, ['1.0', '2.0', '3.0', '4.0'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['3.0'], GeneralizedPublication(True))
+ self.assertEqual([
+ pubs[2].sourcepackagerelease,
+ pubs[2].sourcepackagerelease,
+ None,
+ None,
+ ],
+ [pub.supersededby for pub in pubs])
+
+ def test_dominatePackage_supersedes_with_oldest_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage picks
+ # the oldest of the newer, live versions as the superseding one.
+ pubs = make_spphs_for_versions(self.factory, ['2.7', '2.8', '2.9'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['2.8', '2.9'], GeneralizedPublication(True))
+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+
+ def test_dominatePackage_only_supersedes_with_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage only
+ # considers a newer version as the superseding one.
+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['0.1'], GeneralizedPublication(True))
+ self.assertEqual(None, pubs[1].supersededby)
+ self.assertEqual(PackagePublishingStatus.DELETED, pubs[1].status)
+
+ def test_dominateRemovedSourceVersions_dominates_publications(self):
+ # dominateRemovedSourceVersions finds the publications for a
+ # package and calls dominatePackage on them.
+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2', '0.3'])
+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, package_name, ['0.2'])
+ self.assertEqual([
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ PackagePublishingStatus.DELETED,
+ ],
+ [pub.status for pub in pubs])
+ self.assertEqual(
+ [pubs[1].sourcepackagerelease, None, None],
+ [pub.supersededby for pub in pubs])
+
+ def test_dominateRemovedSourceVersions_ignores_other_pockets(self):
+ # dominateRemovedSourceVersions ignores publications in other
+ # pockets than the one specified.
+ pubs = make_spphs_for_versions(self.factory, ['2.3', '2.4'])
+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+ removeSecurityProxy(pubs[0]).pocket = PackagePublishingPocket.UPDATES
+ removeSecurityProxy(pubs[1]).pocket = PackagePublishingPocket.PROPOSED
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, package_name, ['2.3'])
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+ def test_dominateRemovedSourceVersions_ignores_other_packages(self):
+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+ other_package_name = self.factory.makeSourcePackageName().name
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, other_package_name, ['1.1'])
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[0].status)
+
+ def test_findPublishedSourcePackageNames_finds_package(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [spph.sourcepackagerelease.sourcepackagename.name],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_states(self):
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ spphs = dict(
+ (status, self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=status))
+ for status in PackagePublishingStatus.items)
+ published_spph = spphs[PackagePublishingStatus.PUBLISHED]
+ dominator = self.makeDominator(spphs.values())
+ self.assertContentEqual(
+ [published_spph.sourcepackagerelease.sourcepackagename.name],
+ dominator.findPublishedSourcePackageNames(series, pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_archives(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ dominator = self.makeDominator([spph])
+ dominator.archive = self.factory.makeArchive()
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_series(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ distro = spph.distroseries.distribution
+ other_series = self.factory.makeDistroSeries(distribution=distro)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ other_series, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_pockets(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED,
+ pocket=PackagePublishingPocket.RELEASE)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, PackagePublishingPocket.SECURITY))
+
+ def test_findPublishedSourcePackageNames_does_not_return_duplicates(self):
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ package = self.factory.makeSourcePackageName()
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package))
+ for counter in xrange(2)]
+ dominator = self.makeDominator(spphs)
+ self.assertEqual(
+ [package.name],
+ list(dominator.findPublishedSourcePackageNames(series, pocket)))
+
+ def test_findPublishedSPPHs_finds_published_SPPH(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ package_name = spph.sourcepackagerelease.sourcepackagename.name
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [spph],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, package_name))
+
+ def test_findPublishedSPPHs_ignores_other_states(self):
+ series = self.factory.makeDistroSeries()
+ package = self.factory.makeSourcePackageName()
+ pocket = PackagePublishingPocket.RELEASE
+ spphs = dict(
+ (status, self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=status,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package)))
+ for status in PackagePublishingStatus.items)
+ dominator = self.makeDominator(spphs.values())
+ self.assertContentEqual(
+ [spphs[PackagePublishingStatus.PUBLISHED]],
+ dominator.findPublishedSPPHs(series, pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_archives(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ package = spph.sourcepackagerelease.sourcepackagename
+ dominator = self.makeDominator([spph])
+ dominator.archive = self.factory.makeArchive()
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_series(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ distro = spph.distroseries.distribution
+ package = spph.sourcepackagerelease.sourcepackagename
+ other_series = self.factory.makeDistroSeries(distribution=distro)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ other_series, spph.pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_pockets(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED,
+ pocket=PackagePublishingPocket.RELEASE)
+ package = spph.sourcepackagerelease.sourcepackagename
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, PackagePublishingPocket.SECURITY,
+ package.name))
+
+ def test_findPublishedSPPHs_ignores_other_packages(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ other_package = self.factory.makeSourcePackageName()
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, other_package.name))
=== modified file 'lib/lp/soyuz/interfaces/publishing.py'
--- lib/lp/soyuz/interfaces/publishing.py 2011-09-02 04:51:25 +0000
+++ lib/lp/soyuz/interfaces/publishing.py 2011-09-08 07:38:24 +0000
@@ -195,9 +195,6 @@
the field name and value is the value string.
"""
- def supersede():
- """Supersede this publication."""
-
def requestObsolescence():
"""Make this publication obsolete.
=== modified file 'lib/lp/soyuz/model/publishing.py'
--- lib/lp/soyuz/model/publishing.py 2011-08-31 04:40:44 +0000
+++ lib/lp/soyuz/model/publishing.py 2011-09-08 07:38:24 +0000
@@ -327,8 +327,8 @@
fields = self.buildIndexStanzaFields()
return fields.makeOutput()
- def supersede(self):
- """See `IPublishing`."""
+ def setSuperseded(self):
+ """Set to SUPERSEDED status."""
self.status = PackagePublishingStatus.SUPERSEDED
self.datesuperseded = UTC_NOW
@@ -742,7 +742,7 @@
"Should not dominate unpublished source %s" %
self.sourcepackagerelease.title)
- super(SourcePackagePublishingHistory, self).supersede()
+ self.setSuperseded()
if dominant is not None:
if logger is not None:
@@ -1126,7 +1126,7 @@
self.distroarchseries.architecturetag))
return
- super(BinaryPackagePublishingHistory, self).supersede()
+ self.setSuperseded()
if dominant is not None:
# DDEBs cannot themselves be dominant; they are always dominated
=== added file 'lib/lp/soyuz/scripts/gina/retire.py'
--- lib/lp/soyuz/scripts/gina/retire.py 1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/scripts/gina/retire.py 2011-09-08 07:38:24 +0000
@@ -0,0 +1,64 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Retirement of packages that are removed upstream."""
+
+__metaclass__ = type
+__all__ = [
+ 'dominate_imported_source_packages',
+ ]
+
+from zope.component import getUtility
+
+# XXX JeroenVermeulen 2011-09-08, bug=844550: The GeneralizedPublication
+# import violates import policy and elicits a warning from the test
+# suite. The warning helps remind us to retire this code as soon as
+# possible.
+from lp.archivepublisher.domination import (
+ Dominator,
+ GeneralizedPublication,
+ )
+from lp.registry.interfaces.distribution import IDistributionSet
+
+
+def dominate_imported_source_packages(logger, distro_name, series_name,
+ pocket, packages_map):
+ """Perform domination."""
+ series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
+ dominator = Dominator(logger, series.main_archive)
+
+ # Dominate packages found in the Sources list we're importing.
+ package_names = dominator.findPublishedSourcePackageNames(series, pocket)
+ for package_name in package_names:
+ entries = packages_map.src_map.get(package_name)
+
+ if entries is None:
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a
+ # transitional hack. The database is full of "Published"
+ # Debian SPPHs whose packages have actually been deleted.
+ # In the future such publications should simply be marked
+ # Deleted, but for the legacy baggage we currently carry
+ # around we'll just do traditional domination first: pick
+ # the latest Published version, and mark the rest of the
+ # SPPHs as superseded by that version. The latest version
+ # will then, finally, be marked appropriately Deleted once
+ # we remove this transitional hack.
+ # To remove the transitional hack, just let live_versions
+ # default to the empty list instead of doing this:
+ pubs = dominator.findPublishedSPPHs(series, pocket, package_name)
+ generalization = GeneralizedPublication(is_source=True)
+ pubs_dict = dominator._sortPackages(pubs, generalization)
+ sorted_pubs = pubs_dict[package_name]
+ if len(sorted_pubs) <= 1:
+ # If there's only one published SPPH, the transitional
+ # code will just leave it Published. Don't bother; the
+ # migration will be costly enough as it is.
+ continue
+ live_versions = [sorted_pubs[0].sourcepackagerelease.version]
+ else:
+ live_versions = [
+ entry['Version']
+ for entry in entries if 'Version' in entry]
+
+ dominator.dominateRemovedSourceVersions(
+ series, pocket, package_name, live_versions)
=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
--- lib/lp/soyuz/scripts/tests/test_gina.py 2010-08-20 20:31:18 +0000
+++ lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-08 07:38:24 +0000
@@ -1,13 +1,76 @@
-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
# GNU Affero General Public License version 3 (see the file LICENSE).
from doctest import DocTestSuite
-import unittest
+from unittest import TestLoader
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.registry.interfaces.pocket import PackagePublishingPocket
+from lp.services.log.logger import DevNullLogger
+from lp.soyuz.enums import PackagePublishingStatus
import lp.soyuz.scripts.gina.handlers
+from lp.soyuz.scripts.gina.retire import dominate_imported_source_packages
+from lp.testing import TestCaseWithFactory
+
+
+class FakePackagesMap:
+ def __init__(self, src_map):
+ self.src_map = src_map
+
+
+class TestGina(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def test_dominate_imported_source_packages_dominates_imports(self):
+ # dominate_imported_source_packages dominates the source
+ # packages that Gina imports.
+ logger = DevNullLogger()
+ pub = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ series = pub.distroseries
+ spr = pub.sourcepackagerelease
+ package = spr.sourcepackagename
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, pub.pocket,
+ FakePackagesMap({package.name: []}))
+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+ def test_dominate_imported_source_packages_dominates_deletions(self):
+ # dominate_imported_source_packages dominates the source
+ # packages that have been deleted from the Sources lists that
+ # Gina imports.
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ package = self.factory.makeSourcePackageName()
+ pubs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ archive=series.main_archive, distroseries=series,
+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package, version=version))
+ for version in ['1.0', '1.1', '1.1a']]
+ logger = DevNullLogger()
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, pocket,
+ FakePackagesMap({}))
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is
+ # "transitional" domination which supersedes older versions of
+ # deleted packages with the last known version. Permanent
+ # domination will then mark the last known version as deleted.
+ # For permanent domination, the expected outcome is that all
+ # these publications will be Deleted (but any pre-existing
+ # Superseded publications for older versions will remain
+ # Superseded).
+ self.assertEqual([
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ ],
+ [pub.status for pub in pubs])
def test_suite():
- suite = unittest.TestSuite()
+ suite = TestLoader().loadTestsFromName(__name__)
suite.addTest(DocTestSuite(lp.soyuz.scripts.gina.handlers))
return suite
=== modified file 'scripts/gina.py'
--- scripts/gina.py 2011-08-23 08:35:13 +0000
+++ scripts/gina.py 2011-09-08 07:38:24 +0000
@@ -53,6 +53,7 @@
PoolFileNotFound,
SourcePackageData,
)
+from lp.soyuz.scripts.gina.retire import dominate_imported_source_packages
# Set to non-zero if you'd like to be warned every so often
COUNTDOWN = 0
@@ -152,6 +153,10 @@
packages_map, kdb, package_root, keyrings, importer_handler)
importer_handler.commit()
+ # XXX JeroenVermeulen 2011-09-07 bug=843728: Dominate binaries as well.
+ dominate_imported_source_packages(
+ log, distro, distroseries, pocket, packages_map)
+
if source_only:
log.info('Source only mode... done')
return
@@ -209,9 +214,8 @@
npacks = len(packages_map.src_map)
log.info('%i Source Packages to be imported', npacks)
- for list_source in sorted(
- packages_map.src_map.values(), key=lambda x: x[0].get("Package")):
- for source in list_source:
+ for package in sorted(packages_map.src_map.iterkeys()):
+ for source in packages_map.src_map[package]:
count += 1
attempt_source_package_import(
source, kdb, package_root, keyrings, importer_handler)
@@ -244,10 +248,9 @@
log.info(
'%i Binary Packages to be imported for %s', npacks, archtag)
# Go over binarypackages importing them for this architecture
- for binary in sorted(packages_map.bin_map[archtag].values(),
- key=lambda x: x.get("Package")):
+ for package_name in sorted(packages_map.bin_map[archtag].iterkeys()):
+ binary = packages_map.bin_map[archtag][package_name]
count += 1
- package_name = binary.get("Package", "unknown")
try:
try:
do_one_binarypackage(binary, archtag, kdb, package_root,
