launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #04920
[Merge] lp:~jtv/launchpad/bug-845326 into lp:launchpad
Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/bug-845326 into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #845326 in Launchpad itself: "SPPH should dominate other SPPH for same SPR"
https://bugs.launchpad.net/launchpad/+bug/845326
For more details, see:
https://code.launchpad.net/~jtv/launchpad/bug-845326/+merge/74740
= Summary =
I'm re-doing the domination process so that one Dominator can serve the needs of traditional, latest-publication-record-wins domination; versions-we-see-in-Sources-lists-live domination as needed in Gina; and the as-yet unspecified multiple-versions-can-live-side-by-side domination we are supposed to get in the future.
My code for this was not a full replacement for traditional domination. A single package release can have multiple active publications in the same archive, distroseries, and pocket (e.g. when the package is being re-published into a different component) and traditional domination would mark the newer publication as superseding the older one. The challenge for this branch was to fix that, in a way that was still general enough to serve Gina's needs. All that Gina really knows is what version numbers should survive.
== Proposed fix ==
Extend the new-style domination algorithm: when domination finds multiple publication records for the same package (and archive etc.), for a version that should stay published, then have the newest one supersede the older ones.
(My previous generalized code would keep all publications for the version published. The classic dominator code would only keep the newest publication of the very latest version published and supersede all older ones.)
== Pre-implementation notes ==
William seems to think the solution is sane. As far as we're aware right now, this is the last step before we can land and deploy transitional Gina domination.
== Implementation details ==
The fix itself is fairly straightforward. It's in the first file in the diff. It involves one redundant variable and an extra case in an if/elif cadence, but overall I think it works out pretty cleanly.
== Tests ==
Besides a unit test for the new behaviour, for good measure I also added a massive test for complex combined data. This is not meant to replace proper unit tests; it's too complex for that. But it may reveal any breakage that the unit tests might miss.
{{{
./bin/test -vvc lp.archivepublisher.tests.test_dominator
}}}
== Demo and Q/A ==
A bunch of branches are going to land together. We'll have to make sure that domination still works; that Gina still works; and that Gina now does proper domination.
= Launchpad lint =
There's some pre-existing lint in the dependent branches that either can't be fixed, or would increase the risk of conflicts too much. I did not create any lint of my own, however, and left less than I found.
Checking for conflicts and issues in changed files.
Linting changed files:
lib/lp/archivepublisher/domination.py
lib/lp/soyuz/scripts/gina/dominate.py
lib/lp/soyuz/model/publishing.py
lib/lp/soyuz/doc/gina.txt
lib/lp/soyuz/interfaces/publishing.py
scripts/gina.py
lib/lp/archivepublisher/tests/test_dominator.py
lib/lp/soyuz/scripts/tests/test_gina.py
lib/lp/soyuz/scripts/gina/handlers.py
./lib/lp/soyuz/doc/gina.txt
113: narrative exceeds 78 characters.
162: want exceeds 78 characters.
179: want exceeds 78 characters.
189: narrative uses a moin header.
221: want exceeds 78 characters.
234: want exceeds 78 characters.
240: want exceeds 78 characters.
295: source exceeds 78 characters.
324: narrative uses a moin header.
342: narrative exceeds 78 characters.
354: narrative uses a moin header.
360: narrative exceeds 78 characters.
361: narrative exceeds 78 characters.
459: narrative uses a moin header.
461: narrative exceeds 78 characters.
462: narrative exceeds 78 characters.
477: narrative uses a moin header.
563: narrative exceeds 78 characters.
600: narrative uses a moin header.
657: narrative uses a moin header.
746: narrative uses a moin header.
767: narrative uses a moin header.
780: narrative uses a moin header.
./lib/lp/soyuz/interfaces/publishing.py
381: E261 at least two spaces before inline comment
478: E261 at least two spaces before inline comment
511: E261 at least two spaces before inline comment
681: E261 at least two spaces before inline comment
767: E261 at least two spaces before inline comment
./scripts/gina.py
26: '_pythonpath' imported but unused
--
https://code.launchpad.net/~jtv/launchpad/bug-845326/+merge/74740
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/bug-845326 into lp:launchpad.
=== modified file 'lib/lp/archivepublisher/domination.py'
--- lib/lp/archivepublisher/domination.py 2011-08-30 06:37:55 +0000
+++ lib/lp/archivepublisher/domination.py 2011-09-09 09:33:30 +0000
@@ -53,8 +53,6 @@
__all__ = ['Dominator']
from datetime import timedelta
-import functools
-import operator
import apt_pkg
from storm.expr import (
@@ -68,7 +66,7 @@
flush_database_updates,
sqlvalues,
)
-from canonical.launchpad.interfaces.lpstorm import IMasterStore
+from canonical.launchpad.interfaces.lpstorm import IStore
from lp.registry.model.sourcepackagename import SourcePackageName
from lp.soyuz.enums import (
BinaryPackageFormat,
@@ -87,17 +85,93 @@
apt_pkg.InitSystem()
-def _compare_packages_by_version_and_date(get_release, p1, p2):
- """Compare publications p1 and p2 by their version; using Debian rules.
-
- If the publications are for the same package, compare by datecreated
- instead. This lets newer records win.
- """
- if get_release(p1).id == get_release(p2).id:
- return cmp(p1.datecreated, p2.datecreated)
-
- return apt_pkg.VersionCompare(get_release(p1).version,
- get_release(p2).version)
+def join_spr_spn():
+ """Join condition: SourcePackageRelease/SourcePackageName."""
+ return (
+ SourcePackageName.id == SourcePackageRelease.sourcepackagenameID)
+
+
+def join_spph_spr():
+ """Join condition: SourcePackageRelease/SourcePackagePublishingHistory.
+ """
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return (
+ SourcePackageRelease.id ==
+ SourcePackagePublishingHistory.sourcepackagereleaseID)
+
+
+class SourcePublicationTraits:
+ """Basic generalized attributes for `SourcePackagePublishingHistory`.
+
+ Used by `GeneralizedPublication` to hide the differences from
+ `BinaryPackagePublishingHistory`.
+ """
+ @staticmethod
+ def getPackageName(spph):
+ """Return the name of this publication's source package."""
+ return spph.sourcepackagerelease.sourcepackagename.name
+
+ @staticmethod
+ def getPackageRelease(spph):
+ """Return this publication's `SourcePackageRelease`."""
+ return spph.sourcepackagerelease
+
+
+class BinaryPublicationTraits:
+ """Basic generalized attributes for `BinaryPackagePublishingHistory`.
+
+ Used by `GeneralizedPublication` to hide the differences from
+ `SourcePackagePublishingHistory`.
+ """
+ @staticmethod
+ def getPackageName(bpph):
+ """Return the name of this publication's binary package."""
+ return bpph.binarypackagerelease.binarypackagename.name
+
+ @staticmethod
+ def getPackageRelease(bpph):
+ """Return this publication's `BinaryPackageRelease`."""
+ return bpph.binarypackagerelease
+
+
+class GeneralizedPublication:
+ """Generalize handling of publication records.
+
+ This allows us to write code that can be dealing with either
+ `SourcePackagePublishingHistory`s or `BinaryPackagePublishingHistory`s
+ without caring which. Differences are abstracted away in a traits
+ class.
+ """
+ def __init__(self, is_source=True):
+ if is_source:
+ self.traits = SourcePublicationTraits
+ else:
+ self.traits = BinaryPublicationTraits
+
+ def getPackageName(self, pub):
+ """Get the package's name."""
+ return self.traits.getPackageName(pub)
+
+ def getPackageVersion(self, pub):
+ """Obtain the version string for a publicaiton record."""
+ return self.traits.getPackageRelease(pub).version
+
+ def compare(self, pub1, pub2):
+ """Compare publications by version.
+
+ If both publications are for the same version, their creation dates
+ break the tie.
+ """
+ version_comparison = apt_pkg.VersionCompare(
+ self.getPackageVersion(pub1), self.getPackageVersion(pub2))
+
+ if version_comparison == 0:
+ # Use dates as tie breaker.
+ return cmp(pub1.datecreated, pub2.datecreated)
+ else:
+ return version_comparison
class Dominator:
@@ -116,50 +190,106 @@
self.logger = logger
self.archive = archive
- def _dominatePublications(self, pubs):
+ def dominatePackage(self, publications, live_versions, generalization):
+ """Dominate publications for a single package.
+
+ The latest publication for any version in `live_versions` stays
+ active. Any older publications (including older publications for
+ live versions with multiple publications) are marked as superseded by
+ the respective oldest live releases that are newer than the superseded
+ ones.
+
+ Any versions that are newer than anything in `live_versions` are
+ marked as deleted. This should not be possible in Soyuz-native
+ archives, but it can happen during archive imports when the
+ previous latest version of a package has disappeared from the Sources
+ list we import.
+
+ :param publications: Iterable of publications for the same package,
+ in the same archive, series, and pocket, all with status
+ `PackagePublishingStatus.PUBLISHED`.
+ :param live_versions: Iterable of version strings that are still
+ considered live for this package. The given publications will
+ remain active insofar as they represent any of these versions;
+ older publications will be marked as superseded and newer ones
+ as deleted.
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
+ """
+ # Go through publications from latest version to oldest. This
+ # makes it easy to figure out which release superseded which:
+ # the dominant is always the oldest live release that is newer
+ # than the one being superseded.
+ publications = sorted(
+ publications, cmp=generalization.compare, reverse=True)
+
+ current_dominant = None
+ dominant_version = None
+
+ for pub in publications:
+ version = generalization.getPackageVersion(pub)
+ if dominant_version is not None and version == dominant_version:
+ # This publication is for a live version, but has been
+ # superseded by a newer publication of the same version.
+ # Supersede it.
+ pub.supersede(current_dominant, logger=self.logger)
+ elif version in live_versions:
+ # This publication stays active; if any publications
+ # that follow right after this are to be superseded,
+ # this is the release that they are superseded by.
+ current_dominant = pub
+ dominant_version = version
+ elif current_dominant is None:
+ # This publication is no longer live, but there is no
+ # newer version to supersede it either. Therefore it
+ # must be deleted.
+ pub.requestDeletion(None)
+ else:
+ # This publication is superseded. This is what we're
+ # here to do.
+ pub.supersede(current_dominant, logger=self.logger)
+
+ def _dominatePublications(self, pubs, generalization):
"""Perform dominations for the given publications.
+ Keep the latest published version for each package active,
+ superseding older versions.
+
:param pubs: A dict mapping names to a list of publications. Every
publication must be PUBLISHED or PENDING, and the first in each
list will be treated as dominant (so should be the latest).
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
"""
self.logger.debug("Dominating packages...")
-
- for name in pubs.keys():
- assert pubs[name], (
- "Empty list of publications for %s" % name)
- for pubrec in pubs[name][1:]:
- pubrec.supersede(pubs[name][0], logger=self.logger)
-
- def _sortPackages(self, pkglist, is_source=True):
+ for name, publications in pubs.iteritems():
+ assert publications, "Empty list of publications for %s." % name
+ # Since this always picks the latest version as the live
+ # one, this dominatePackage call will never result in a
+ # deletion.
+ latest_version = generalization.getPackageVersion(publications[0])
+ self.dominatePackage(
+ publications, [latest_version], generalization)
+
+ def _sortPackages(self, pkglist, generalization):
"""Map out packages by name, and sort by descending version.
:param pkglist: An iterable of `SourcePackagePublishingHistory` or
`BinaryPackagePublishingHistory`.
- :param is_source: Whether this call involves source package
- publications. If so, work with `SourcePackagePublishingHistory`.
- If not, work with `BinaryPackagepublishingHistory`.
- :return: A dict mapping each package name (as UTF-8 encoded string)
- to a list of publications from `pkglist`, newest first.
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
+ :return: A dict mapping each package name to a list of publications
+ from `pkglist`, newest first.
"""
self.logger.debug("Sorting packages...")
- if is_source:
- get_release = operator.attrgetter("sourcepackagerelease")
- get_name = operator.attrgetter("sourcepackagename")
- else:
- get_release = operator.attrgetter("binarypackagerelease")
- get_name = operator.attrgetter("binarypackagename")
-
outpkgs = {}
for inpkg in pkglist:
- key = get_name(get_release(inpkg)).name.encode('utf-8')
+ key = generalization.getPackageName(inpkg)
outpkgs.setdefault(key, []).append(inpkg)
- sort_order = functools.partial(
- _compare_packages_by_version_and_date, get_release)
for package_pubs in outpkgs.itervalues():
- package_pubs.sort(cmp=sort_order, reverse=True)
+ package_pubs.sort(cmp=generalization.compare, reverse=True)
return outpkgs
@@ -287,6 +417,8 @@
# Avoid circular imports.
from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
+ generalization = GeneralizedPublication(is_source=False)
+
for distroarchseries in distroseries.architectures:
self.logger.debug(
"Performing domination across %s/%s (%s)",
@@ -312,7 +444,7 @@
),
group_by=BinaryPackageName.id,
having=Count(BinaryPackagePublishingHistory.id) > 1)
- binaries = IMasterStore(BinaryPackagePublishingHistory).find(
+ binaries = IStore(BinaryPackagePublishingHistory).find(
BinaryPackagePublishingHistory,
BinaryPackageRelease.id ==
BinaryPackagePublishingHistory.binarypackagereleaseID,
@@ -322,7 +454,21 @@
BinaryPackageFormat.DDEB,
bpph_location_clauses)
self.logger.debug("Dominating binaries...")
- self._dominatePublications(self._sortPackages(binaries, False))
+ self._dominatePublications(
+ self._sortPackages(binaries, generalization), generalization)
+
+ def _composeActiveSourcePubsCondition(self, distroseries, pocket):
+ """Compose ORM condition for restricting relevant source pubs."""
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return And(
+ SourcePackagePublishingHistory.status ==
+ PackagePublishingStatus.PUBLISHED,
+ SourcePackagePublishingHistory.distroseries == distroseries,
+ SourcePackagePublishingHistory.archive == self.archive,
+ SourcePackagePublishingHistory.pocket == pocket,
+ )
def dominateSources(self, distroseries, pocket):
"""Perform domination on source package publications.
@@ -332,38 +478,76 @@
"""
# Avoid circular imports.
from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ generalization = GeneralizedPublication(is_source=True)
+
self.logger.debug(
"Performing domination across %s/%s (Source)",
distroseries.name, pocket.title)
- spph_location_clauses = And(
- SourcePackagePublishingHistory.status ==
- PackagePublishingStatus.PUBLISHED,
- SourcePackagePublishingHistory.distroseries == distroseries,
- SourcePackagePublishingHistory.archive == self.archive,
- SourcePackagePublishingHistory.pocket == pocket,
- )
+
+ spph_location_clauses = self._composeActiveSourcePubsCondition(
+ distroseries, pocket)
+ having_multiple_active_publications = (
+ Count(SourcePackagePublishingHistory.id) > 1)
candidate_source_names = Select(
SourcePackageName.id,
- And(
- SourcePackageRelease.sourcepackagenameID ==
- SourcePackageName.id,
- SourcePackagePublishingHistory.sourcepackagereleaseID ==
- SourcePackageRelease.id,
- spph_location_clauses,
- ),
+ And(join_spph_spr(), join_spr_spn(), spph_location_clauses),
group_by=SourcePackageName.id,
- having=Count(SourcePackagePublishingHistory.id) > 1)
- sources = IMasterStore(SourcePackagePublishingHistory).find(
+ having=having_multiple_active_publications)
+ sources = IStore(SourcePackagePublishingHistory).find(
SourcePackagePublishingHistory,
- SourcePackageRelease.id ==
- SourcePackagePublishingHistory.sourcepackagereleaseID,
+ join_spph_spr(),
SourcePackageRelease.sourcepackagenameID.is_in(
candidate_source_names),
spph_location_clauses)
+
self.logger.debug("Dominating sources...")
- self._dominatePublications(self._sortPackages(sources))
+ self._dominatePublications(
+ self._sortPackages(sources, generalization), generalization)
flush_database_updates()
+ def findPublishedSourcePackageNames(self, distroseries, pocket):
+ """Find names of currently published source packages."""
+ result = IStore(SourcePackageName).find(
+ SourcePackageName.name,
+ join_spph_spr(),
+ join_spr_spn(),
+ self._composeActiveSourcePubsCondition(distroseries, pocket))
+ return result.config(distinct=True)
+
+ def findPublishedSPPHs(self, distroseries, pocket, package_name):
+ """Find currently published source publications for given package."""
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return IStore(SourcePackagePublishingHistory).find(
+ SourcePackagePublishingHistory,
+ join_spph_spr(),
+ join_spr_spn(),
+ SourcePackageName.name == package_name,
+ self._composeActiveSourcePubsCondition(distroseries, pocket))
+
+ def dominateRemovedSourceVersions(self, distroseries, pocket,
+ package_name, live_versions):
+ """Dominate source publications based on a set of "live" versions.
+
+ Active publications for the "live" versions will remain active. All
+ other active publications for the same package (and the same archive,
+ distroseries, and pocket) are marked superseded.
+
+ Unlike traditional domination, this allows multiple versions of a
+ package to stay active in the same distroseries, archive, and pocket.
+
+ :param distroseries: `DistroSeries` to dominate.
+ :param pocket: `PackagePublishingPocket` to dominate.
+ :param package_name: Source package name, as text.
+ :param live_versions: Iterable of all version strings that are to
+ remain active.
+ """
+ generalization = GeneralizedPublication(is_source=True)
+ pubs = self.findPublishedSPPHs(distroseries, pocket, package_name)
+ self.dominatePackage(pubs, live_versions, generalization)
+
def judge(self, distroseries, pocket):
"""Judge superseded sources and binaries."""
# Avoid circular imports.
=== modified file 'lib/lp/archivepublisher/tests/test_dominator.py'
--- lib/lp/archivepublisher/tests/test_dominator.py 2011-02-04 05:11:00 +0000
+++ lib/lp/archivepublisher/tests/test_dominator.py 2011-09-09 09:33:30 +0000
@@ -1,18 +1,30 @@
-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
# GNU Affero General Public License version 3 (see the file LICENSE).
"""Tests for domination.py."""
__metaclass__ = type
+import apt_pkg
import datetime
+from operator import attrgetter
+from zope.security.proxy import removeSecurityProxy
from canonical.database.sqlbase import flush_database_updates
-from lp.archivepublisher.domination import Dominator, STAY_OF_EXECUTION
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.archivepublisher.domination import (
+ Dominator,
+ GeneralizedPublication,
+ STAY_OF_EXECUTION,
+ )
from lp.archivepublisher.publishing import Publisher
+from lp.registry.interfaces.pocket import PackagePublishingPocket
from lp.registry.interfaces.series import SeriesStatus
+from lp.services.log.logger import DevNullLogger
from lp.soyuz.enums import PackagePublishingStatus
+from lp.soyuz.interfaces.publishing import ISourcePackagePublishingHistory
from lp.soyuz.tests.test_publishing import TestNativePublishingBase
+from lp.testing import TestCaseWithFactory
class TestDominator(TestNativePublishingBase):
@@ -50,6 +62,8 @@
foo_10_source, foo_10_binaries[0])
def dominateAndCheck(self, dominant, dominated, supersededby):
+ generalization = GeneralizedPublication(
+ is_source=ISourcePackagePublishingHistory.providedBy(dominant))
dominator = Dominator(self.logger, self.ubuntutest.main_archive)
# The _dominate* test methods require a dictionary where the
@@ -58,7 +72,7 @@
# and dominated, the subsequents.
pubs = {'foo': [dominant, dominated]}
- dominator._dominatePublications(pubs)
+ dominator._dominatePublications(pubs, generalization)
flush_database_updates()
# The dominant version remains correctly published.
@@ -145,7 +159,9 @@
# This isn't a really good exception. It should probably be
# something more indicative of bad input.
self.assertRaises(
- AssertionError, dominator._dominatePublications, pubs)
+ AssertionError,
+ dominator._dominatePublications,
+ pubs, GeneralizedPublication(True))
class TestDomination(TestNativePublishingBase):
@@ -200,3 +216,533 @@
TestDomination.setUp(self)
self.ubuntutest['breezy-autotest'].status = (
SeriesStatus.OBSOLETE)
+
+
+def make_spphs_for_versions(factory, versions):
+ """Create publication records for each of `versions`.
+
+ They records are created in the same order in which they are specified.
+ Make the order irregular to prove that version ordering is not a
+ coincidence of object creation order etc.
+
+ Versions may also be identical; each publication record will still have
+ its own package release.
+ """
+ spn = factory.makeSourcePackageName()
+ distroseries = factory.makeDistroSeries()
+ pocket = factory.getAnyPocket()
+ sprs = [
+ factory.makeSourcePackageRelease(
+ sourcepackagename=spn, version=version)
+ for version in versions]
+ return [
+ factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=pocket,
+ sourcepackagerelease=spr,
+ status=PackagePublishingStatus.PUBLISHED)
+ for spr in sprs]
+
+
+def list_source_versions(spphs):
+ """Extract the versions from `spphs` as a list, in the same order."""
+ return [spph.sourcepackagerelease.version for spph in spphs]
+
+
+def alter_creation_dates(spphs, ages):
+ """Set `datecreated` on each of `spphs` according to `ages`.
+
+ :param spphs: Iterable of `SourcePackagePublishingHistory`. Their
+ respective creation dates will be offset by the respective ages found
+ in `ages` (with the two being matched up in the same order).
+ :param ages: Iterable of ages. Must provide the same number of items as
+ `spphs`. Ages are `timedelta` objects that will be subtracted from
+ the creation dates on the respective records in `spph`.
+ """
+ for spph, age in zip(spphs, ages):
+ spph.datecreated -= age
+
+
+class TestGeneralizedPublication(TestCaseWithFactory):
+ """Test publication generalization helpers."""
+
+ layer = ZopelessDatabaseLayer
+
+ def test_getPackageVersion_gets_source_version(self):
+ spph = self.factory.makeSourcePackagePublishingHistory()
+ self.assertEqual(
+ spph.sourcepackagerelease.version,
+ GeneralizedPublication(is_source=True).getPackageVersion(spph))
+
+ def test_getPackageVersion_gets_binary_version(self):
+ bpph = self.factory.makeBinaryPackagePublishingHistory()
+ self.assertEqual(
+ bpph.binarypackagerelease.version,
+ GeneralizedPublication(is_source=False).getPackageVersion(bpph))
+
+ def test_compare_sorts_versions(self):
+ versions = [
+ '1.1v2',
+ '1.1v1',
+ '1.1v3',
+ ]
+ spphs = make_spphs_for_versions(self.factory, versions)
+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+ self.assertEqual(
+ sorted(versions), list_source_versions(sorted_spphs))
+
+ def test_compare_orders_versions_by_debian_rules(self):
+ versions = [
+ '1.1.0',
+ '1.10',
+ '1.1',
+ '1.1ubuntu0',
+ ]
+ spphs = make_spphs_for_versions(self.factory, versions)
+
+ debian_sorted_versions = sorted(versions, cmp=apt_pkg.VersionCompare)
+
+ # Assumption: in this case, Debian version ordering is not the
+ # same as alphabetical version ordering.
+ self.assertNotEqual(sorted(versions), debian_sorted_versions)
+
+ # The compare method produces the Debian ordering.
+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+ self.assertEqual(
+ sorted(versions, cmp=apt_pkg.VersionCompare),
+ list_source_versions(sorted_spphs))
+
+ def test_compare_breaks_tie_with_creation_date(self):
+ # When two publications are tied for comparison because they are
+ # for the same package release, they are ordered by creation
+ # date.
+ distroseries = self.factory.makeDistroSeries()
+ pocket = self.factory.getAnyPocket()
+ spr = self.factory.makeSourcePackageRelease()
+ ages = [
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ datetime.timedelta(3),
+ ]
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ sourcepackagerelease=spr, distroseries=distroseries,
+ pocket=pocket)
+ for counter in xrange(len(ages))]
+ alter_creation_dates(spphs, ages)
+
+ self.assertEqual(
+ [spphs[2], spphs[0], spphs[1]],
+ sorted(spphs, cmp=GeneralizedPublication().compare))
+
+ def test_compare_breaks_tie_for_releases_with_same_version(self):
+ # When two publications are tied for comparison because they
+ # belong to releases with the same version string, they are
+ # ordered by creation date.
+ version = "1.%d" % self.factory.getUniqueInteger()
+ ages = [
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ datetime.timedelta(3),
+ ]
+ distroseries = self.factory.makeDistroSeries()
+ pocket = self.factory.getAnyPocket()
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=pocket,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ version=version))
+ for counter in xrange(len(ages))]
+ alter_creation_dates(spphs, ages)
+
+ self.assertEqual(
+ [spphs[2], spphs[0], spphs[1]],
+ sorted(spphs, cmp=GeneralizedPublication().compare))
+
+
+def jumble(ordered_list):
+ """Jumble the elements of `ordered_list` into a weird order.
+
+ Ordering is very important in domination. We jumble some of our lists to
+ insure against "lucky coincidences" that might give our tests the right
+ answers for the wrong reasons.
+ """
+ even = [
+ item for offset, item in enumerate(ordered_list) if offset % 2 == 0]
+ odd = [
+ item for offset, item in enumerate(ordered_list) if offset % 2 != 0]
+ return list(reversed(odd)) + even
+
+
+class TestDominatorMethods(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def makeDominator(self, publications):
+ """Create a `Dominator` suitable for `publications`."""
+ if len(publications) == 0:
+ archive = self.factory.makeArchive()
+ else:
+ archive = publications[0].archive
+ return Dominator(DevNullLogger(), archive)
+
+ def test_dominatePackage_survives_empty_publications_list(self):
+ # Nothing explodes when dominatePackage is called with an empty
+ # packages list.
+ self.makeDominator([]).dominatePackage(
+ [], [], GeneralizedPublication(True))
+ # The test is that we get here without error.
+ pass
+
+ def test_dominatePackage_leaves_live_version_untouched(self):
+ # dominatePackage does not supersede live versions.
+ [pub] = make_spphs_for_versions(self.factory, ['3.1'])
+ self.makeDominator([pub]).dominatePackage(
+ [pub], ['3.1'], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pub.status)
+
+ def test_dominatePackage_deletes_dead_version_without_successor(self):
+ # dominatePackage marks non-live package versions without
+ # superseding versions as deleted.
+ [pub] = make_spphs_for_versions(self.factory, ['1.1'])
+ self.makeDominator([pub]).dominatePackage(
+ [pub], [], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+ def test_dominatePackage_supersedes_older_pub_with_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage
+ # designates a newer live version as the superseding version.
+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['1.1'], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.SUPERSEDED, pubs[0].status)
+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+ def test_dominatePackage_only_supersedes_with_live_pub(self):
+ # When marking a package as superseded, dominatePackage will
+ # only pick a live version as the superseding one.
+ pubs = make_spphs_for_versions(
+ self.factory, ['1.0', '2.0', '3.0', '4.0'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['3.0'], GeneralizedPublication(True))
+ self.assertEqual([
+ pubs[2].sourcepackagerelease,
+ pubs[2].sourcepackagerelease,
+ None,
+ None,
+ ],
+ [pub.supersededby for pub in pubs])
+
+ def test_dominatePackage_supersedes_with_oldest_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage picks
+ # the oldest of the newer, live versions as the superseding one.
+ pubs = make_spphs_for_versions(self.factory, ['2.7', '2.8', '2.9'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['2.8', '2.9'], GeneralizedPublication(True))
+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+
+ def test_dominatePackage_only_supersedes_with_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage only
+ # considers a newer version as the superseding one.
+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['0.1'], GeneralizedPublication(True))
+ self.assertEqual(None, pubs[1].supersededby)
+ self.assertEqual(PackagePublishingStatus.DELETED, pubs[1].status)
+
+ def test_dominatePackage_supersedes_replaced_pub_for_live_version(self):
+ # Even if a publication record is for a live version, a newer
+ # one for the same version supersedes it.
+ spr = self.factory.makeSourcePackageRelease()
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ pubs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ archive=series.main_archive, distroseries=series,
+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+ sourcepackagerelease=spr)
+ for counter in xrange(3)]
+ alter_creation_dates(pubs, [
+ datetime.timedelta(3),
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ ])
+
+ self.makeDominator(pubs).dominatePackage(
+ pubs, [spr.version], GeneralizedPublication(True))
+ self.assertEqual([
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ ],
+ [pub.status for pub in pubs])
+ self.assertEqual(
+ [spr, spr, None], [pub.supersededby for pub in pubs])
+
+ def test_dominatePackage_advanced_scenario(self):
+ # Put dominatePackage through its paces with complex combined
+ # data.
+ # This test should be redundant in theory (which in theory
+ # equates practice but in practice does not). If this fails,
+ # don't just patch up the code or this test. Create unit tests
+ # that specifically cover the difference, then change the code
+ # and/or adapt this test to return to harmony.
+ series = self.factory.makeDistroSeries()
+ package = self.factory.makeSourcePackageName()
+ pocket = PackagePublishingPocket.RELEASE
+
+ versions = ["1.%d" % number for number in xrange(4)]
+
+ # We have one package releases for each version.
+ relevant_releases = dict(
+ (version, self.factory.makeSourcePackageRelease(
+ sourcepackagename=package, version=version))
+ for version in jumble(versions))
+
+ # Each of those releases is subsequently published in
+ # different components.
+ components = jumble(
+ [self.factory.makeComponent() for version in versions])
+
+ # Map versions to lists of publications for that version, from
+ # oldest to newest. Each re-publishing into a different
+ # component is meant to supersede publication into the previous
+ # component.
+ pubs_by_version = dict(
+ (version, [
+ self.factory.makeSourcePackagePublishingHistory(
+ archive=series.main_archive, distroseries=series,
+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+ sourcepackagerelease=relevant_releases[version],
+ component=component)
+ for component in components])
+ for version in jumble(versions))
+
+ ages = jumble(
+ [datetime.timedelta(age) for age in xrange(len(versions))])
+
+ # Actually the "oldest to newest" order on the publications only
+ # applies to their creation dates. Their creation orders are
+ # irrelevant.
+ for pubs_list in pubs_by_version.itervalues():
+ alter_creation_dates(pubs_list, ages)
+ pubs_list.sort(key=attrgetter('datecreated'))
+
+ live_versions = ["1.1", "1.2"]
+ last_version_alive = sorted(live_versions)[-1]
+
+ all_pubs = sum(pubs_by_version.itervalues(), [])
+ Dominator(DevNullLogger(), series.main_archive).dominatePackage(
+ all_pubs, live_versions, GeneralizedPublication(True))
+
+ for version in reversed(versions):
+ pubs = pubs_by_version[version]
+
+ if version in live_versions:
+ # Beware: loop-carried variable. Used locally as well,
+ # but tells later iterations what the highest-versioned
+ # release so far was. This is used in tracking
+ # supersededby links.
+ superseding_release = pubs[-1].sourcepackagerelease
+
+ if version in live_versions:
+ # The live versions' latest publications are Published,
+ # their older ones Superseded.
+ expected_status = (
+ [PackagePublishingStatus.SUPERSEDED] * (len(pubs) - 1) +
+ [PackagePublishingStatus.PUBLISHED])
+ expected_supersededby = (
+ [superseding_release] * (len(pubs) - 1) + [None])
+ elif version < last_version_alive:
+ # The superseded versions older than the last live
+ # version have all been superseded.
+ expected_status = (
+ [PackagePublishingStatus.SUPERSEDED] * len(pubs))
+ expected_supersededby = [superseding_release] * len(pubs)
+ else:
+ # Versions that are newer than any live release have
+ # been deleted.
+ expected_status = (
+ [PackagePublishingStatus.DELETED] * len(pubs))
+ expected_supersededby = [None] * len(pubs)
+
+ self.assertEqual(expected_status, [pub.status for pub in pubs])
+ self.assertEqual(
+ expected_supersededby, [pub.supersededby for pub in pubs])
+
+ def test_dominateRemovedSourceVersions_dominates_publications(self):
+ # dominateRemovedSourceVersions finds the publications for a
+ # package and calls dominatePackage on them.
+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2', '0.3'])
+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, package_name, ['0.2'])
+ self.assertEqual([
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ PackagePublishingStatus.DELETED,
+ ],
+ [pub.status for pub in pubs])
+ self.assertEqual(
+ [pubs[1].sourcepackagerelease, None, None],
+ [pub.supersededby for pub in pubs])
+
+ def test_dominateRemovedSourceVersions_ignores_other_pockets(self):
+ # dominateRemovedSourceVersions ignores publications in other
+ # pockets than the one specified.
+ pubs = make_spphs_for_versions(self.factory, ['2.3', '2.4'])
+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+ removeSecurityProxy(pubs[0]).pocket = PackagePublishingPocket.UPDATES
+ removeSecurityProxy(pubs[1]).pocket = PackagePublishingPocket.PROPOSED
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, package_name, ['2.3'])
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+ def test_dominateRemovedSourceVersions_ignores_other_packages(self):
+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+ other_package_name = self.factory.makeSourcePackageName().name
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, other_package_name, ['1.1'])
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[0].status)
+
+ def test_findPublishedSourcePackageNames_finds_package(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [spph.sourcepackagerelease.sourcepackagename.name],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_states(self):
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ spphs = dict(
+ (status, self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=status))
+ for status in PackagePublishingStatus.items)
+ published_spph = spphs[PackagePublishingStatus.PUBLISHED]
+ dominator = self.makeDominator(spphs.values())
+ self.assertContentEqual(
+ [published_spph.sourcepackagerelease.sourcepackagename.name],
+ dominator.findPublishedSourcePackageNames(series, pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_archives(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ dominator = self.makeDominator([spph])
+ dominator.archive = self.factory.makeArchive()
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_series(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ distro = spph.distroseries.distribution
+ other_series = self.factory.makeDistroSeries(distribution=distro)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ other_series, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_pockets(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED,
+ pocket=PackagePublishingPocket.RELEASE)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, PackagePublishingPocket.SECURITY))
+
+ def test_findPublishedSourcePackageNames_does_not_return_duplicates(self):
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ package = self.factory.makeSourcePackageName()
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package))
+ for counter in xrange(2)]
+ dominator = self.makeDominator(spphs)
+ self.assertEqual(
+ [package.name],
+ list(dominator.findPublishedSourcePackageNames(series, pocket)))
+
+ def test_findPublishedSPPHs_finds_published_SPPH(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ package_name = spph.sourcepackagerelease.sourcepackagename.name
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [spph],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, package_name))
+
+ def test_findPublishedSPPHs_ignores_other_states(self):
+ series = self.factory.makeDistroSeries()
+ package = self.factory.makeSourcePackageName()
+ pocket = PackagePublishingPocket.RELEASE
+ spphs = dict(
+ (status, self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=status,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package)))
+ for status in PackagePublishingStatus.items)
+ dominator = self.makeDominator(spphs.values())
+ self.assertContentEqual(
+ [spphs[PackagePublishingStatus.PUBLISHED]],
+ dominator.findPublishedSPPHs(series, pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_archives(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ package = spph.sourcepackagerelease.sourcepackagename
+ dominator = self.makeDominator([spph])
+ dominator.archive = self.factory.makeArchive()
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_series(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ distro = spph.distroseries.distribution
+ package = spph.sourcepackagerelease.sourcepackagename
+ other_series = self.factory.makeDistroSeries(distribution=distro)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ other_series, spph.pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_pockets(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED,
+ pocket=PackagePublishingPocket.RELEASE)
+ package = spph.sourcepackagerelease.sourcepackagename
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, PackagePublishingPocket.SECURITY,
+ package.name))
+
+ def test_findPublishedSPPHs_ignores_other_packages(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ other_package = self.factory.makeSourcePackageName()
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, other_package.name))
=== modified file 'lib/lp/soyuz/doc/gina.txt'
--- lib/lp/soyuz/doc/gina.txt 2011-07-29 11:35:28 +0000
+++ lib/lp/soyuz/doc/gina.txt 2011-09-09 09:33:30 +0000
@@ -8,6 +8,7 @@
Get the current counts of stuff in the database:
>>> from canonical.launchpad.database.emailaddress import EmailAddress
+ >>> from lp.soyuz.interfaces.publishing import active_publishing_status
>>> from lp.soyuz.model.publishing import (
... BinaryPackagePublishingHistory,
... SourcePackagePublishingHistory)
@@ -564,35 +565,34 @@
that's what overrides actually do.
>>> from canonical.database.sqlbase import sqlvalues
- >>> from lp.soyuz.enums import PackagePublishingStatus
- >>> x11_pub = SSPPH.select("""sourcepackagerelease = %s
- ... AND distroseries = %s
- ... AND status in (%s, %s)""" %
- ... sqlvalues(x11p, breezy,
- ... PackagePublishingStatus.PUBLISHED,
- ... PackagePublishingStatus.PENDING),
- ... orderBy=["-datecreated"])[0]
+ >>> x11_pub = SSPPH.select("""
+ ... sourcepackagerelease = %s AND
+ ... distroseries = %s AND
+ ... status in %s
+ ... """ % sqlvalues(
+ ... x11p, breezy, active_publishing_status),
+ ... orderBy=["-datecreated"])[0]
>>> print x11_pub.section.name
net
- >>> ed_pub = SBPPH.select("""binarypackagerelease = %s
- ... AND distroarchseries = %s
- ... AND status in (%s, %s)""" %
- ... sqlvalues(ed, breezy_i386,
- ... PackagePublishingStatus.PUBLISHED,
- ... PackagePublishingStatus.PENDING),
- ... orderBy=["-datecreated"])[0]
+ >>> ed_pub = SBPPH.select("""
+ ... binarypackagerelease = %s AND
+ ... distroarchseries = %s AND
+ ... status in %s
+ ... """ % sqlvalues(
+ ... ed, breezy_i386, active_publishing_status),
+ ... orderBy=["-datecreated"])[0]
>>> print ed_pub.priority
Extra
>>> n = SourcePackageName.selectOneBy(name="archive-copier")
>>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
... version="0.3.6")
- >>> ac_pub = SSPPH.select("""sourcepackagerelease = %s
- ... AND distroseries = %s
- ... AND status in (%s, %s)""" %
- ... sqlvalues(ac, breezy,
- ... PackagePublishingStatus.PUBLISHED,
- ... PackagePublishingStatus.PENDING),
- ... orderBy=["-datecreated"])[0]
+ >>> ac_pub = SSPPH.select("""
+ ... sourcepackagerelease = %s AND
+ ... distroseries = %s AND
+ ... status in %s
+ ... """ % sqlvalues(
+ ... ac, breezy, active_publishing_status),
+ ... orderBy=["-datecreated"])[0]
>>> print ac_pub.component.name
universe
@@ -720,7 +720,7 @@
>>> transaction.commit()
-There is now a number of source publications in PENDING status for the
+There is now a number of source publications in PUBLISHED status for the
targetted distroseries, 'lenny'.
>>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))
@@ -728,7 +728,7 @@
12
>>> print set([pub.status.name for pub in lenny_sources])
- set(['PENDING'])
+ set(['PUBLISHED'])
As mentioned before, lenny/i386 is empty, no binaries were imported.
Also, the number of binaries published in the whole debian distribution
=== modified file 'lib/lp/soyuz/interfaces/publishing.py'
--- lib/lp/soyuz/interfaces/publishing.py 2011-09-02 04:51:25 +0000
+++ lib/lp/soyuz/interfaces/publishing.py 2011-09-09 09:33:30 +0000
@@ -195,9 +195,6 @@
the field name and value is the value string.
"""
- def supersede():
- """Supersede this publication."""
-
def requestObsolescence():
"""Make this publication obsolete.
=== modified file 'lib/lp/soyuz/model/publishing.py'
--- lib/lp/soyuz/model/publishing.py 2011-08-31 04:40:44 +0000
+++ lib/lp/soyuz/model/publishing.py 2011-09-09 09:33:30 +0000
@@ -114,10 +114,6 @@
from lp.soyuz.scripts.changeoverride import ArchiveOverriderError
-PENDING = PackagePublishingStatus.PENDING
-PUBLISHED = PackagePublishingStatus.PUBLISHED
-
-
# XXX cprov 2006-08-18: move it away, perhaps archivepublisher/pool.py
def makePoolPath(source_name, component_name):
@@ -327,8 +323,8 @@
fields = self.buildIndexStanzaFields()
return fields.makeOutput()
- def supersede(self):
- """See `IPublishing`."""
+ def setSuperseded(self):
+ """Set to SUPERSEDED status."""
self.status = PackagePublishingStatus.SUPERSEDED
self.datesuperseded = UTC_NOW
@@ -742,7 +738,7 @@
"Should not dominate unpublished source %s" %
self.sourcepackagerelease.title)
- super(SourcePackagePublishingHistory, self).supersede()
+ self.setSuperseded()
if dominant is not None:
if logger is not None:
@@ -1081,7 +1077,7 @@
return IMasterStore(BinaryPackagePublishingHistory).find(
BinaryPackagePublishingHistory,
BinaryPackagePublishingHistory.status.is_in(
- [PUBLISHED, PENDING]),
+ active_publishing_status),
BinaryPackagePublishingHistory.distroarchseriesID.is_in(
available_architectures),
binarypackagerelease=self.binarypackagerelease,
@@ -1101,7 +1097,7 @@
return IMasterStore(BinaryPackagePublishingHistory).find(
BinaryPackagePublishingHistory,
BinaryPackagePublishingHistory.status.is_in(
- [PUBLISHED, PENDING]),
+ active_publishing_status),
BinaryPackagePublishingHistory.distroarchseries ==
self.distroarchseries,
binarypackagerelease=self.binarypackagerelease.debug_package,
@@ -1126,7 +1122,7 @@
self.distroarchseries.architecturetag))
return
- super(BinaryPackagePublishingHistory, self).supersede()
+ self.setSuperseded()
if dominant is not None:
# DDEBs cannot themselves be dominant; they are always dominated
=== added file 'lib/lp/soyuz/scripts/gina/dominate.py'
--- lib/lp/soyuz/scripts/gina/dominate.py 1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/scripts/gina/dominate.py 2011-09-09 09:33:30 +0000
@@ -0,0 +1,82 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Retirement of packages that are removed upstream."""
+
+__metaclass__ = type
+__all__ = [
+ 'dominate_imported_source_packages',
+ ]
+
+from zope.component import getUtility
+
+# XXX JeroenVermeulen 2011-09-08, bug=844550: The GeneralizedPublication
+# import violates import policy and elicits a warning from the test
+# suite. The warning helps remind us to retire this code as soon as
+# possible.
+from lp.archivepublisher.domination import (
+ Dominator,
+ GeneralizedPublication,
+ )
+from lp.registry.interfaces.distribution import IDistributionSet
+
+
+def dominate_imported_source_packages(logger, distro_name, series_name,
+ pocket, packages_map):
+ """Perform domination."""
+ series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
+ dominator = Dominator(logger, series.main_archive)
+
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a transitional
+ # hack. Gina used to create SPPHs in Pending state. We cleaned up
+ # the bulk of them, and changed the code to create Published ones, but
+ # some new ones will have been created since.
+ # Update those to match what the new Gina does.
+ from canonical.launchpad.interfaces.lpstorm import IStore
+ from lp.soyuz.enums import PackagePublishingStatus
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+ SPPH = SourcePackagePublishingHistory
+ store = IStore(SPPH)
+ spphs = store.find(
+ SPPH,
+ SPPH.archive == series.main_archive,
+ SPPH.distroseries == series,
+ SPPH.pocket == pocket,
+ SPPH.status == PackagePublishingStatus.PENDING)
+ spphs.set(status=PackagePublishingStatus.PUBLISHED)
+
+ # Dominate packages found in the Sources list we're importing.
+ package_names = dominator.findPublishedSourcePackageNames(series, pocket)
+ for package_name in package_names:
+ entries = packages_map.src_map.get(package_name)
+
+ if entries is None:
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a
+ # transitional hack. The database is full of "Published"
+ # Debian SPPHs whose packages have actually been deleted.
+ # In the future such publications should simply be marked
+ # Deleted, but for the legacy baggage we currently carry
+ # around we'll just do traditional domination first: pick
+ # the latest Published version, and mark the rest of the
+ # SPPHs as superseded by that version. The latest version
+ # will then, finally, be marked appropriately Deleted once
+ # we remove this transitional hack.
+ # To remove the transitional hack, just let live_versions
+ # default to the empty list instead of doing this:
+ pubs = dominator.findPublishedSPPHs(series, pocket, package_name)
+ generalization = GeneralizedPublication(is_source=True)
+ pubs_dict = dominator._sortPackages(pubs, generalization)
+ sorted_pubs = pubs_dict[package_name]
+ if len(sorted_pubs) <= 1:
+ # If there's only one published SPPH, the transitional
+ # code will just leave it Published. Don't bother; the
+ # migration will be costly enough as it is.
+ continue
+ live_versions = [sorted_pubs[0].sourcepackagerelease.version]
+ else:
+ live_versions = [
+ entry['Version']
+ for entry in entries if 'Version' in entry]
+
+ dominator.dominateRemovedSourceVersions(
+ series, pocket, package_name, live_versions)
=== modified file 'lib/lp/soyuz/scripts/gina/handlers.py'
--- lib/lp/soyuz/scripts/gina/handlers.py 2011-05-20 07:43:58 +0000
+++ lib/lp/soyuz/scripts/gina/handlers.py 2011-09-09 09:33:30 +0000
@@ -35,12 +35,12 @@
from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet
from canonical.launchpad.scripts import log
from lp.archivepublisher.diskpool import poolify
+from lp.archiveuploader.changesfile import ChangesFile
from lp.archiveuploader.tagfiles import parse_tagfile
from lp.archiveuploader.utils import (
determine_binary_file_type,
determine_source_file_type,
)
-from lp.archiveuploader.changesfile import ChangesFile
from lp.buildmaster.enums import BuildStatus
from lp.registry.interfaces.person import (
IPersonSet,
@@ -54,7 +54,10 @@
)
from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet
from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet
-from lp.soyuz.interfaces.publishing import IPublishingSet
+from lp.soyuz.interfaces.publishing import (
+ active_publishing_status,
+ IPublishingSet,
+ )
from lp.soyuz.model.component import Component
from lp.soyuz.model.files import (
BinaryPackageFile,
@@ -723,8 +726,6 @@
source_publishinghistory.status.title)
return
- # Create the Publishing entry, with status PENDING so that we
- # can republish this later into a Soyuz archive.
entry = getUtility(IPublishingSet).newSourcePublication(
distroseries=self.distroseries,
sourcepackagerelease=sourcepackagerelease,
@@ -732,6 +733,7 @@
section=section,
pocket=self.pocket,
archive=archive)
+ entry.setPublished()
log.info('Source package %s (%s) published' % (
entry.sourcepackagerelease.sourcepackagename.name,
entry.sourcepackagerelease.version))
@@ -742,16 +744,14 @@
from lp.soyuz.model.publishing import (
SourcePackagePublishingHistory)
- ret = SourcePackagePublishingHistory.select(
- """sourcepackagerelease = %s
- AND distroseries = %s
- AND archive = %s
- AND status in (%s, %s)""" %
- sqlvalues(sourcepackagerelease, self.distroseries,
- self.distroseries.main_archive,
- PackagePublishingStatus.PUBLISHED,
- PackagePublishingStatus.PENDING),
- orderBy=["-datecreated"])
+ ret = SourcePackagePublishingHistory.select("""
+ sourcepackagerelease = %s AND
+ distroseries = %s AND
+ archive = %s AND
+ status in %s""" % sqlvalues(
+ sourcepackagerelease, self.distroseries,
+ self.distroseries.main_archive, active_publishing_status),
+ orderBy=["-datecreated"])
ret = list(ret)
if ret:
return ret[0]
@@ -917,14 +917,6 @@
"for package %s (%s)" %
(build.id, binary.package, binary.version))
else:
-
- # XXX Debonzi 2005-05-16: Check it later
- # if bin.gpg_signing_key_owner:
- # key = self.getGPGKey(bin.gpg_signing_key,
- # *bin.gpg_signing_key_owner)
- # else:
- key = None
-
processor = distroarchinfo['processor']
build = getUtility(IBinaryPackageBuildSet).new(
processor=processor.id,
@@ -948,8 +940,7 @@
def publish(self, binarypackage, bpdata):
"""Create the publishing entry on db if does not exist."""
# Avoid circular imports.
- from lp.soyuz.model.publishing import (
- BinaryPackagePublishingHistory)
+ from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
# These need to be pulled from the binary package data, not the
# binary package release: the data represents data from /this
@@ -983,22 +974,20 @@
binpkg_publishinghistory.status.title)
return
-
- # Create the Publishing entry with status PENDING.
BinaryPackagePublishingHistory(
- binarypackagerelease = binarypackage.id,
- component = component.id,
- section = section.id,
- priority = priority,
- distroarchseries = self.distroarchseries.id,
- status = PackagePublishingStatus.PENDING,
- datecreated = UTC_NOW,
- datepublished = UTC_NOW,
- pocket = self.pocket,
- datesuperseded = None,
- supersededby = None,
- datemadepending = None,
- dateremoved = None,
+ binarypackagerelease=binarypackage.id,
+ component=component.id,
+ section=section.id,
+ priority=priority,
+ distroarchseries=self.distroarchseries.id,
+ status=PackagePublishingStatus.PUBLISHED,
+ datecreated=UTC_NOW,
+ datepublished=UTC_NOW,
+ pocket=self.pocket,
+ datesuperseded=None,
+ supersededby=None,
+ datemadepending=None,
+ dateremoved=None,
archive=archive)
log.info('BinaryPackage %s-%s published into %s.' % (
@@ -1008,19 +997,16 @@
def _checkPublishing(self, binarypackage):
"""Query for the publishing entry"""
# Avoid circular imports.
- from lp.soyuz.model.publishing import (
- BinaryPackagePublishingHistory)
+ from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
- ret = BinaryPackagePublishingHistory.select(
- """binarypackagerelease = %s
- AND distroarchseries = %s
- AND archive = %s
- AND status in (%s, %s)""" %
- sqlvalues(binarypackage, self.distroarchseries,
- self.distroarchseries.main_archive,
- PackagePublishingStatus.PUBLISHED,
- PackagePublishingStatus.PENDING),
- orderBy=["-datecreated"])
+ ret = BinaryPackagePublishingHistory.select("""
+ binarypackagerelease = %s AND
+ distroarchseries = %s AND
+ archive = %s AND
+ status in %s""" % sqlvalues(
+ binarypackage, self.distroarchseries,
+ self.distroarchseries.main_archive, active_publishing_status),
+ orderBy=["-datecreated"])
ret = list(ret)
if ret:
return ret[0]
=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
--- lib/lp/soyuz/scripts/tests/test_gina.py 2010-08-20 20:31:18 +0000
+++ lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-09 09:33:30 +0000
@@ -1,13 +1,181 @@
-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
# GNU Affero General Public License version 3 (see the file LICENSE).
from doctest import DocTestSuite
-import unittest
+from unittest import TestLoader
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.registry.interfaces.pocket import PackagePublishingPocket
+from lp.services.log.logger import DevNullLogger
+from lp.soyuz.enums import PackagePublishingStatus
+from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
import lp.soyuz.scripts.gina.handlers
+from lp.soyuz.scripts.gina.handlers import (
+ BinaryPackagePublisher,
+ SourcePackagePublisher,
+ )
+from lp.soyuz.scripts.gina.packages import (
+ BinaryPackageData,
+ SourcePackageData,
+ )
+from lp.testing import TestCaseWithFactory
+
+
+class FakePackagesMap:
+ def __init__(self, src_map):
+ self.src_map = src_map
+
+
+class TestGina(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def test_dominate_imported_source_packages_dominates_imports(self):
+ # dominate_imported_source_packages dominates the source
+ # packages that Gina imports.
+ logger = DevNullLogger()
+ pub = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ series = pub.distroseries
+ spr = pub.sourcepackagerelease
+ package = spr.sourcepackagename
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, pub.pocket,
+ FakePackagesMap({package.name: []}))
+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+ def test_dominate_imported_source_packages_dominates_deletions(self):
+ # dominate_imported_source_packages dominates the source
+ # packages that have been deleted from the Sources lists that
+ # Gina imports.
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ package = self.factory.makeSourcePackageName()
+ pubs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ archive=series.main_archive, distroseries=series,
+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package, version=version))
+ for version in ['1.0', '1.1', '1.1a']]
+ logger = DevNullLogger()
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, pocket,
+ FakePackagesMap({}))
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is
+ # "transitional" domination which supersedes older versions of
+ # deleted packages with the last known version. Permanent
+ # domination will then mark the last known version as deleted.
+ # For permanent domination, the expected outcome is that all
+ # these publications will be Deleted (but any pre-existing
+ # Superseded publications for older versions will remain
+ # Superseded).
+ self.assertEqual([
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ ],
+ [pub.status for pub in pubs])
+
+ def test_dominate_imported_source_packages_cleans_up_pending_spphs(self):
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
+ # Gina domination, dominate_imported_source_packages turns any
+ # remaining Pending SPPHS into Published ones.
+ series = self.factory.makeDistroSeries()
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ status=PackagePublishingStatus.PENDING)
+ spr = spph.sourcepackagerelease
+ package_name = spr.sourcepackagename.name
+ logger = DevNullLogger()
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, spph.pocket,
+ FakePackagesMap({package_name: [{"Version": spr.version}]}))
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
+
+ def test_dominate_imported_source_packages_cleans_up_first(self):
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
+ # Gina domination, dominate_imported_source_packages turns any
+ # remaining Pending SPPHS into Published ones. It does this
+ # *before* dominating, so no domination happens while some of
+ # the SPPHs are still mistakenly Pending (which would result in
+ # mistaken deletions).
+ series = self.factory.makeDistroSeries()
+ package = self.factory.makeSourcePackageName()
+ pocket = PackagePublishingPocket.RELEASE
+ versions = ['1.0', '1.1']
+ statuses_before = [
+ PackagePublishingStatus.PUBLISHED,
+ PackagePublishingStatus.PENDING,
+ ]
+ statuses_after = [
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ ]
+ live_version = versions[-1]
+ sprs = [
+ self.factory.makeSourcePackageRelease(
+ sourcepackagename=package, version=version)
+ for version in versions]
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ archive=series.main_archive, distroseries=series,
+ sourcepackagerelease=spr, pocket=pocket, status=status)
+ for spr, status in zip(sprs, statuses_before)]
+
+ logger = DevNullLogger()
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, pocket,
+ FakePackagesMap({package.name: [{"Version": live_version}]}))
+
+ self.assertEqual(statuses_after, [spph.status for spph in spphs])
+
+
+class TestSourcePackagePublisher(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def test_publish_creates_published_publication(self):
+ maintainer = self.factory.makePerson()
+ series = self.factory.makeDistroSeries()
+ section = self.factory.makeSection()
+ pocket = PackagePublishingPocket.RELEASE
+ spr = self.factory.makeSourcePackageRelease()
+
+ publisher = SourcePackagePublisher(series, pocket, None)
+ publisher.publish(spr, SourcePackageData(
+ component='main', section=section.name, version='1.0',
+ maintainer=maintainer.preferredemail, architecture='all',
+ files='foo.py', binaries='foo.py'))
+
+ [spph] = series.main_archive.getPublishedSources()
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
+
+
+class TestBinaryPackagePublisher(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def test_publish_creates_published_publication(self):
+ maintainer = self.factory.makePerson()
+ series = self.factory.makeDistroArchSeries()
+ section = self.factory.makeSection()
+ pocket = PackagePublishingPocket.RELEASE
+ bpr = self.factory.makeBinaryPackageRelease()
+
+ publisher = BinaryPackagePublisher(series, pocket, None)
+ publisher.publish(bpr, BinaryPackageData(
+ component='main', section=section.name, version='1.0',
+ maintainer=maintainer.preferredemail, architecture='all',
+ files='foo.py', binaries='foo.py', size=128, installed_size=1024,
+ md5sum='e83b5dd68079d727a494a469d40dc8db', description='test',
+ summary='Test!'))
+
+ [bpph] = series.main_archive.getAllPublishedBinaries()
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, bpph.status)
def test_suite():
- suite = unittest.TestSuite()
+ suite = TestLoader().loadTestsFromName(__name__)
suite.addTest(DocTestSuite(lp.soyuz.scripts.gina.handlers))
return suite
=== modified file 'scripts/gina.py'
--- scripts/gina.py 2011-08-23 08:35:13 +0000
+++ scripts/gina.py 2011-09-09 09:33:30 +0000
@@ -38,6 +38,7 @@
MangledArchiveError,
PackagesMap,
)
+from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
from lp.soyuz.scripts.gina.handlers import (
DataSetupError,
ImporterHandler,
@@ -152,6 +153,10 @@
packages_map, kdb, package_root, keyrings, importer_handler)
importer_handler.commit()
+ # XXX JeroenVermeulen 2011-09-07 bug=843728: Dominate binaries as well.
+ dominate_imported_source_packages(
+ log, distro, distroseries, pocket, packages_map)
+
if source_only:
log.info('Source only mode... done')
return
@@ -209,9 +214,8 @@
npacks = len(packages_map.src_map)
log.info('%i Source Packages to be imported', npacks)
- for list_source in sorted(
- packages_map.src_map.values(), key=lambda x: x[0].get("Package")):
- for source in list_source:
+ for package in sorted(packages_map.src_map.iterkeys()):
+ for source in packages_map.src_map[package]:
count += 1
attempt_source_package_import(
source, kdb, package_root, keyrings, importer_handler)
@@ -244,10 +248,9 @@
log.info(
'%i Binary Packages to be imported for %s', npacks, archtag)
# Go over binarypackages importing them for this architecture
- for binary in sorted(packages_map.bin_map[archtag].values(),
- key=lambda x: x.get("Package")):
+ for package_name in sorted(packages_map.bin_map[archtag].iterkeys()):
+ binary = packages_map.bin_map[archtag][package_name]
count += 1
- package_name = binary.get("Package", "unknown")
try:
try:
do_one_binarypackage(binary, archtag, kdb, package_root,
