launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #04901
[Merge] lp:~jtv/launchpad/bug-844577 into lp:launchpad
Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/bug-844577 into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #844577 in Launchpad itself: "Gina imports should be Published, not Pending"
https://bugs.launchpad.net/launchpad/+bug/844577
For more details, see:
https://code.launchpad.net/~jtv/launchpad/bug-844577/+merge/74567
= Summary =
When Gina imports a package version (which would currently have to be a Debian source package release, but could be an Ubuntu-derived source or binary package release), it creates a BinaryPackagePublishingHistory or SourcePackagePublishingHistory (as the case may be) in Pending state. So basically all the Debian package releases we ever imported from Debian are basically sitting in Hades, waiting patiently for their time to return to the land of the living.
That time has come. We're making Gina run the Dominator (it's a long story) and this will only work on Published records.
== Proposed fix ==
Make Gina produce Published records instead. We'll also have to do some in-DB updates. Or more precisely, vast numbers of them.
== Pre-implementation notes ==
Discussed with William. The Pending status only made sense for the original Ubuntu import into Launchpad. Now, Published is the more appropriate status. (In practice it hasn't mattered much until now; you could call this part of a tech-debt fix).
== Implementation details ==
The way it's done is nastily asymmetric between sources and binaries: in the source case we use the IPublishingSet utility to create a Pending SPPH, and then use its API to make it Published. In the binary case, the status is passed directly to the constructor. But if Soyuz people like it this way, this is not the time to argue.
You'll also note a few minor (and largely irrelevant) cleanups of how statuses are mentioned: some unused variables that were used as shorthand for PackagePublishingStatus.* but which were not in use, and some literal spellings of what is clearly meant to be active_publishing_status. No functional changes. There's also an interesting 6-year-old XXX from someone who doesn't work here any more, about how to initialize a variable that wasn't being used. If we haven't noticed any problems with it, either things are fine as they are or the comment clearly isn't doing the trick.
== Tests ==
{{{
./bin/test -vvc lp.soyuz.scripts.tests.test_gina
}}}
== Demo and Q/A ==
Running Gina should still work. But now it will create SPPHs and BPPHs in Published status, rather than Pending.
= Launchpad lint =
Checking for conflicts and issues in changed files.
Linting changed files:
lib/lp/archivepublisher/domination.py
lib/lp/soyuz/scripts/gina/dominate.py
lib/lp/soyuz/model/publishing.py
lib/lp/soyuz/interfaces/publishing.py
scripts/gina.py
lib/lp/archivepublisher/tests/test_dominator.py
lib/lp/soyuz/scripts/tests/test_gina.py
lib/lp/soyuz/scripts/gina/handlers.py
./lib/lp/soyuz/interfaces/publishing.py
381: E261 at least two spaces before inline comment
478: E261 at least two spaces before inline comment
511: E261 at least two spaces before inline comment
681: E261 at least two spaces before inline comment
767: E261 at least two spaces before inline comment
./scripts/gina.py
26: '_pythonpath' imported but unused
--
https://code.launchpad.net/~jtv/launchpad/bug-844577/+merge/74567
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/bug-844577 into lp:launchpad.
=== modified file 'lib/lp/archivepublisher/domination.py'
--- lib/lp/archivepublisher/domination.py 2011-08-30 06:37:55 +0000
+++ lib/lp/archivepublisher/domination.py 2011-09-08 09:37:30 +0000
@@ -53,8 +53,6 @@
__all__ = ['Dominator']
from datetime import timedelta
-import functools
-import operator
import apt_pkg
from storm.expr import (
@@ -68,7 +66,7 @@
flush_database_updates,
sqlvalues,
)
-from canonical.launchpad.interfaces.lpstorm import IMasterStore
+from canonical.launchpad.interfaces.lpstorm import IStore
from lp.registry.model.sourcepackagename import SourcePackageName
from lp.soyuz.enums import (
BinaryPackageFormat,
@@ -87,17 +85,93 @@
apt_pkg.InitSystem()
-def _compare_packages_by_version_and_date(get_release, p1, p2):
- """Compare publications p1 and p2 by their version; using Debian rules.
-
- If the publications are for the same package, compare by datecreated
- instead. This lets newer records win.
- """
- if get_release(p1).id == get_release(p2).id:
- return cmp(p1.datecreated, p2.datecreated)
-
- return apt_pkg.VersionCompare(get_release(p1).version,
- get_release(p2).version)
+def join_spr_spn():
+ """Join condition: SourcePackageRelease/SourcePackageName."""
+ return (
+ SourcePackageName.id == SourcePackageRelease.sourcepackagenameID)
+
+
+def join_spph_spr():
+ """Join condition: SourcePackageRelease/SourcePackagePublishingHistory.
+ """
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return (
+ SourcePackageRelease.id ==
+ SourcePackagePublishingHistory.sourcepackagereleaseID)
+
+
+class SourcePublicationTraits:
+ """Basic generalized attributes for `SourcePackagePublishingHistory`.
+
+ Used by `GeneralizedPublication` to hide the differences from
+ `BinaryPackagePublishingHistory`.
+ """
+ @staticmethod
+ def getPackageName(spph):
+ """Return the name of this publication's source package."""
+ return spph.sourcepackagerelease.sourcepackagename.name
+
+ @staticmethod
+ def getPackageRelease(spph):
+ """Return this publication's `SourcePackageRelease`."""
+ return spph.sourcepackagerelease
+
+
+class BinaryPublicationTraits:
+ """Basic generalized attributes for `BinaryPackagePublishingHistory`.
+
+ Used by `GeneralizedPublication` to hide the differences from
+ `SourcePackagePublishingHistory`.
+ """
+ @staticmethod
+ def getPackageName(bpph):
+ """Return the name of this publication's binary package."""
+ return bpph.binarypackagerelease.binarypackagename.name
+
+ @staticmethod
+ def getPackageRelease(bpph):
+ """Return this publication's `BinaryPackageRelease`."""
+ return bpph.binarypackagerelease
+
+
+class GeneralizedPublication:
+ """Generalize handling of publication records.
+
+ This allows us to write code that can be dealing with either
+ `SourcePackagePublishingHistory`s or `BinaryPackagePublishingHistory`s
+ without caring which. Differences are abstracted away in a traits
+ class.
+ """
+ def __init__(self, is_source=True):
+ if is_source:
+ self.traits = SourcePublicationTraits
+ else:
+ self.traits = BinaryPublicationTraits
+
+ def getPackageName(self, pub):
+ """Get the package's name."""
+ return self.traits.getPackageName(pub)
+
+ def getPackageVersion(self, pub):
+ """Obtain the version string for a publicaiton record."""
+ return self.traits.getPackageRelease(pub).version
+
+ def compare(self, pub1, pub2):
+ """Compare publications by version.
+
+ If both publications are for the same version, their creation dates
+ break the tie.
+ """
+ version_comparison = apt_pkg.VersionCompare(
+ self.getPackageVersion(pub1), self.getPackageVersion(pub2))
+
+ if version_comparison == 0:
+ # Use dates as tie breaker.
+ return cmp(pub1.datecreated, pub2.datecreated)
+ else:
+ return version_comparison
class Dominator:
@@ -116,50 +190,95 @@
self.logger = logger
self.archive = archive
- def _dominatePublications(self, pubs):
+ def dominatePackage(self, publications, live_versions, generalization):
+ """Dominate publications for a single package.
+
+ Active publications for versions in `live_versions` stay active.
+ Any older versions are marked as superseded by the respective
+ oldest live versions that are newer than the superseded ones.
+
+ Any versions that are newer than anything in `live_versions` are
+ marked as deleted. This should not be possible in Soyuz-native
+ archives, but it can happen during archive imports when the
+ previous latest version of a package has disappeared from the Sources
+ list we import.
+
+ :param publications: Iterable of publications for the same package,
+ in the same archive, series, and pocket, all with status
+ `PackagePublishingStatus.PUBLISHED`.
+ :param live_versions: Iterable of version strings that are still
+ considered live for this package. The given publications will
+ remain active insofar as they represent any of these versions;
+ older publications will be marked as superseded and newer ones
+ as deleted.
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
+ """
+ # Go through publications from latest version to oldest. This
+ # makes it easy to figure out which release superseded which:
+ # the dominant is always the oldest live release that is newer
+ # than the one being superseded.
+ publications = sorted(
+ publications, cmp=generalization.compare, reverse=True)
+
+ current_dominant = None
+ for pub in publications:
+ if generalization.getPackageVersion(pub) in live_versions:
+ # This publication stays active; if any publications
+ # that follow right after this are to be superseded,
+ # this is the release that they are superseded by.
+ current_dominant = pub
+ elif current_dominant is None:
+ # This publication is no longer live, but there is no
+ # newer version to supersede it either. Therefore it
+ # must be deleted.
+ pub.requestDeletion(None)
+ else:
+ # This publication is superseded. This is what we're
+ # here to do.
+ pub.supersede(current_dominant, logger=self.logger)
+
+ def _dominatePublications(self, pubs, generalization):
"""Perform dominations for the given publications.
+ Keep the latest published version for each package active,
+ superseding older versions.
+
:param pubs: A dict mapping names to a list of publications. Every
publication must be PUBLISHED or PENDING, and the first in each
list will be treated as dominant (so should be the latest).
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
"""
self.logger.debug("Dominating packages...")
-
- for name in pubs.keys():
- assert pubs[name], (
- "Empty list of publications for %s" % name)
- for pubrec in pubs[name][1:]:
- pubrec.supersede(pubs[name][0], logger=self.logger)
-
- def _sortPackages(self, pkglist, is_source=True):
+ for name, publications in pubs.iteritems():
+ assert publications, "Empty list of publications for %s." % name
+ # Since this always picks the latest version as the live
+ # one, this dominatePackage call will never result in a
+ # deletion.
+ latest_version = generalization.getPackageVersion(publications[0])
+ self.dominatePackage(
+ publications, [latest_version], generalization)
+
+ def _sortPackages(self, pkglist, generalization):
"""Map out packages by name, and sort by descending version.
:param pkglist: An iterable of `SourcePackagePublishingHistory` or
`BinaryPackagePublishingHistory`.
- :param is_source: Whether this call involves source package
- publications. If so, work with `SourcePackagePublishingHistory`.
- If not, work with `BinaryPackagepublishingHistory`.
- :return: A dict mapping each package name (as UTF-8 encoded string)
- to a list of publications from `pkglist`, newest first.
+ :param generalization: A `GeneralizedPublication` helper representing
+ the kind of publications these are--source or binary.
+ :return: A dict mapping each package name to a list of publications
+ from `pkglist`, newest first.
"""
self.logger.debug("Sorting packages...")
- if is_source:
- get_release = operator.attrgetter("sourcepackagerelease")
- get_name = operator.attrgetter("sourcepackagename")
- else:
- get_release = operator.attrgetter("binarypackagerelease")
- get_name = operator.attrgetter("binarypackagename")
-
outpkgs = {}
for inpkg in pkglist:
- key = get_name(get_release(inpkg)).name.encode('utf-8')
+ key = generalization.getPackageName(inpkg)
outpkgs.setdefault(key, []).append(inpkg)
- sort_order = functools.partial(
- _compare_packages_by_version_and_date, get_release)
for package_pubs in outpkgs.itervalues():
- package_pubs.sort(cmp=sort_order, reverse=True)
+ package_pubs.sort(cmp=generalization.compare, reverse=True)
return outpkgs
@@ -287,6 +406,8 @@
# Avoid circular imports.
from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
+ generalization = GeneralizedPublication(is_source=False)
+
for distroarchseries in distroseries.architectures:
self.logger.debug(
"Performing domination across %s/%s (%s)",
@@ -312,7 +433,7 @@
),
group_by=BinaryPackageName.id,
having=Count(BinaryPackagePublishingHistory.id) > 1)
- binaries = IMasterStore(BinaryPackagePublishingHistory).find(
+ binaries = IStore(BinaryPackagePublishingHistory).find(
BinaryPackagePublishingHistory,
BinaryPackageRelease.id ==
BinaryPackagePublishingHistory.binarypackagereleaseID,
@@ -322,7 +443,21 @@
BinaryPackageFormat.DDEB,
bpph_location_clauses)
self.logger.debug("Dominating binaries...")
- self._dominatePublications(self._sortPackages(binaries, False))
+ self._dominatePublications(
+ self._sortPackages(binaries, generalization), generalization)
+
+ def _composeActiveSourcePubsCondition(self, distroseries, pocket):
+ """Compose ORM condition for restricting relevant source pubs."""
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return And(
+ SourcePackagePublishingHistory.status ==
+ PackagePublishingStatus.PUBLISHED,
+ SourcePackagePublishingHistory.distroseries == distroseries,
+ SourcePackagePublishingHistory.archive == self.archive,
+ SourcePackagePublishingHistory.pocket == pocket,
+ )
def dominateSources(self, distroseries, pocket):
"""Perform domination on source package publications.
@@ -332,38 +467,76 @@
"""
# Avoid circular imports.
from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ generalization = GeneralizedPublication(is_source=True)
+
self.logger.debug(
"Performing domination across %s/%s (Source)",
distroseries.name, pocket.title)
- spph_location_clauses = And(
- SourcePackagePublishingHistory.status ==
- PackagePublishingStatus.PUBLISHED,
- SourcePackagePublishingHistory.distroseries == distroseries,
- SourcePackagePublishingHistory.archive == self.archive,
- SourcePackagePublishingHistory.pocket == pocket,
- )
+
+ spph_location_clauses = self._composeActiveSourcePubsCondition(
+ distroseries, pocket)
+ having_multiple_active_publications = (
+ Count(SourcePackagePublishingHistory.id) > 1)
candidate_source_names = Select(
SourcePackageName.id,
- And(
- SourcePackageRelease.sourcepackagenameID ==
- SourcePackageName.id,
- SourcePackagePublishingHistory.sourcepackagereleaseID ==
- SourcePackageRelease.id,
- spph_location_clauses,
- ),
+ And(join_spph_spr(), join_spr_spn(), spph_location_clauses),
group_by=SourcePackageName.id,
- having=Count(SourcePackagePublishingHistory.id) > 1)
- sources = IMasterStore(SourcePackagePublishingHistory).find(
+ having=having_multiple_active_publications)
+ sources = IStore(SourcePackagePublishingHistory).find(
SourcePackagePublishingHistory,
- SourcePackageRelease.id ==
- SourcePackagePublishingHistory.sourcepackagereleaseID,
+ join_spph_spr(),
SourcePackageRelease.sourcepackagenameID.is_in(
candidate_source_names),
spph_location_clauses)
+
self.logger.debug("Dominating sources...")
- self._dominatePublications(self._sortPackages(sources))
+ self._dominatePublications(
+ self._sortPackages(sources, generalization), generalization)
flush_database_updates()
+ def findPublishedSourcePackageNames(self, distroseries, pocket):
+ """Find names of currently published source packages."""
+ result = IStore(SourcePackageName).find(
+ SourcePackageName.name,
+ join_spph_spr(),
+ join_spr_spn(),
+ self._composeActiveSourcePubsCondition(distroseries, pocket))
+ return result.config(distinct=True)
+
+ def findPublishedSPPHs(self, distroseries, pocket, package_name):
+ """Find currently published source publications for given package."""
+ # Avoid circular imports.
+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+ return IStore(SourcePackagePublishingHistory).find(
+ SourcePackagePublishingHistory,
+ join_spph_spr(),
+ join_spr_spn(),
+ SourcePackageName.name == package_name,
+ self._composeActiveSourcePubsCondition(distroseries, pocket))
+
+ def dominateRemovedSourceVersions(self, distroseries, pocket,
+ package_name, live_versions):
+ """Dominate source publications based on a set of "live" versions.
+
+ Active publications for the "live" versions will remain active. All
+ other active publications for the same package (and the same archive,
+ distroseries, and pocket) are marked superseded.
+
+ Unlike traditional domination, this allows multiple versions of a
+ package to stay active in the same distroseries, archive, and pocket.
+
+ :param distroseries: `DistroSeries` to dominate.
+ :param pocket: `PackagePublishingPocket` to dominate.
+ :param package_name: Source package name, as text.
+ :param live_versions: Iterable of all version strings that are to
+ remain active.
+ """
+ generalization = GeneralizedPublication(is_source=True)
+ pubs = self.findPublishedSPPHs(distroseries, pocket, package_name)
+ self.dominatePackage(pubs, live_versions, generalization)
+
def judge(self, distroseries, pocket):
"""Judge superseded sources and binaries."""
# Avoid circular imports.
=== modified file 'lib/lp/archivepublisher/tests/test_dominator.py'
--- lib/lp/archivepublisher/tests/test_dominator.py 2011-02-04 05:11:00 +0000
+++ lib/lp/archivepublisher/tests/test_dominator.py 2011-09-08 09:37:30 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
# GNU Affero General Public License version 3 (see the file LICENSE).
"""Tests for domination.py."""
@@ -7,12 +7,24 @@
import datetime
+import apt_pkg
+from zope.security.proxy import removeSecurityProxy
+
from canonical.database.sqlbase import flush_database_updates
-from lp.archivepublisher.domination import Dominator, STAY_OF_EXECUTION
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.archivepublisher.domination import (
+ Dominator,
+ GeneralizedPublication,
+ STAY_OF_EXECUTION,
+ )
from lp.archivepublisher.publishing import Publisher
+from lp.registry.interfaces.pocket import PackagePublishingPocket
from lp.registry.interfaces.series import SeriesStatus
+from lp.services.log.logger import DevNullLogger
from lp.soyuz.enums import PackagePublishingStatus
+from lp.soyuz.interfaces.publishing import ISourcePackagePublishingHistory
from lp.soyuz.tests.test_publishing import TestNativePublishingBase
+from lp.testing import TestCaseWithFactory
class TestDominator(TestNativePublishingBase):
@@ -50,6 +62,8 @@
foo_10_source, foo_10_binaries[0])
def dominateAndCheck(self, dominant, dominated, supersededby):
+ generalization = GeneralizedPublication(
+ is_source=ISourcePackagePublishingHistory.providedBy(dominant))
dominator = Dominator(self.logger, self.ubuntutest.main_archive)
# The _dominate* test methods require a dictionary where the
@@ -58,7 +72,7 @@
# and dominated, the subsequents.
pubs = {'foo': [dominant, dominated]}
- dominator._dominatePublications(pubs)
+ dominator._dominatePublications(pubs, generalization)
flush_database_updates()
# The dominant version remains correctly published.
@@ -145,7 +159,9 @@
# This isn't a really good exception. It should probably be
# something more indicative of bad input.
self.assertRaises(
- AssertionError, dominator._dominatePublications, pubs)
+ AssertionError,
+ dominator._dominatePublications,
+ pubs, GeneralizedPublication(True))
class TestDomination(TestNativePublishingBase):
@@ -200,3 +216,398 @@
TestDomination.setUp(self)
self.ubuntutest['breezy-autotest'].status = (
SeriesStatus.OBSOLETE)
+
+
+def make_spphs_for_versions(factory, versions):
+ """Create publication records for each of `versions`.
+
+ They records are created in the same order in which they are specified.
+ Make the order irregular to prove that version ordering is not a
+ coincidence of object creation order etc.
+
+ Versions may also be identical; each publication record will still have
+ its own package release.
+ """
+ spn = factory.makeSourcePackageName()
+ distroseries = factory.makeDistroSeries()
+ pocket = factory.getAnyPocket()
+ sprs = [
+ factory.makeSourcePackageRelease(
+ sourcepackagename=spn, version=version)
+ for version in versions]
+ return [
+ factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=pocket,
+ sourcepackagerelease=spr,
+ status=PackagePublishingStatus.PUBLISHED)
+ for spr in sprs]
+
+
+def list_source_versions(spphs):
+ """Extract the versions from `spphs` as a list, in the same order."""
+ return [spph.sourcepackagerelease.version for spph in spphs]
+
+
+class TestGeneralizedPublication(TestCaseWithFactory):
+ """Test publication generalization helpers."""
+
+ layer = ZopelessDatabaseLayer
+
+ def alterCreationDates(self, spphs, ages):
+ """Set `datecreated` on each of `spphs` according to `ages`.
+
+ :param spphs: Iterable of `SourcePackagePublishingHistory`. Their
+ respective creation dates will be offset by the respective ages
+ found in `ages` (with the two being matched up in the same order).
+ :param ages: Iterable of ages. Must provide the same number of items
+ as `spphs`. Ages are `timedelta` objects that will be subtracted
+ from the creation dates on the respective records in `spph`.
+ """
+ for spph, age in zip(spphs, ages):
+ spph.datecreated -= age
+
+ def test_getPackageVersion_gets_source_version(self):
+ spph = self.factory.makeSourcePackagePublishingHistory()
+ self.assertEqual(
+ spph.sourcepackagerelease.version,
+ GeneralizedPublication(is_source=True).getPackageVersion(spph))
+
+ def test_getPackageVersion_gets_binary_version(self):
+ bpph = self.factory.makeBinaryPackagePublishingHistory()
+ self.assertEqual(
+ bpph.binarypackagerelease.version,
+ GeneralizedPublication(is_source=False).getPackageVersion(bpph))
+
+ def test_compare_sorts_versions(self):
+ versions = [
+ '1.1v2',
+ '1.1v1',
+ '1.1v3',
+ ]
+ spphs = make_spphs_for_versions(self.factory, versions)
+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+ self.assertEqual(
+ sorted(versions), list_source_versions(sorted_spphs))
+
+ def test_compare_orders_versions_by_debian_rules(self):
+ versions = [
+ '1.1.0',
+ '1.10',
+ '1.1',
+ '1.1ubuntu0',
+ ]
+ spphs = make_spphs_for_versions(self.factory, versions)
+
+ debian_sorted_versions = sorted(versions, cmp=apt_pkg.VersionCompare)
+
+ # Assumption: in this case, Debian version ordering is not the
+ # same as alphabetical version ordering.
+ self.assertNotEqual(sorted(versions), debian_sorted_versions)
+
+ # The compare method produces the Debian ordering.
+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+ self.assertEqual(
+ sorted(versions, cmp=apt_pkg.VersionCompare),
+ list_source_versions(sorted_spphs))
+
+ def test_compare_breaks_tie_with_creation_date(self):
+ # When two publications are tied for comparison because they are
+ # for the same package release, they are ordered by creation
+ # date.
+ distroseries = self.factory.makeDistroSeries()
+ pocket = self.factory.getAnyPocket()
+ spr = self.factory.makeSourcePackageRelease()
+ ages = [
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ datetime.timedelta(3),
+ ]
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ sourcepackagerelease=spr, distroseries=distroseries,
+ pocket=pocket)
+ for counter in xrange(len(ages))]
+ self.alterCreationDates(spphs, ages)
+
+ self.assertEqual(
+ [spphs[2], spphs[0], spphs[1]],
+ sorted(spphs, cmp=GeneralizedPublication().compare))
+
+ def test_compare_breaks_tie_for_releases_with_same_version(self):
+ # When two publications are tied for comparison because they
+ # belong to releases with the same version string, they are
+ # ordered by creation date.
+ version = "1.%d" % self.factory.getUniqueInteger()
+ ages = [
+ datetime.timedelta(2),
+ datetime.timedelta(1),
+ datetime.timedelta(3),
+ ]
+ distroseries = self.factory.makeDistroSeries()
+ pocket = self.factory.getAnyPocket()
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=pocket,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ version=version))
+ for counter in xrange(len(ages))]
+ self.alterCreationDates(spphs, ages)
+
+ self.assertEqual(
+ [spphs[2], spphs[0], spphs[1]],
+ sorted(spphs, cmp=GeneralizedPublication().compare))
+
+
+class TestDominatorMethods(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def makeDominator(self, publications):
+ """Create a `Dominator` suitable for `publications`."""
+ if len(publications) == 0:
+ archive = self.factory.makeArchive()
+ else:
+ archive = publications[0].archive
+ return Dominator(DevNullLogger(), archive)
+
+ def test_dominatePackage_survives_empty_publications_list(self):
+ # Nothing explodes when dominatePackage is called with an empty
+ # packages list.
+ self.makeDominator([]).dominatePackage(
+ [], [], GeneralizedPublication(True))
+ # The test is that we get here without error.
+ pass
+
+ def test_dominatePackage_leaves_live_version_untouched(self):
+ # dominatePackage does not supersede live versions.
+ [pub] = make_spphs_for_versions(self.factory, ['3.1'])
+ self.makeDominator([pub]).dominatePackage(
+ [pub], ['3.1'], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pub.status)
+
+ def test_dominatePackage_deletes_dead_version_without_successor(self):
+ # dominatePackage marks non-live package versions without
+ # superseding versions as deleted.
+ [pub] = make_spphs_for_versions(self.factory, ['1.1'])
+ self.makeDominator([pub]).dominatePackage(
+ [pub], [], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+ def test_dominatePackage_supersedes_older_pub_with_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage
+ # designates a newer live version as the superseding version.
+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['1.1'], GeneralizedPublication(True))
+ self.assertEqual(PackagePublishingStatus.SUPERSEDED, pubs[0].status)
+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+ def test_dominatePackage_only_supersedes_with_live_pub(self):
+ # When marking a package as superseded, dominatePackage will
+ # only pick a live version as the superseding one.
+ pubs = make_spphs_for_versions(
+ self.factory, ['1.0', '2.0', '3.0', '4.0'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['3.0'], GeneralizedPublication(True))
+ self.assertEqual([
+ pubs[2].sourcepackagerelease,
+ pubs[2].sourcepackagerelease,
+ None,
+ None,
+ ],
+ [pub.supersededby for pub in pubs])
+
+ def test_dominatePackage_supersedes_with_oldest_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage picks
+ # the oldest of the newer, live versions as the superseding one.
+ pubs = make_spphs_for_versions(self.factory, ['2.7', '2.8', '2.9'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['2.8', '2.9'], GeneralizedPublication(True))
+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+
+ def test_dominatePackage_only_supersedes_with_newer_live_pub(self):
+ # When marking a package as superseded, dominatePackage only
+ # considers a newer version as the superseding one.
+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2'])
+ self.makeDominator(pubs).dominatePackage(
+ pubs, ['0.1'], GeneralizedPublication(True))
+ self.assertEqual(None, pubs[1].supersededby)
+ self.assertEqual(PackagePublishingStatus.DELETED, pubs[1].status)
+
+ def test_dominateRemovedSourceVersions_dominates_publications(self):
+ # dominateRemovedSourceVersions finds the publications for a
+ # package and calls dominatePackage on them.
+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2', '0.3'])
+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, package_name, ['0.2'])
+ self.assertEqual([
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ PackagePublishingStatus.DELETED,
+ ],
+ [pub.status for pub in pubs])
+ self.assertEqual(
+ [pubs[1].sourcepackagerelease, None, None],
+ [pub.supersededby for pub in pubs])
+
+ def test_dominateRemovedSourceVersions_ignores_other_pockets(self):
+ # dominateRemovedSourceVersions ignores publications in other
+ # pockets than the one specified.
+ pubs = make_spphs_for_versions(self.factory, ['2.3', '2.4'])
+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+ removeSecurityProxy(pubs[0]).pocket = PackagePublishingPocket.UPDATES
+ removeSecurityProxy(pubs[1]).pocket = PackagePublishingPocket.PROPOSED
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, package_name, ['2.3'])
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+ def test_dominateRemovedSourceVersions_ignores_other_packages(self):
+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+ other_package_name = self.factory.makeSourcePackageName().name
+ self.makeDominator(pubs).dominateRemovedSourceVersions(
+ pubs[0].distroseries, pubs[0].pocket, other_package_name, ['1.1'])
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[0].status)
+
+ def test_findPublishedSourcePackageNames_finds_package(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [spph.sourcepackagerelease.sourcepackagename.name],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_states(self):
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ spphs = dict(
+ (status, self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=status))
+ for status in PackagePublishingStatus.items)
+ published_spph = spphs[PackagePublishingStatus.PUBLISHED]
+ dominator = self.makeDominator(spphs.values())
+ self.assertContentEqual(
+ [published_spph.sourcepackagerelease.sourcepackagename.name],
+ dominator.findPublishedSourcePackageNames(series, pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_archives(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ dominator = self.makeDominator([spph])
+ dominator.archive = self.factory.makeArchive()
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_series(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ distro = spph.distroseries.distribution
+ other_series = self.factory.makeDistroSeries(distribution=distro)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ other_series, spph.pocket))
+
+ def test_findPublishedSourcePackageNames_ignores_other_pockets(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED,
+ pocket=PackagePublishingPocket.RELEASE)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSourcePackageNames(
+ spph.distroseries, PackagePublishingPocket.SECURITY))
+
+ def test_findPublishedSourcePackageNames_does_not_return_duplicates(self):
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ package = self.factory.makeSourcePackageName()
+ spphs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package))
+ for counter in xrange(2)]
+ dominator = self.makeDominator(spphs)
+ self.assertEqual(
+ [package.name],
+ list(dominator.findPublishedSourcePackageNames(series, pocket)))
+
+ def test_findPublishedSPPHs_finds_published_SPPH(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ package_name = spph.sourcepackagerelease.sourcepackagename.name
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [spph],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, package_name))
+
+ def test_findPublishedSPPHs_ignores_other_states(self):
+ series = self.factory.makeDistroSeries()
+ package = self.factory.makeSourcePackageName()
+ pocket = PackagePublishingPocket.RELEASE
+ spphs = dict(
+ (status, self.factory.makeSourcePackagePublishingHistory(
+ distroseries=series, archive=series.main_archive,
+ pocket=pocket, status=status,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package)))
+ for status in PackagePublishingStatus.items)
+ dominator = self.makeDominator(spphs.values())
+ self.assertContentEqual(
+ [spphs[PackagePublishingStatus.PUBLISHED]],
+ dominator.findPublishedSPPHs(series, pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_archives(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ package = spph.sourcepackagerelease.sourcepackagename
+ dominator = self.makeDominator([spph])
+ dominator.archive = self.factory.makeArchive()
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_series(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ distro = spph.distroseries.distribution
+ package = spph.sourcepackagerelease.sourcepackagename
+ other_series = self.factory.makeDistroSeries(distribution=distro)
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ other_series, spph.pocket, package.name))
+
+ def test_findPublishedSPPHs_ignores_other_pockets(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED,
+ pocket=PackagePublishingPocket.RELEASE)
+ package = spph.sourcepackagerelease.sourcepackagename
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, PackagePublishingPocket.SECURITY,
+ package.name))
+
+ def test_findPublishedSPPHs_ignores_other_packages(self):
+ spph = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ other_package = self.factory.makeSourcePackageName()
+ dominator = self.makeDominator([spph])
+ self.assertContentEqual(
+ [],
+ dominator.findPublishedSPPHs(
+ spph.distroseries, spph.pocket, other_package.name))
=== modified file 'lib/lp/soyuz/interfaces/publishing.py'
--- lib/lp/soyuz/interfaces/publishing.py 2011-09-02 04:51:25 +0000
+++ lib/lp/soyuz/interfaces/publishing.py 2011-09-08 09:37:30 +0000
@@ -195,9 +195,6 @@
the field name and value is the value string.
"""
- def supersede():
- """Supersede this publication."""
-
def requestObsolescence():
"""Make this publication obsolete.
=== modified file 'lib/lp/soyuz/model/publishing.py'
--- lib/lp/soyuz/model/publishing.py 2011-08-31 04:40:44 +0000
+++ lib/lp/soyuz/model/publishing.py 2011-09-08 09:37:30 +0000
@@ -114,10 +114,6 @@
from lp.soyuz.scripts.changeoverride import ArchiveOverriderError
-PENDING = PackagePublishingStatus.PENDING
-PUBLISHED = PackagePublishingStatus.PUBLISHED
-
-
# XXX cprov 2006-08-18: move it away, perhaps archivepublisher/pool.py
def makePoolPath(source_name, component_name):
@@ -327,8 +323,8 @@
fields = self.buildIndexStanzaFields()
return fields.makeOutput()
- def supersede(self):
- """See `IPublishing`."""
+ def setSuperseded(self):
+ """Set to SUPERSEDED status."""
self.status = PackagePublishingStatus.SUPERSEDED
self.datesuperseded = UTC_NOW
@@ -742,7 +738,7 @@
"Should not dominate unpublished source %s" %
self.sourcepackagerelease.title)
- super(SourcePackagePublishingHistory, self).supersede()
+ self.setSuperseded()
if dominant is not None:
if logger is not None:
@@ -1081,7 +1077,7 @@
return IMasterStore(BinaryPackagePublishingHistory).find(
BinaryPackagePublishingHistory,
BinaryPackagePublishingHistory.status.is_in(
- [PUBLISHED, PENDING]),
+ active_publishing_status),
BinaryPackagePublishingHistory.distroarchseriesID.is_in(
available_architectures),
binarypackagerelease=self.binarypackagerelease,
@@ -1101,7 +1097,7 @@
return IMasterStore(BinaryPackagePublishingHistory).find(
BinaryPackagePublishingHistory,
BinaryPackagePublishingHistory.status.is_in(
- [PUBLISHED, PENDING]),
+ active_publishing_status),
BinaryPackagePublishingHistory.distroarchseries ==
self.distroarchseries,
binarypackagerelease=self.binarypackagerelease.debug_package,
@@ -1126,7 +1122,7 @@
self.distroarchseries.architecturetag))
return
- super(BinaryPackagePublishingHistory, self).supersede()
+ self.setSuperseded()
if dominant is not None:
# DDEBs cannot themselves be dominant; they are always dominated
=== added file 'lib/lp/soyuz/scripts/gina/dominate.py'
--- lib/lp/soyuz/scripts/gina/dominate.py 1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/scripts/gina/dominate.py 2011-09-08 09:37:30 +0000
@@ -0,0 +1,64 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Retirement of packages that are removed upstream."""
+
+__metaclass__ = type
+__all__ = [
+ 'dominate_imported_source_packages',
+ ]
+
+from zope.component import getUtility
+
+# XXX JeroenVermeulen 2011-09-08, bug=844550: The GeneralizedPublication
+# import violates import policy and elicits a warning from the test
+# suite. The warning helps remind us to retire this code as soon as
+# possible.
+from lp.archivepublisher.domination import (
+ Dominator,
+ GeneralizedPublication,
+ )
+from lp.registry.interfaces.distribution import IDistributionSet
+
+
+def dominate_imported_source_packages(logger, distro_name, series_name,
+ pocket, packages_map):
+ """Perform domination."""
+ series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
+ dominator = Dominator(logger, series.main_archive)
+
+ # Dominate packages found in the Sources list we're importing.
+ package_names = dominator.findPublishedSourcePackageNames(series, pocket)
+ for package_name in package_names:
+ entries = packages_map.src_map.get(package_name)
+
+ if entries is None:
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a
+ # transitional hack. The database is full of "Published"
+ # Debian SPPHs whose packages have actually been deleted.
+ # In the future such publications should simply be marked
+ # Deleted, but for the legacy baggage we currently carry
+ # around we'll just do traditional domination first: pick
+ # the latest Published version, and mark the rest of the
+ # SPPHs as superseded by that version. The latest version
+ # will then, finally, be marked appropriately Deleted once
+ # we remove this transitional hack.
+ # To remove the transitional hack, just let live_versions
+ # default to the empty list instead of doing this:
+ pubs = dominator.findPublishedSPPHs(series, pocket, package_name)
+ generalization = GeneralizedPublication(is_source=True)
+ pubs_dict = dominator._sortPackages(pubs, generalization)
+ sorted_pubs = pubs_dict[package_name]
+ if len(sorted_pubs) <= 1:
+ # If there's only one published SPPH, the transitional
+ # code will just leave it Published. Don't bother; the
+ # migration will be costly enough as it is.
+ continue
+ live_versions = [sorted_pubs[0].sourcepackagerelease.version]
+ else:
+ live_versions = [
+ entry['Version']
+ for entry in entries if 'Version' in entry]
+
+ dominator.dominateRemovedSourceVersions(
+ series, pocket, package_name, live_versions)
=== modified file 'lib/lp/soyuz/scripts/gina/handlers.py'
--- lib/lp/soyuz/scripts/gina/handlers.py 2011-05-20 07:43:58 +0000
+++ lib/lp/soyuz/scripts/gina/handlers.py 2011-09-08 09:37:30 +0000
@@ -35,12 +35,12 @@
from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet
from canonical.launchpad.scripts import log
from lp.archivepublisher.diskpool import poolify
+from lp.archiveuploader.changesfile import ChangesFile
from lp.archiveuploader.tagfiles import parse_tagfile
from lp.archiveuploader.utils import (
determine_binary_file_type,
determine_source_file_type,
)
-from lp.archiveuploader.changesfile import ChangesFile
from lp.buildmaster.enums import BuildStatus
from lp.registry.interfaces.person import (
IPersonSet,
@@ -54,7 +54,10 @@
)
from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet
from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet
-from lp.soyuz.interfaces.publishing import IPublishingSet
+from lp.soyuz.interfaces.publishing import (
+ active_publishing_status,
+ IPublishingSet,
+ )
from lp.soyuz.model.component import Component
from lp.soyuz.model.files import (
BinaryPackageFile,
@@ -723,8 +726,6 @@
source_publishinghistory.status.title)
return
- # Create the Publishing entry, with status PENDING so that we
- # can republish this later into a Soyuz archive.
entry = getUtility(IPublishingSet).newSourcePublication(
distroseries=self.distroseries,
sourcepackagerelease=sourcepackagerelease,
@@ -732,6 +733,7 @@
section=section,
pocket=self.pocket,
archive=archive)
+ entry.setPublished()
log.info('Source package %s (%s) published' % (
entry.sourcepackagerelease.sourcepackagename.name,
entry.sourcepackagerelease.version))
@@ -742,16 +744,14 @@
from lp.soyuz.model.publishing import (
SourcePackagePublishingHistory)
- ret = SourcePackagePublishingHistory.select(
- """sourcepackagerelease = %s
- AND distroseries = %s
- AND archive = %s
- AND status in (%s, %s)""" %
- sqlvalues(sourcepackagerelease, self.distroseries,
- self.distroseries.main_archive,
- PackagePublishingStatus.PUBLISHED,
- PackagePublishingStatus.PENDING),
- orderBy=["-datecreated"])
+ ret = SourcePackagePublishingHistory.select("""
+ sourcepackagerelease = %s AND
+ distroseries = %s AND
+ archive = %s AND
+ status in %s""" % sqlvalues(
+ sourcepackagerelease, self.distroseries,
+ self.distroseries.main_archive, active_publishing_status),
+ orderBy=["-datecreated"])
ret = list(ret)
if ret:
return ret[0]
@@ -917,14 +917,6 @@
"for package %s (%s)" %
(build.id, binary.package, binary.version))
else:
-
- # XXX Debonzi 2005-05-16: Check it later
- # if bin.gpg_signing_key_owner:
- # key = self.getGPGKey(bin.gpg_signing_key,
- # *bin.gpg_signing_key_owner)
- # else:
- key = None
-
processor = distroarchinfo['processor']
build = getUtility(IBinaryPackageBuildSet).new(
processor=processor.id,
@@ -948,8 +940,7 @@
def publish(self, binarypackage, bpdata):
"""Create the publishing entry on db if does not exist."""
# Avoid circular imports.
- from lp.soyuz.model.publishing import (
- BinaryPackagePublishingHistory)
+ from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
# These need to be pulled from the binary package data, not the
# binary package release: the data represents data from /this
@@ -983,22 +974,20 @@
binpkg_publishinghistory.status.title)
return
-
- # Create the Publishing entry with status PENDING.
BinaryPackagePublishingHistory(
- binarypackagerelease = binarypackage.id,
- component = component.id,
- section = section.id,
- priority = priority,
- distroarchseries = self.distroarchseries.id,
- status = PackagePublishingStatus.PENDING,
- datecreated = UTC_NOW,
- datepublished = UTC_NOW,
- pocket = self.pocket,
- datesuperseded = None,
- supersededby = None,
- datemadepending = None,
- dateremoved = None,
+ binarypackagerelease=binarypackage.id,
+ component=component.id,
+ section=section.id,
+ priority=priority,
+ distroarchseries=self.distroarchseries.id,
+ status=PackagePublishingStatus.PUBLISHED,
+ datecreated=UTC_NOW,
+ datepublished=UTC_NOW,
+ pocket=self.pocket,
+ datesuperseded=None,
+ supersededby=None,
+ datemadepending=None,
+ dateremoved=None,
archive=archive)
log.info('BinaryPackage %s-%s published into %s.' % (
@@ -1008,19 +997,16 @@
def _checkPublishing(self, binarypackage):
"""Query for the publishing entry"""
# Avoid circular imports.
- from lp.soyuz.model.publishing import (
- BinaryPackagePublishingHistory)
+ from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
- ret = BinaryPackagePublishingHistory.select(
- """binarypackagerelease = %s
- AND distroarchseries = %s
- AND archive = %s
- AND status in (%s, %s)""" %
- sqlvalues(binarypackage, self.distroarchseries,
- self.distroarchseries.main_archive,
- PackagePublishingStatus.PUBLISHED,
- PackagePublishingStatus.PENDING),
- orderBy=["-datecreated"])
+ ret = BinaryPackagePublishingHistory.select("""
+ binarypackagerelease = %s AND
+ distroarchseries = %s AND
+ archive = %s AND
+ status in %s""" % sqlvalues(
+ binarypackage, self.distroarchseries,
+ self.distroarchseries.main_archive, active_publishing_status),
+ orderBy=["-datecreated"])
ret = list(ret)
if ret:
return ret[0]
=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
--- lib/lp/soyuz/scripts/tests/test_gina.py 2010-08-20 20:31:18 +0000
+++ lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-08 09:37:30 +0000
@@ -1,13 +1,128 @@
-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
# GNU Affero General Public License version 3 (see the file LICENSE).
from doctest import DocTestSuite
-import unittest
+from unittest import TestLoader
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.registry.interfaces.pocket import PackagePublishingPocket
+from lp.services.log.logger import DevNullLogger
+from lp.soyuz.enums import PackagePublishingStatus
+from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
import lp.soyuz.scripts.gina.handlers
+from lp.soyuz.scripts.gina.handlers import (
+ BinaryPackagePublisher,
+ SourcePackagePublisher,
+ )
+from lp.soyuz.scripts.gina.packages import (
+ BinaryPackageData,
+ SourcePackageData,
+ )
+from lp.testing import TestCaseWithFactory
+
+
+class FakePackagesMap:
+ def __init__(self, src_map):
+ self.src_map = src_map
+
+
+class TestGina(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def test_dominate_imported_source_packages_dominates_imports(self):
+ # dominate_imported_source_packages dominates the source
+ # packages that Gina imports.
+ logger = DevNullLogger()
+ pub = self.factory.makeSourcePackagePublishingHistory(
+ status=PackagePublishingStatus.PUBLISHED)
+ series = pub.distroseries
+ spr = pub.sourcepackagerelease
+ package = spr.sourcepackagename
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, pub.pocket,
+ FakePackagesMap({package.name: []}))
+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+ def test_dominate_imported_source_packages_dominates_deletions(self):
+ # dominate_imported_source_packages dominates the source
+ # packages that have been deleted from the Sources lists that
+ # Gina imports.
+ series = self.factory.makeDistroSeries()
+ pocket = PackagePublishingPocket.RELEASE
+ package = self.factory.makeSourcePackageName()
+ pubs = [
+ self.factory.makeSourcePackagePublishingHistory(
+ archive=series.main_archive, distroseries=series,
+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
+ sourcepackagename=package, version=version))
+ for version in ['1.0', '1.1', '1.1a']]
+ logger = DevNullLogger()
+ dominate_imported_source_packages(
+ logger, series.distribution.name, series.name, pocket,
+ FakePackagesMap({}))
+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is
+ # "transitional" domination which supersedes older versions of
+ # deleted packages with the last known version. Permanent
+ # domination will then mark the last known version as deleted.
+ # For permanent domination, the expected outcome is that all
+ # these publications will be Deleted (but any pre-existing
+ # Superseded publications for older versions will remain
+ # Superseded).
+ self.assertEqual([
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.SUPERSEDED,
+ PackagePublishingStatus.PUBLISHED,
+ ],
+ [pub.status for pub in pubs])
+
+
+class TestSourcePackagePublisher(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def test_publish_creates_published_publication(self):
+ maintainer = self.factory.makePerson()
+ series = self.factory.makeDistroSeries()
+ section = self.factory.makeSection()
+ pocket = PackagePublishingPocket.RELEASE
+ spr = self.factory.makeSourcePackageRelease()
+
+ publisher = SourcePackagePublisher(series, pocket, None)
+ publisher.publish(spr, SourcePackageData(
+ component='main', section=section.name, version='1.0',
+ maintainer=maintainer.preferredemail, architecture='all',
+ files='foo.py', binaries='foo.py'))
+
+ [spph] = series.main_archive.getPublishedSources()
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
+
+
+class TestBinaryPackagePublisher(TestCaseWithFactory):
+
+ layer = ZopelessDatabaseLayer
+
+ def test_publish_creates_published_publication(self):
+ maintainer = self.factory.makePerson()
+ series = self.factory.makeDistroArchSeries()
+ section = self.factory.makeSection()
+ pocket = PackagePublishingPocket.RELEASE
+ bpr = self.factory.makeBinaryPackageRelease()
+
+ publisher = BinaryPackagePublisher(series, pocket, None)
+ publisher.publish(bpr, BinaryPackageData(
+ component='main', section=section.name, version='1.0',
+ maintainer=maintainer.preferredemail, architecture='all',
+ files='foo.py', binaries='foo.py', size=128, installed_size=1024,
+ md5sum='e83b5dd68079d727a494a469d40dc8db', description='test',
+ summary='Test!'))
+
+ [bpph] = series.main_archive.getAllPublishedBinaries()
+ self.assertEqual(PackagePublishingStatus.PUBLISHED, bpph.status)
def test_suite():
- suite = unittest.TestSuite()
+ suite = TestLoader().loadTestsFromName(__name__)
suite.addTest(DocTestSuite(lp.soyuz.scripts.gina.handlers))
return suite
=== modified file 'scripts/gina.py'
--- scripts/gina.py 2011-08-23 08:35:13 +0000
+++ scripts/gina.py 2011-09-08 09:37:30 +0000
@@ -38,6 +38,7 @@
MangledArchiveError,
PackagesMap,
)
+from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
from lp.soyuz.scripts.gina.handlers import (
DataSetupError,
ImporterHandler,
@@ -152,6 +153,10 @@
packages_map, kdb, package_root, keyrings, importer_handler)
importer_handler.commit()
+ # XXX JeroenVermeulen 2011-09-07 bug=843728: Dominate binaries as well.
+ dominate_imported_source_packages(
+ log, distro, distroseries, pocket, packages_map)
+
if source_only:
log.info('Source only mode... done')
return
@@ -209,9 +214,8 @@
npacks = len(packages_map.src_map)
log.info('%i Source Packages to be imported', npacks)
- for list_source in sorted(
- packages_map.src_map.values(), key=lambda x: x[0].get("Package")):
- for source in list_source:
+ for package in sorted(packages_map.src_map.iterkeys()):
+ for source in packages_map.src_map[package]:
count += 1
attempt_source_package_import(
source, kdb, package_root, keyrings, importer_handler)
@@ -244,10 +248,9 @@
log.info(
'%i Binary Packages to be imported for %s', npacks, archtag)
# Go over binarypackages importing them for this architecture
- for binary in sorted(packages_map.bin_map[archtag].values(),
- key=lambda x: x.get("Package")):
+ for package_name in sorted(packages_map.bin_map[archtag].iterkeys()):
+ binary = packages_map.bin_map[archtag][package_name]
count += 1
- package_name = binary.get("Package", "unknown")
try:
try:
do_one_binarypackage(binary, archtag, kdb, package_root,
