launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #03411
[Merge] lp:~jtv/launchpad/db-bug-768342 into lp:launchpad/db-devel
Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/db-bug-768342 into lp:launchpad/db-devel.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~jtv/launchpad/db-bug-768342/+merge/58682
= Summary =
The generate-contents script runs apt-ftparchive, which may take a long time. It's not very nice to save up all its log output and dump it just once at the end: we want live output.
== Proposed fix ==
Luckily this is all neatly isolated in a function called "execute." I replaced its subprocess.Popen implementation with one based on CommandSpawner (and its helpful friends for logging and result tracking).
== Pre-implementation notes ==
This was Julian's one gripe from testing the branch for bug 735621, currently landing, which is a requisit for this one. We couldn't think of any meaningful tests for it, since I just re-implemented an existing, tested function.
== Implementation details ==
CommandSpawner is really meant for running multiple commands in parallel, but there's nothing against using it for just one.
== Tests ==
{{{
./bin/test -vvc lp.archivepublisher.tests.test_generate_contents
}}}
== Demo and Q/A ==
Run the generate-contents script on dogfood (or some other sizable archive) and watch its output. It won't be exactly per-line granularity because of buffering, but there should be some output from apt-ftparchive before it completes.
= Launchpad lint =
This lint was previously present and is, as far as I can see, inevitable:
Checking for conflicts and issues in changed files.
Linting changed files:
lib/canonical/config/schema-lazr.conf
lib/lp/archivepublisher/tests/test_generate_contents_files.py
lib/lp/archivepublisher/scripts/generate_contents_files.py
lib/canonical/launchpad/ftests/script.py
cronscripts/publishing/gen-contents/apt_conf_dist.template
cronscripts/publishing/gen-contents/Contents.top
scripts/generate-contents-files.py
cronscripts/publishing/gen-contents/apt_conf_header.template
lib/lp/services/command_spawner.py
./lib/canonical/config/schema-lazr.conf
536: Line exceeds 78 characters.
619: Line exceeds 78 characters.
995: Line exceeds 78 characters.
1084: Line exceeds 78 characters.
./cronscripts/publishing/gen-contents/apt_conf_dist.template
4: Line exceeds 78 characters.
5: Line exceeds 78 characters.
./scripts/generate-contents-files.py
8: '_pythonpath' imported but unused
--
https://code.launchpad.net/~jtv/launchpad/db-bug-768342/+merge/58682
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/db-bug-768342 into lp:launchpad/db-devel.
=== modified file 'cronscripts/publishing/gen-contents/Contents.top'
--- cronscripts/publishing/gen-contents/Contents.top 2006-05-31 22:27:14 +0000
+++ cronscripts/publishing/gen-contents/Contents.top 2011-04-21 14:25:52 +0000
@@ -1,6 +1,6 @@
-This file maps each file available in the Ubuntu Linux system to
-the package from which it originates. It includes packages from the
-DIST distribution for the ARCH architecture.
+This file maps each file available in the %(distrotitle)s
+system to the package from which it originates. It includes packages
+from the DIST distribution for the ARCH architecture.
You can use this list to determine which package contains a specific
file, or whether or not a specific file is available. The list is
=== modified file 'cronscripts/publishing/gen-contents/apt_conf_dist.template'
--- cronscripts/publishing/gen-contents/apt_conf_dist.template 2007-11-27 18:05:38 +0000
+++ cronscripts/publishing/gen-contents/apt_conf_dist.template 2011-04-21 14:25:52 +0000
@@ -1,13 +1,13 @@
-tree "dists/$SUITE"
+tree "dists/%(suite)s"
{
- FileList "/srv/launchpad.net/ubuntu-contents/ubuntu-overrides/$SUITE_$(SECTION)_binary-$(ARCH)";
- SourceFileList "/srv/launchpad.net/ubuntu-contents/ubuntu-overrides/$SUITE_$(SECTION)_source";
+ FileList "%(content_archive)s/%(distribution)s-overrides/%(suite)s_$(SECTION)_binary-$(ARCH)";
+ SourceFileList "%(content_archive)s/%(distribution)s-overrides/%(suite)s_$(SECTION)_source";
Sections "main restricted universe multiverse";
- Architectures "$ARCHS source";
- BinOverride "override.$SUITE.$(SECTION)";
- SrcOverride "override.$SUITE.$(SECTION).src";
- ExtraOverride "override.$SUITE.extra.$(SECTION)";
+ Architectures "%(architectures)s source";
+ BinOverride "override.%(suite)s.$(SECTION)";
+ SrcOverride "override.%(suite)s.$(SECTION).src";
+ ExtraOverride "override.%(suite)s.extra.$(SECTION)";
// we need the plain text content to compare before copy to the real tree
Packages::Compress ". gzip";
Sources::Compress ". gzip";
=== modified file 'cronscripts/publishing/gen-contents/apt_conf_header.template'
--- cronscripts/publishing/gen-contents/apt_conf_header.template 2007-11-27 18:24:35 +0000
+++ cronscripts/publishing/gen-contents/apt_conf_header.template 2011-04-21 14:25:52 +0000
@@ -2,9 +2,9 @@
{
// Content archive stores the results and caches, since they are
// incompatible with the normal ones.
- ArchiveDir "/srv/launchpad.net/ubuntu-contents/ubuntu";
- CacheDir "/srv/launchpad.net/ubuntu-contents/ubuntu-cache";
- OverrideDir "/srv/launchpad.net/ubuntu-contents/ubuntu-overrides";
+ ArchiveDir "%(content_archive)s/%(distribution)s";
+ CacheDir "%(content_archive)s/%(distribution)s-cache";
+ OverrideDir "%(content_archive)s/%(distribution)s-overrides";
};
@@ -16,6 +16,6 @@
TreeDefault
{
// Header for Contents file.
- Contents::Header "/srv/launchpad.net/ubuntu-contents/ubuntu-misc/Contents.top";
+ Contents::Header "%(content_archive)s/%(distribution)s-misc/Contents.top";
};
=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf 2011-04-13 18:48:42 +0000
+++ lib/canonical/config/schema-lazr.conf 2011-04-21 14:25:52 +0000
@@ -24,6 +24,17 @@
# datatype: string
dbuser: archivepublisher
+# Base directory for auxiliary archive where Contents files will be
+# generated.
+#
+# Subdirectories inside this directory will be named after the
+# distribution. For example, if content_archive_root is set to
+# /srv/launchpad.net then Ubuntu's Contents files will be created in
+# /srv/launchpad.net/ubuntu-contents
+#
+# datatype: string
+content_archive_root: /var/tmp/archive
+
# Location where the run-parts directories for publish-ftpmaster
# customization are to be found. Absolute path, or path relative to the
# Launchpad source tree, or "none" to skip execution of run-parts.
=== modified file 'lib/canonical/launchpad/ftests/script.py'
--- lib/canonical/launchpad/ftests/script.py 2009-06-25 05:30:52 +0000
+++ lib/canonical/launchpad/ftests/script.py 2011-04-21 14:25:52 +0000
@@ -4,7 +4,10 @@
"""Helper functions for running external commands."""
__metaclass__ = type
-__all__ = ['run_script']
+__all__ = [
+ 'run_command',
+ 'run_script',
+ ]
import subprocess
import sys
@@ -48,4 +51,3 @@
interpreter_args.extend(args)
return run_command(sys.executable, interpreter_args, input)
-
=== added file 'lib/lp/archivepublisher/scripts/generate_contents_files.py'
--- lib/lp/archivepublisher/scripts/generate_contents_files.py 1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/scripts/generate_contents_files.py 2011-04-21 14:25:52 +0000
@@ -0,0 +1,320 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Archive Contents files generator."""
+
+__metaclass__ = type
+__all__ = [
+ 'GenerateContentsFiles',
+ ]
+
+from optparse import OptionValueError
+import os
+from zope.component import getUtility
+
+from canonical.config import config
+from lp.archivepublisher.config import getPubConfig
+from lp.registry.interfaces.distribution import IDistributionSet
+from lp.services.command_spawner import (
+ CommandSpawner,
+ OutputLineHandler,
+ ReturnCodeReceiver,
+ )
+from lp.services.scripts.base import (
+ LaunchpadScript,
+ LaunchpadScriptFailure,
+ )
+from lp.services.utils import file_exists
+from lp.soyuz.scripts.ftpmaster import LpQueryDistro
+
+
+COMPONENTS = [
+ 'main',
+ 'restricted',
+ 'universe',
+ 'multiverse',
+ ]
+
+
+def differ_in_content(one_file, other_file):
+ """Do the two named files have different contents?"""
+ one_exists = file_exists(one_file)
+ other_exists = file_exists(other_file)
+ if any([one_exists, other_exists]):
+ return (
+ one_exists != other_exists or
+ file(one_file).read() != file(other_file).read())
+ else:
+ return False
+
+
+class StoreArgument:
+ """Local helper for receiving `LpQueryDistro` results."""
+
+ def __call__(self, argument):
+ """Store call argument."""
+ self.argument = argument
+
+
+def get_template(template_name):
+ """Return path of given template in this script's templates directory."""
+ return os.path.join(
+ config.root, "cronscripts", "publishing", "gen-contents",
+ template_name)
+
+
+def execute(logger, command, args=None):
+ """Execute a shell command.
+
+ :param logger: Output from the command will be logged here.
+ :param command: Command to execute, as a string.
+ :param args: Optional list of arguments for `command`.
+ :raises LaunchpadScriptFailure: If the command returns failure.
+ """
+ command_line = [command]
+ if args is not None:
+ command_line += args
+ description = ' '.join(command_line)
+
+ logger.debug("Execute: %s", description)
+ # Some of these commands can take a long time. Use CommandSpawner
+ # and friends to provide "live" log output. Simpler ways of running
+ # commands tend to save it all up and then dump it at the end, or
+ # have trouble logging it as neat lines.
+ stderr_logger = OutputLineHandler(logger.warn)
+ stdout_logger = OutputLineHandler(logger.debug)
+ receiver = ReturnCodeReceiver()
+ spawner = CommandSpawner()
+ spawner.start(
+ command_line, completion_handler=receiver,
+ stderr_handler=stderr_logger, stdout_handler=stdout_logger)
+ spawner.complete()
+ stdout_logger.finalize()
+ stderr_logger.finalize()
+ if receiver.returncode != 0:
+ raise LaunchpadScriptFailure(
+ "Failure while running command: %s" % description)
+
+
+def move_file(old_path, new_path):
+ """Rename file `old_path` to `new_path`.
+
+ Mercilessly delete any file that may already exist at `new_path`.
+ """
+ if file_exists(new_path):
+ os.remove(new_path)
+ os.rename(old_path, new_path)
+
+
+class GenerateContentsFiles(LaunchpadScript):
+
+ distribution = None
+
+ def add_my_options(self):
+ """See `LaunchpadScript`."""
+ self.parser.add_option(
+ "-d", "--distribution", dest="distribution", default=None,
+ help="Distribution to generate Contents files for.")
+
+ @property
+ def name(self):
+ """See `LaunchpadScript`."""
+ # Include distribution name. Clearer to admins, but also
+ # puts runs for different distributions under separate
+ # locks so that they can run simultaneously.
+ return "%s-%s" % (self._name, self.options.distribution)
+
+ def processOptions(self):
+ """Handle command-line options."""
+ if self.options.distribution is None:
+ raise OptionValueError("Specify a distribution.")
+
+ self.distribution = getUtility(IDistributionSet).getByName(
+ self.options.distribution)
+ if self.distribution is None:
+ raise OptionValueError(
+ "Distribution '%s' not found." % self.options.distribution)
+
+ def setUpContentArchive(self):
+ """Make sure the `content_archive` directories exist."""
+ self.logger.debug("Ensuring that we have a private tree in place.")
+ for suffix in ['cache', 'misc']:
+ dirname = '-'.join([self.distribution.name, suffix])
+ path = os.path.join(self.content_archive, dirname)
+ if not file_exists(path):
+ os.makedirs(path)
+
+ def queryDistro(self, request, options=None):
+ """Call the query-distro script about `self.distribution`."""
+ args = ['-d', self.distribution.name]
+ if options is not None:
+ args += options
+ args.append(request)
+ query_distro = LpQueryDistro(test_args=args)
+ query_distro.logger = self.logger
+ query_distro.txn = self.txn
+ receiver = StoreArgument()
+ query_distro.runAction(presenter=receiver)
+ return receiver.argument
+
+ def getPocketSuffixes(self):
+ """Query the distribution's pocket suffixes."""
+ return self.queryDistro("pocket_suffixes").split()
+
+ def getSuites(self):
+ """Query the distribution's suites."""
+ return self.queryDistro("supported").split()
+
+ def getPockets(self):
+ """Return suites that are actually supported in this distribution."""
+ pockets = []
+ pocket_suffixes = self.getPocketSuffixes()
+ for suite in self.getSuites():
+ for pocket_suffix in pocket_suffixes:
+ pocket = suite + pocket_suffix
+ if file_exists(os.path.join(self.config.distsroot, pocket)):
+ pockets.append(pocket)
+ return pockets
+
+ def getArchs(self):
+ """Query architectures supported by the distribution."""
+ devel = self.queryDistro("development")
+ return self.queryDistro("archs", options=["-s", devel]).split()
+
+ def getDirs(self, archs):
+ """Subdirectories needed for each component."""
+ return ['source', 'debian-installer'] + [
+ 'binary-%s' % arch for arch in archs]
+
+ def writeAptContentsConf(self, suites, archs):
+ """Write apt-contents.conf file."""
+ output_dirname = '%s-misc' % self.distribution.name
+ output_path = os.path.join(
+ self.content_archive, output_dirname, "apt-contents.conf")
+ output_file = file(output_path, 'w')
+
+ parameters = {
+ 'architectures': ' '.join(archs),
+ 'content_archive': self.content_archive,
+ 'distribution': self.distribution.name,
+ }
+
+ header = get_template('apt_conf_header.template')
+ output_file.write(file(header).read() % parameters)
+
+ dist_template = file(get_template('apt_conf_dist.template')).read()
+ for suite in suites:
+ parameters['suite'] = suite
+ output_file.write(dist_template % parameters)
+
+ output_file.close()
+
+ def createComponentDirs(self, suites, archs):
+ """Create the content archive's tree for all of its components."""
+ for suite in suites:
+ for component in COMPONENTS:
+ for directory in self.getDirs(archs):
+ path = os.path.join(
+ self.content_archive, self.distribution.name, 'dists',
+ suite, component, directory)
+ if not file_exists(path):
+ self.logger.debug("Creating %s.", path)
+ os.makedirs(path)
+
+ def copyOverrides(self):
+ """Copy overrides into the content archive."""
+ if file_exists(self.config.overrideroot):
+ execute(self.logger, "cp", [
+ "-a",
+ self.config.overrideroot,
+ "%s/" % self.content_archive,
+ ])
+ else:
+ self.logger.debug("Did not find overrides; not copying.")
+
+ def writeContentsTop(self):
+ """Write Contents.top file."""
+ output_filename = os.path.join(
+ self.content_archive, '%s-misc' % self.distribution.name,
+ "Contents.top")
+ parameters = {
+ 'distrotitle': self.distribution.title,
+ }
+ output_file = file(output_filename, 'w')
+ text = file(get_template("Contents.top")).read() % parameters
+ output_file.write(text)
+ output_file.close()
+
+ def runAptFTPArchive(self):
+ """Run apt-ftparchive to produce the Contents files."""
+ execute(self.logger, "apt-ftparchive", [
+ "generate",
+ os.path.join(
+ self.content_archive, "%s-misc" % self.distribution.name,
+ "apt-contents.conf"),
+ ])
+
+ def generateContentsFiles(self):
+ """Generate Contents files."""
+ self.logger.debug(
+ "Running apt in private tree to generate new contents.")
+ self.copyOverrides()
+ self.writeContentsTop()
+ self.runAptFTPArchive()
+
+ def updateContentsFile(self, suite, arch):
+ """Update Contents file, if it has changed."""
+ contents_dir = os.path.join(
+ self.content_archive, self.distribution.name, 'dists', suite)
+ contents_filename = "Contents-%s" % arch
+ last_contents = os.path.join(contents_dir, ".%s" % contents_filename)
+ current_contents = os.path.join(contents_dir, contents_filename)
+
+ # Avoid rewriting unchanged files; mirrors would have to
+ # re-fetch them unnecessarily.
+ if differ_in_content(current_contents, last_contents):
+ self.logger.debug(
+ "Installing new Contents file for %s/%s.", suite, arch)
+
+ new_contents = os.path.join(
+ contents_dir, "%s.gz" % contents_filename)
+ contents_dest = os.path.join(
+ self.config.distsroot, suite, "%s.gz" % contents_filename)
+
+ move_file(current_contents, last_contents)
+ move_file(new_contents, contents_dest)
+ os.chmod(contents_dest, 0664)
+ else:
+ self.logger.debug(
+ "Skipping unmodified Contents file for %s/%s.", suite, arch)
+
+ def updateContentsFiles(self, suites, archs):
+ """Update all Contents files that have changed."""
+ self.logger.debug("Comparing contents files with public tree.")
+ for suite in suites:
+ for arch in archs:
+ self.updateContentsFile(suite, arch)
+
+ def setUp(self):
+ """Prepare configuration and filesystem state for the script's work.
+
+ This is idempotent: run it as often as you like. (For example,
+ a test may call `setUp` prior to calling `main` which again
+ invokes `setUp`).
+ """
+ self.processOptions()
+ self.config = getPubConfig(self.distribution.main_archive)
+ self.content_archive = os.path.join(
+ config.archivepublisher.content_archive_root,
+ self.distribution.name + "-contents")
+ self.setUpContentArchive()
+
+ def main(self):
+ """See `LaunchpadScript`."""
+ self.setUp()
+ suites = self.getPockets()
+ archs = self.getArchs()
+ self.writeAptContentsConf(suites, archs)
+ self.createComponentDirs(suites, archs)
+ self.generateContentsFiles()
+ self.updateContentsFiles(suites, archs)
=== added file 'lib/lp/archivepublisher/tests/test_generate_contents_files.py'
--- lib/lp/archivepublisher/tests/test_generate_contents_files.py 1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/tests/test_generate_contents_files.py 2011-04-21 14:25:52 +0000
@@ -0,0 +1,291 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Test for the `generate-contents-files` script."""
+
+__metaclass__ = type
+
+from optparse import OptionValueError
+import os.path
+from textwrap import dedent
+
+from canonical.config import config
+from canonical.testing.layers import (
+ LaunchpadZopelessLayer,
+ ZopelessDatabaseLayer,
+ )
+from lp.archivepublisher.scripts.generate_contents_files import (
+ differ_in_content,
+ execute,
+ GenerateContentsFiles,
+ move_file,
+ )
+from lp.services.log.logger import DevNullLogger
+from lp.services.scripts.base import LaunchpadScriptFailure
+from lp.services.utils import file_exists
+from lp.testing import TestCaseWithFactory
+
+
+def write_file(filename, content=""):
+ """Write `content` to `filename`, and flush."""
+ output_file = file(filename, 'w')
+ output_file.write(content)
+ output_file.close()
+
+
+def fake_overrides(script, distroseries):
+ """Fake overrides files so `script` can run `apt-ftparchive`."""
+ os.makedirs(script.config.overrideroot)
+
+ components = ['main', 'restricted', 'universe', 'multiverse']
+ architectures = script.getArchs()
+ suffixes = components + ['extra.' + component for component in components]
+ for suffix in suffixes:
+ write_file(os.path.join(
+ script.config.overrideroot,
+ "override.%s.%s" % (distroseries.name, suffix)))
+
+ for component in components:
+ write_file(os.path.join(
+ script.config.overrideroot,
+ "%s_%s_source" % (distroseries.name, component)))
+ for arch in architectures:
+ write_file(os.path.join(
+ script.config.overrideroot,
+ "%s_%s_binary-%s" % (distroseries.name, component, arch)))
+
+
+class TestHelpers(TestCaseWithFactory):
+ """Tests for the module's helper functions."""
+
+ layer = ZopelessDatabaseLayer
+
+ def test_differ_in_content_returns_true_if_one_file_does_not_exist(self):
+ # A nonexistent file differs from an existing one.
+ self.useTempDir()
+ write_file('one', self.factory.getUniqueString())
+ self.assertTrue(differ_in_content('one', 'other'))
+
+ def test_differ_in_content_returns_false_for_identical_files(self):
+ # Identical files do not differ.
+ self.useTempDir()
+ text = self.factory.getUniqueString()
+ write_file('one', text)
+ write_file('other', text)
+ self.assertFalse(differ_in_content('one', 'other'))
+
+ def test_differ_in_content_returns_true_for_differing_files(self):
+ # Files with different contents differ.
+ self.useTempDir()
+ write_file('one', self.factory.getUniqueString())
+ write_file('other', self.factory.getUniqueString())
+ self.assertTrue(differ_in_content('one', 'other'))
+
+ def test_differ_in_content_returns_false_if_neither_file_exists(self):
+ # Nonexistent files do not differ.
+ self.useTempDir()
+ self.assertFalse(differ_in_content('one', 'other'))
+
+ def test_execute_raises_if_command_fails(self):
+ # execute checks its command's return value. If it's nonzero
+ # (as with /bin/false), it raises a LaunchpadScriptFailure.
+ logger = DevNullLogger()
+ self.assertRaises(
+ LaunchpadScriptFailure, execute, logger, "/bin/false")
+
+ def test_execute_executes_command(self):
+ # execute really does execute its command. If we tell it to
+ # "touch" a new file, that file really gets created.
+ self.useTempDir()
+ logger = DevNullLogger()
+ filename = self.factory.getUniqueString()
+ execute(logger, "touch", [filename])
+ self.assertTrue(file_exists(filename))
+
+ def test_move_file_renames_file(self):
+ # move_file renames a file from its old name to its new name.
+ self.useTempDir()
+ text = self.factory.getUniqueString()
+ write_file("old_name", text)
+ move_file("old_name", "new_name")
+ self.assertEqual(text, file("new_name").read())
+
+ def test_move_file_overwrites_old_file(self):
+ # If move_file finds another file in the way, that file gets
+ # deleted.
+ self.useTempDir()
+ write_file("new_name", self.factory.getUniqueString())
+ new_text = self.factory.getUniqueString()
+ write_file("old_name", new_text)
+ move_file("old_name", "new_name")
+ self.assertEqual(new_text, file("new_name").read())
+
+
+class TestGenerateContentsFiles(TestCaseWithFactory):
+ """Tests for the actual `GenerateContentsFiles` script."""
+
+ layer = LaunchpadZopelessLayer
+
+ def makeContentArchive(self):
+ """Prepare a "content archive" directory for script tests."""
+ content_archive = self.makeTemporaryDirectory()
+ config.push("content-archive", dedent("""\
+ [archivepublisher]
+ content_archive_root: %s
+ """ % content_archive))
+ self.addCleanup(config.pop, "content-archive")
+ return content_archive
+
+ def makeDistro(self):
+ """Create a distribution for testing.
+
+ The distribution will have a root directory set up, which will
+ be cleaned up after the test.
+ """
+ return self.factory.makeDistribution(
+ publish_root_dir=unicode(self.makeTemporaryDirectory()))
+
+ def makeScript(self, distribution=None):
+ """Create a script for testing."""
+ if distribution is None:
+ distribution = self.makeDistro()
+ script = GenerateContentsFiles(test_args=['-d', distribution.name])
+ script.logger = DevNullLogger()
+ script.setUp()
+ return script
+
+ def test_name_is_consistent(self):
+ # Script instances for the same distro get the same name.
+ distro = self.factory.makeDistribution()
+ self.assertEqual(
+ GenerateContentsFiles(test_args=['-d', distro.name]).name,
+ GenerateContentsFiles(test_args=['-d', distro.name]).name)
+
+ def test_name_is_unique_for_each_distro(self):
+ # Script instances for different distros get different names.
+ self.assertNotEqual(
+ GenerateContentsFiles(
+ test_args=['-d', self.factory.makeDistribution().name]).name,
+ GenerateContentsFiles(
+ test_args=['-d', self.factory.makeDistribution().name]).name)
+
+ def test_requires_distro(self):
+ # The --distribution or -d argument is mandatory.
+ script = GenerateContentsFiles(test_args=[])
+ self.assertRaises(OptionValueError, script.processOptions)
+
+ def test_requires_real_distro(self):
+ # An incorrect distribution name is flagged as an invalid option
+ # value.
+ script = GenerateContentsFiles(
+ test_args=['-d', self.factory.getUniqueString()])
+ self.assertRaises(OptionValueError, script.processOptions)
+
+ def test_looks_up_distro(self):
+ # The script looks up and keeps the distribution named on the
+ # command line.
+ distro = self.factory.makeDistribution()
+ script = self.makeScript(distro)
+ self.assertEqual(distro, script.distribution)
+
+ def test_queryDistro(self):
+ # queryDistro is a helper that invokes LpQueryDistro.
+ distroseries = self.factory.makeDistroSeries()
+ script = self.makeScript(distroseries.distribution)
+ script.processOptions()
+ self.assertEqual(distroseries.name, script.queryDistro('supported'))
+
+ def test_getArchs(self):
+ # getArchs returns a list of architectures in the distribution.
+ das = self.factory.makeDistroArchSeries()
+ script = self.makeScript(das.distroseries.distribution)
+ self.assertEqual([das.architecturetag], script.getArchs())
+
+ def test_getSuites(self):
+ # getSuites returns the suites in the distribution. The main
+ # suite has the same name as the distro, without suffix.
+ script = self.makeScript()
+ distroseries = self.factory.makeDistroSeries(
+ distribution=script.distribution)
+ self.assertIn(distroseries.name, script.getSuites())
+
+ def test_getPockets(self):
+ # getPockets returns the full names (distroseries-pocket) of the
+ # pockets that have packages to publish.
+ distro = self.makeDistro()
+ distroseries = self.factory.makeDistroSeries(distribution=distro)
+ package = self.factory.makeSuiteSourcePackage(distroseries)
+ script = self.makeScript(distro)
+ os.makedirs(os.path.join(script.config.distsroot, package.suite))
+ self.assertEqual([package.suite], script.getPockets())
+
+ def test_writeAptContentsConf_writes_header(self):
+ # writeAptContentsConf writes apt-contents.conf. At a minimum
+ # this will include a header based on apt_conf_header.template,
+ # with the right distribution name interpolated.
+ self.makeContentArchive()
+ distro = self.makeDistro()
+ script = self.makeScript(distro)
+ script.writeAptContentsConf([], [])
+ apt_contents_conf = file(
+ "%s/%s-misc/apt-contents.conf"
+ % (script.content_archive, distro.name)).read()
+ self.assertIn('\nDefault\n{', apt_contents_conf)
+ self.assertIn(distro.name, apt_contents_conf)
+
+ def test_writeAptContentsConf_writes_suite_sections(self):
+ # writeAptContentsConf adds sections based on
+ # apt_conf_dist.template for every suite, with certain
+ # parameters interpolated.
+ content_archive = self.makeContentArchive()
+ distro = self.makeDistro()
+ script = self.makeScript(distro)
+ suite = self.factory.getUniqueString('suite')
+ arch = self.factory.getUniqueString('arch')
+ script.writeAptContentsConf([suite], [arch])
+ apt_contents_conf = file(
+ "%s/%s-misc/apt-contents.conf"
+ % (script.content_archive, distro.name)).read()
+ self.assertIn('tree "dists/%s"\n' % suite, apt_contents_conf)
+ overrides_path = os.path.join(
+ content_archive, distro.name + "-contents",
+ distro.name + "-overrides")
+ self.assertIn('FileList "%s' % overrides_path, apt_contents_conf)
+ self.assertIn('Architectures "%s source";' % arch, apt_contents_conf)
+
+ def test_writeContentsTop(self):
+ # writeContentsTop writes a Contents.top file based on a
+ # standard template, with the distribution's title interpolated.
+ content_archive = self.makeContentArchive()
+ distro = self.makeDistro()
+ script = self.makeScript(distro)
+ script.writeContentsTop()
+ contents_top = file(
+ "%s/%s-contents/%s-misc/Contents.top"
+ % (content_archive, distro.name, distro.name)).read()
+ self.assertIn("This file maps", contents_top)
+ self.assertIn(distro.title, contents_top)
+
+ def test_main(self):
+ # If run end-to-end, the script generates Contents.gz files.
+ self.makeContentArchive()
+ distro = self.makeDistro()
+ distroseries = self.factory.makeDistroSeries(distribution=distro)
+ processor = self.factory.makeProcessor()
+ das = self.factory.makeDistroArchSeries(
+ distroseries=distroseries, processorfamily=processor.family)
+ package = self.factory.makeSuiteSourcePackage(distroseries)
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=package.pocket)
+ self.factory.makeBinaryPackageBuild(
+ distroarchseries=das, pocket=package.pocket,
+ processor=processor)
+ suite = package.suite
+ script = self.makeScript(distro)
+ os.makedirs(os.path.join(script.config.distsroot, package.suite))
+ self.assertNotEqual([], script.getPockets())
+ fake_overrides(script, distroseries)
+ script.main()
+ self.assertTrue(file_exists(os.path.join(
+ script.config.distsroot, suite,
+ "Contents-%s.gz" % das.architecturetag)))
=== modified file 'lib/lp/services/command_spawner.py'
--- lib/lp/services/command_spawner.py 2011-04-06 03:43:28 +0000
+++ lib/lp/services/command_spawner.py 2011-04-21 14:25:52 +0000
@@ -99,9 +99,9 @@
:param stderr_handler: Callback to handle error output received from
the sub-process. Must take the output as its sole argument. May
be called any number of times as output comes in.
- :param failure_handler: Callback to be invoked, exactly once, when the
- sub-process exits. Must take `command`'s numeric return code as
- its sole argument.
+ :param completion_handler: Callback to be invoked, exactly once, when
+ the sub-process exits. Must take `command`'s numeric return code
+ as its sole argument.
"""
process = self._spawn(command)
handlers = {
@@ -172,9 +172,9 @@
try:
output = pipe_file.read()
except IOError, e:
+ # "Resource temporarily unavailable"--not an error really,
+ # just means there's nothing to read.
if e.errno != errno.EAGAIN:
- # "Resource temporarily unavailable"--not an error
- # really, just means there's nothing to read.
raise
else:
if len(output) > 0:
=== added file 'scripts/generate-contents-files.py'
--- scripts/generate-contents-files.py 1970-01-01 00:00:00 +0000
+++ scripts/generate-contents-files.py 2011-04-21 14:25:52 +0000
@@ -0,0 +1,19 @@
+#!/usr/bin/python -S
+#
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Master distro publishing script."""
+
+import _pythonpath
+
+from canonical.config import config
+from lp.archivepublisher.scripts.generate_contents_files import (
+ GenerateContentsFiles,
+ )
+
+
+if __name__ == '__main__':
+ script = GenerateContentsFiles(
+ "generate-contents", dbuser=config.archivepublisher.dbuser)
+ script.lock_and_run()
