launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #03376
[Merge] lp:~jtv/launchpad/db-bug-735621 into lp:launchpad/db-devel
Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/db-bug-735621 into lp:launchpad/db-devel.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #735621 in Launchpad itself: "Re-write cronscripts/publishing/gen-contents/generate-contents in Python"
https://bugs.launchpad.net/launchpad/+bug/735621
For more details, see:
https://code.launchpad.net/~jtv/launchpad/db-bug-735621/+merge/58321
= Summary =
See bug 735621. This converts an untested soyuz bash script to python.
The script in question is cronscripts/publishing/gen-contents/generate-contents.
In addition to converting the script to python and adding tests, this also breaks the hard-coded tie to Ubuntu and permits the same script to run in parallel for multiple distributions (though not for the same distribution, since it would probably cause trouble).
== Implementation details ==
In the diff you'll see a few templates that are used for writing various configuration scripts that tell the Debian machinery what to do. These were plain text files with a few replacements made by custom code. I replaced that with python string interpolation. The parameters are passed as a dict so they can be accessed by name. There's no other documentation of how this works or what parameters are passed. With "bzr log" showing me one change in 2007 and another in 2006 for these files, it's probably more cost-effective for the next engineer to touch this to figure things out to the extent needed than it is to write and maintain documentation.
I couldn't place the script in its old location; it won't import _pythonpath if I put it there. But I gave it a more descriptive name.
== Test ==
I couldn't find any tests for the old script. There's only the new test:
{{{
./bin/test -vvc lp.archivepublisher.tests.test_generate_contents_files
}}}
This doesn't cover executing the script itself. The test is massively expensive as it is; I don't think another 5+ seconds of ZCML parsing will pull their weight. There's nothing in there but the usual "import and run LaunchpadScript" boilerplate. I tried it by hand.
A more interesting hole in test coverage is database privileges. The script barely accesses the database at all, but it calls LpQueryDistro which does. The archive publisher's database user (as set in the lazr config) should have the required privileges. Is this worth an expensive extra test? I'm not sure.
== Demo and Q/A ==
One of the Soyuz gurus will have to verify this. Apparently the script is so costly that it doesn't even get run on dogfood. I don't expect to see any changes in performance characteristics.
= Launchpad lint =
The lint is all bogus: files we shouldn't check and one warning we can't avoid.
Checking for conflicts and issues in changed files.
Linting changed files:
lib/canonical/config/schema-lazr.conf
lib/lp/archivepublisher/tests/test_generate_contents_files.py
lib/lp/archivepublisher/scripts/generate_contents_files.py
lib/canonical/launchpad/ftests/script.py
cronscripts/publishing/gen-contents/apt_conf_dist.template
cronscripts/publishing/gen-contents/Contents.top
cronscripts/generate-contents-files.py
cronscripts/publishing/gen-contents/apt_conf_header.template
./lib/canonical/config/schema-lazr.conf
536: Line exceeds 78 characters.
619: Line exceeds 78 characters.
995: Line exceeds 78 characters.
1084: Line exceeds 78 characters.
./cronscripts/publishing/gen-contents/apt_conf_dist.template
4: Line exceeds 78 characters.
5: Line exceeds 78 characters.
./cronscripts/generate-contents-files.py
8: '_pythonpath' imported but unused
--
https://code.launchpad.net/~jtv/launchpad/db-bug-735621/+merge/58321
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/db-bug-735621 into lp:launchpad/db-devel.
=== modified file 'cronscripts/publishing/gen-contents/Contents.top'
--- cronscripts/publishing/gen-contents/Contents.top 2006-05-31 22:27:14 +0000
+++ cronscripts/publishing/gen-contents/Contents.top 2011-04-19 15:01:04 +0000
@@ -1,6 +1,6 @@
-This file maps each file available in the Ubuntu Linux system to
-the package from which it originates. It includes packages from the
-DIST distribution for the ARCH architecture.
+This file maps each file available in the %(distrotitle)s
+system to the package from which it originates. It includes packages
+from the DIST distribution for the ARCH architecture.
You can use this list to determine which package contains a specific
file, or whether or not a specific file is available. The list is
=== modified file 'cronscripts/publishing/gen-contents/apt_conf_dist.template'
--- cronscripts/publishing/gen-contents/apt_conf_dist.template 2007-11-27 18:05:38 +0000
+++ cronscripts/publishing/gen-contents/apt_conf_dist.template 2011-04-19 15:01:04 +0000
@@ -1,13 +1,13 @@
-tree "dists/$SUITE"
+tree "dists/%(suite)s"
{
- FileList "/srv/launchpad.net/ubuntu-contents/ubuntu-overrides/$SUITE_$(SECTION)_binary-$(ARCH)";
- SourceFileList "/srv/launchpad.net/ubuntu-contents/ubuntu-overrides/$SUITE_$(SECTION)_source";
+ FileList "%(content_archive)s/%(distribution)s-overrides/%(suite)s_$(SECTION)_binary-$(ARCH)";
+ SourceFileList "%(content_archive)s/%(distribution)s-overrides/%(suite)s_$(SECTION)_source";
Sections "main restricted universe multiverse";
- Architectures "$ARCHS source";
- BinOverride "override.$SUITE.$(SECTION)";
- SrcOverride "override.$SUITE.$(SECTION).src";
- ExtraOverride "override.$SUITE.extra.$(SECTION)";
+ Architectures "%(architectures)s source";
+ BinOverride "override.%(suite)s.$(SECTION)";
+ SrcOverride "override.%(suite)s.$(SECTION).src";
+ ExtraOverride "override.%(suite)s.extra.$(SECTION)";
// we need the plain text content to compare before copy to the real tree
Packages::Compress ". gzip";
Sources::Compress ". gzip";
=== modified file 'cronscripts/publishing/gen-contents/apt_conf_header.template'
--- cronscripts/publishing/gen-contents/apt_conf_header.template 2007-11-27 18:24:35 +0000
+++ cronscripts/publishing/gen-contents/apt_conf_header.template 2011-04-19 15:01:04 +0000
@@ -2,9 +2,9 @@
{
// Content archive stores the results and caches, since they are
// incompatible with the normal ones.
- ArchiveDir "/srv/launchpad.net/ubuntu-contents/ubuntu";
- CacheDir "/srv/launchpad.net/ubuntu-contents/ubuntu-cache";
- OverrideDir "/srv/launchpad.net/ubuntu-contents/ubuntu-overrides";
+ ArchiveDir "%(content_archive)s/%(distribution)s";
+ CacheDir "%(content_archive)s/%(distribution)s-cache";
+ OverrideDir "%(content_archive)s/%(distribution)s-overrides";
};
@@ -16,6 +16,6 @@
TreeDefault
{
// Header for Contents file.
- Contents::Header "/srv/launchpad.net/ubuntu-contents/ubuntu-misc/Contents.top";
+ Contents::Header "%(content_archive)s/%(distribution)s-misc/Contents.top";
};
=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf 2011-04-13 18:48:42 +0000
+++ lib/canonical/config/schema-lazr.conf 2011-04-19 15:01:04 +0000
@@ -24,6 +24,17 @@
# datatype: string
dbuser: archivepublisher
+# Base directory for auxiliary archive where Contents files will be
+# generated.
+#
+# Subdirectories inside this directory will be named after the
+# distribution. For example, if content_archive_root is set to
+# /srv/launchpad.net then Ubuntu's Contents files will be created in
+# /srv/launchpad.net/ubuntu-contents
+#
+# datatype: string
+content_archive_root: /var/tmp/archive
+
# Location where the run-parts directories for publish-ftpmaster
# customization are to be found. Absolute path, or path relative to the
# Launchpad source tree, or "none" to skip execution of run-parts.
=== modified file 'lib/canonical/launchpad/ftests/script.py'
--- lib/canonical/launchpad/ftests/script.py 2009-06-25 05:30:52 +0000
+++ lib/canonical/launchpad/ftests/script.py 2011-04-19 15:01:04 +0000
@@ -4,7 +4,10 @@
"""Helper functions for running external commands."""
__metaclass__ = type
-__all__ = ['run_script']
+__all__ = [
+ 'run_command',
+ 'run_script',
+ ]
import subprocess
import sys
@@ -48,4 +51,3 @@
interpreter_args.extend(args)
return run_command(sys.executable, interpreter_args, input)
-
=== added file 'lib/lp/archivepublisher/scripts/generate_contents_files.py'
--- lib/lp/archivepublisher/scripts/generate_contents_files.py 1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/scripts/generate_contents_files.py 2011-04-19 15:01:04 +0000
@@ -0,0 +1,303 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Archive Contents files generator."""
+
+__metaclass__ = type
+__all__ = [
+ 'GenerateContentsFiles',
+ ]
+
+from optparse import OptionValueError
+import os
+from zope.component import getUtility
+
+from canonical.config import config
+from canonical.launchpad.ftests.script import run_command
+from lp.archivepublisher.config import getPubConfig
+from lp.registry.interfaces.distribution import IDistributionSet
+from lp.services.scripts.base import (
+ LaunchpadScript,
+ LaunchpadScriptFailure,
+ )
+from lp.services.utils import file_exists
+from lp.soyuz.scripts.ftpmaster import LpQueryDistro
+
+
+COMPONENTS = [
+ 'main',
+ 'restricted',
+ 'universe',
+ 'multiverse',
+ ]
+
+
+def differ_in_content(one_file, other_file):
+ """Do the two named files have different contents?"""
+ one_exists = file_exists(one_file)
+ other_exists = file_exists(other_file)
+ if any([one_exists, other_exists]):
+ return (
+ one_exists != other_exists or
+ file(one_file).read() != file(other_file).read())
+ else:
+ return False
+
+
+class StoreArgument:
+ """Local helper for receiving `LpQueryDistro` results."""
+
+ def __call__(self, argument):
+ """Store call argument."""
+ self.argument = argument
+
+
+def get_template(template_name):
+ """Return path of given template in this script's templates directory."""
+ return os.path.join(
+ config.root, "cronscripts", "publishing", "gen-contents",
+ template_name)
+
+
+def execute(logger, command, args=None):
+ """Execute a shell command.
+
+ :param logger: Output from the command will be logged here.
+ :param command_line: Command to execute, as a list of tokens.
+ :raises LaunchpadScriptFailure: If the command returns failure.
+ """
+ if args is None:
+ description = command
+ else:
+ description = command + ' ' + ' '.join(args)
+ logger.debug("Execute: %s", description)
+ retval, stdout, stderr = run_command(command, args)
+ logger.debug(stdout)
+ logger.warn(stderr)
+ if retval != 0:
+ raise LaunchpadScriptFailure(
+ "Failure while running command: %s" % description)
+
+
+def move_file(old_path, new_path):
+ """Rename file `old_path` to `new_path`.
+
+ Mercilessly delete any file that may already exist at `new_path`.
+ """
+ if file_exists(new_path):
+ os.remove(new_path)
+ os.rename(old_path, new_path)
+
+
+class GenerateContentsFiles(LaunchpadScript):
+
+ distribution = None
+
+ def add_my_options(self):
+ """See `LaunchpadScript`."""
+ self.parser.add_option(
+ "-d", "--distribution", dest="distribution", default=None,
+ help="Distribution to generate Contents files for.")
+
+ @property
+ def name(self):
+ """See `LaunchpadScript`."""
+ # Include distribution name. Clearer to admins, but also
+ # puts runs for different distributions under separate
+ # locks so that they can run simultaneously.
+ return "%s-%s" % (self._name, self.options.distribution)
+
+ def processOptions(self):
+ """Handle command-line options."""
+ if self.options.distribution is None:
+ raise OptionValueError("Specify a distribution.")
+
+ self.distribution = getUtility(IDistributionSet).getByName(
+ self.options.distribution)
+ if self.distribution is None:
+ raise OptionValueError(
+ "Distribution '%s' not found." % self.options.distribution)
+
+ def setUpContentArchive(self):
+ """Make sure the `content_archive` directories exist."""
+ self.logger.debug("Ensuring that we have a private tree in place.")
+ for suffix in ['cache', 'misc']:
+ dirname = '-'.join([self.distribution.name, suffix])
+ path = os.path.join(self.content_archive, dirname)
+ if not file_exists(path):
+ os.makedirs(path)
+
+ def queryDistro(self, request, options=None):
+ """Call the query-distro script about `self.distribution`."""
+ args = ['-d', self.distribution.name]
+ if options is not None:
+ args += options
+ args.append(request)
+ query_distro = LpQueryDistro(test_args=args)
+ query_distro.logger = self.logger
+ query_distro.txn = self.txn
+ receiver = StoreArgument()
+ query_distro.runAction(presenter=receiver)
+ return receiver.argument
+
+ def getPocketSuffixes(self):
+ """Query the distribution's pocket suffixes."""
+ return self.queryDistro("pocket_suffixes").split()
+
+ def getSuites(self):
+ """Query the distribution's suites."""
+ return self.queryDistro("supported").split()
+
+ def getPockets(self):
+ """Return suites that are actually supported in this distribution."""
+ pockets = []
+ pocket_suffixes = self.getPocketSuffixes()
+ for suite in self.getSuites():
+ for pocket_suffix in pocket_suffixes:
+ pocket = suite + pocket_suffix
+ if file_exists(os.path.join(self.config.distsroot, pocket)):
+ pockets.append(pocket)
+ return pockets
+
+ def getArchs(self):
+ """Query architectures supported by the distribution."""
+ devel = self.queryDistro("development")
+ return self.queryDistro("archs", options=["-s", devel]).split()
+
+ def getDirs(self, archs):
+ """Subdirectories needed for each component."""
+ return ['source', 'debian-installer'] + [
+ 'binary-%s' % arch for arch in archs]
+
+ def writeAptContentsConf(self, suites, archs):
+ """Write apt-contents.conf file."""
+ output_dirname = '%s-misc' % self.distribution.name
+ output_path = os.path.join(
+ self.content_archive, output_dirname, "apt-contents.conf")
+ output_file = file(output_path, 'w')
+
+ parameters = {
+ 'architectures': ' '.join(archs),
+ 'content_archive': self.content_archive,
+ 'distribution': self.distribution.name,
+ }
+
+ header = get_template('apt_conf_header.template')
+ output_file.write(file(header).read() % parameters)
+
+ dist_template = file(get_template('apt_conf_dist.template')).read()
+ for suite in suites:
+ parameters['suite'] = suite
+ output_file.write(dist_template % parameters)
+
+ output_file.close()
+
+ def createComponentDirs(self, suites, archs):
+ """Create the content archive's tree for all of its components."""
+ for suite in suites:
+ for component in COMPONENTS:
+ for directory in self.getDirs(archs):
+ path = os.path.join(
+ self.content_archive, self.distribution.name, 'dists',
+ suite, component, directory)
+ if not file_exists(path):
+ self.logger.debug("Creating %s.", path)
+ os.makedirs(path)
+
+ def copyOverrides(self):
+ """Copy overrides into the content archive."""
+ if file_exists(self.config.overrideroot):
+ execute(self.logger, "cp", [
+ "-a",
+ self.config.overrideroot,
+ "%s/" % self.content_archive,
+ ])
+ else:
+ self.logger.debug("Did not find overrides; not copying.")
+
+ def writeContentsTop(self):
+ """Write Contents.top file."""
+ output_filename = os.path.join(
+ self.content_archive, '%s-misc' % self.distribution.name,
+ "Contents.top")
+ parameters = {
+ 'distrotitle': self.distribution.title,
+ }
+ output_file = file(output_filename, 'w')
+ text = file(get_template("Contents.top")).read() % parameters
+ output_file.write(text)
+ output_file.close()
+
+ def runAptFTPArchive(self):
+ """Run apt-ftparchive to produce the Contents files."""
+ execute(self.logger, "apt-ftparchive", [
+ "generate",
+ os.path.join(
+ self.content_archive, "%s-misc" % self.distribution.name,
+ "apt-contents.conf"),
+ ])
+
+ def generateContentsFiles(self):
+ """Generate Contents files."""
+ self.logger.debug(
+ "Running apt in private tree to generate new contents.")
+ self.copyOverrides()
+ self.writeContentsTop()
+ self.runAptFTPArchive()
+
+ def updateContentsFile(self, suite, arch):
+ """Update Contents file, if it has changed."""
+ contents_dir = os.path.join(
+ self.content_archive, self.distribution.name, 'dists', suite)
+ contents_filename = "Contents-%s" % arch
+ last_contents = os.path.join(contents_dir, ".%s" % contents_filename)
+ current_contents = os.path.join(contents_dir, contents_filename)
+
+ # Avoid rewriting unchanged files; mirrors would have to
+ # re-fetch them unnecessarily.
+ if differ_in_content(current_contents, last_contents):
+ self.logger.debug(
+ "Installing new Contents file for %s/%s.", suite, arch)
+
+ new_contents = os.path.join(
+ contents_dir, "%s.gz" % contents_filename)
+ contents_dest = os.path.join(
+ self.config.distsroot, suite, "%s.gz" % contents_filename)
+
+ move_file(current_contents, last_contents)
+ move_file(new_contents, contents_dest)
+ os.chmod(contents_dest, 0664)
+ else:
+ self.logger.debug(
+ "Skipping unmodified Contents file for %s/%s.", suite, arch)
+
+ def updateContentsFiles(self, suites, archs):
+ """Update all Contents files that have changed."""
+ self.logger.debug("Comparing contents files with public tree.")
+ for suite in suites:
+ for arch in archs:
+ self.updateContentsFile(suite, arch)
+
+ def setUp(self):
+ """Prepare configuration and filesystem state for the script's work.
+
+ This is idempotent: run it as often as you like. (For example,
+ a test may call `setUp` prior to calling `main` which again
+ invokes `setUp`).
+ """
+ self.processOptions()
+ self.config = getPubConfig(self.distribution.main_archive)
+ self.content_archive = os.path.join(
+ config.archivepublisher.content_archive_root,
+ self.distribution.name + "-contents")
+ self.setUpContentArchive()
+
+ def main(self):
+ """See `LaunchpadScript`."""
+ self.setUp()
+ suites = self.getPockets()
+ archs = self.getArchs()
+ self.writeAptContentsConf(suites, archs)
+ self.createComponentDirs(suites, archs)
+ self.generateContentsFiles()
+ self.updateContentsFiles(suites, archs)
=== added file 'lib/lp/archivepublisher/tests/test_generate_contents_files.py'
--- lib/lp/archivepublisher/tests/test_generate_contents_files.py 1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/tests/test_generate_contents_files.py 2011-04-19 15:01:04 +0000
@@ -0,0 +1,258 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Test for the `generate-contents-files` script."""
+
+__metaclass__ = type
+
+from optparse import OptionValueError
+import os.path
+from textwrap import dedent
+
+from canonical.config import config
+from canonical.testing.layers import (
+ LaunchpadZopelessLayer,
+ ZopelessDatabaseLayer,
+ )
+from lp.archivepublisher.scripts.generate_contents_files import (
+ differ_in_content,
+ execute,
+ GenerateContentsFiles,
+ move_file,
+ )
+from lp.services.log.logger import DevNullLogger
+from lp.services.scripts.base import LaunchpadScriptFailure
+from lp.services.utils import file_exists
+from lp.testing import TestCaseWithFactory
+
+
+def write_file(filename, content=""):
+ """Write `content` to `filename`, and flush."""
+ output_file = file(filename, 'w')
+ output_file.write(content)
+ output_file.close()
+
+
+def fake_overrides(script, distroseries):
+ """Fake overrides files so `script` can run `apt-ftparchive`."""
+ os.makedirs(script.config.overrideroot)
+
+ components = ['main', 'restricted', 'universe', 'multiverse']
+ architectures = script.getArchs()
+ suffixes = components + ['extra.' + component for component in components]
+ for suffix in suffixes:
+ write_file(os.path.join(
+ script.config.overrideroot,
+ "override.%s.%s" % (distroseries.name, suffix)))
+
+ for component in components:
+ write_file(os.path.join(
+ script.config.overrideroot,
+ "%s_%s_source" % (distroseries.name, component)))
+ for arch in architectures:
+ write_file(os.path.join(
+ script.config.overrideroot,
+ "%s_%s_binary-%s" % (distroseries.name, component, arch)))
+
+
+class TestHelpers(TestCaseWithFactory):
+ """Tests for the module's helper functions."""
+
+ layer = ZopelessDatabaseLayer
+
+ def test_differ_in_content_returns_true_if_one_file_does_not_exist(self):
+ self.useTempDir()
+ write_file('one', self.factory.getUniqueString())
+ self.assertTrue(differ_in_content('one', 'other'))
+
+ def test_differ_in_content_returns_false_for_identical_files(self):
+ self.useTempDir()
+ text = self.factory.getUniqueString()
+ write_file('one', text)
+ write_file('other', text)
+ self.assertFalse(differ_in_content('one', 'other'))
+
+ def test_differ_in_content_returns_true_for_differing_files(self):
+ self.useTempDir()
+ write_file('one', self.factory.getUniqueString())
+ write_file('other', self.factory.getUniqueString())
+ self.assertTrue(differ_in_content('one', 'other'))
+
+ def test_differ_in_content_returns_false_if_neither_file_exists(self):
+ self.useTempDir()
+ self.assertFalse(differ_in_content('one', 'other'))
+
+ def test_execute_raises_if_command_fails(self):
+ logger = DevNullLogger()
+ self.assertRaises(
+ LaunchpadScriptFailure, execute, logger, "/bin/false")
+
+ def test_execute_executes_command(self):
+ self.useTempDir()
+ logger = DevNullLogger()
+ filename = self.factory.getUniqueString()
+ execute(logger, "touch", [filename])
+ self.assertTrue(file_exists(filename))
+
+ def test_move_file_renames_file(self):
+ self.useTempDir()
+ text = self.factory.getUniqueString()
+ write_file("old_name", text)
+ move_file("old_name", "new_name")
+ self.assertEqual(text, file("new_name").read())
+
+ def test_move_file_overwrites_old_file(self):
+ self.useTempDir()
+ write_file("new_name", self.factory.getUniqueString())
+ new_text = self.factory.getUniqueString()
+ write_file("old_name", new_text)
+ move_file("old_name", "new_name")
+ self.assertEqual(new_text, file("new_name").read())
+
+
+class TestGenerateContentsFiles(TestCaseWithFactory):
+ """Tests for the actual `GenerateContentsFiles` script."""
+
+ layer = LaunchpadZopelessLayer
+
+ def makeContentArchive(self):
+ """Prepare a "content archive" directory for script tests."""
+ content_archive = self.makeTemporaryDirectory()
+ config.push("content-archive", dedent("""\
+ [archivepublisher]
+ content_archive_root: %s
+ """ % content_archive))
+ self.addCleanup(config.pop, "content-archive")
+ return content_archive
+
+ def makeDistro(self):
+ """Create a distribution for testing.
+
+ The distribution will have a root directory set up, which will
+ be cleaned up after the test.
+ """
+ return self.factory.makeDistribution(
+ publish_root_dir=unicode(self.makeTemporaryDirectory()))
+
+ def makeScript(self, distribution=None):
+ """Create a script for testing."""
+ if distribution is None:
+ distribution = self.makeDistro()
+ script = GenerateContentsFiles(test_args=['-d', distribution.name])
+ script.logger = DevNullLogger()
+ script.setUp()
+ return script
+
+ def test_name_is_consistent(self):
+ distro = self.factory.makeDistribution()
+ self.assertEqual(
+ GenerateContentsFiles(test_args=['-d', distro.name]).name,
+ GenerateContentsFiles(test_args=['-d', distro.name]).name)
+
+ def test_name_is_unique_for_each_distro(self):
+ self.assertNotEqual(
+ GenerateContentsFiles(
+ test_args=['-d', self.factory.makeDistribution().name]).name,
+ GenerateContentsFiles(
+ test_args=['-d', self.factory.makeDistribution().name]).name)
+
+ def test_requires_distro(self):
+ script = GenerateContentsFiles(test_args=[])
+ self.assertRaises(OptionValueError, script.processOptions)
+
+ def test_requires_real_distro(self):
+ script = GenerateContentsFiles(
+ test_args=['-d', self.factory.getUniqueString()])
+ self.assertRaises(OptionValueError, script.processOptions)
+
+ def test_looks_up_distro(self):
+ distro = self.factory.makeDistribution()
+ script = self.makeScript(distro)
+ self.assertEqual(distro, script.distribution)
+
+ def test_queryDistro(self):
+ distroseries = self.factory.makeDistroSeries()
+ script = self.makeScript(distroseries.distribution)
+ script.processOptions()
+ self.assertEqual(distroseries.name, script.queryDistro('supported'))
+
+ def test_getArchs(self):
+ das = self.factory.makeDistroArchSeries()
+ script = self.makeScript(das.distroseries.distribution)
+ self.assertEqual([das.architecturetag], script.getArchs())
+
+ def test_getSuites(self):
+ script = self.makeScript()
+ distroseries = self.factory.makeDistroSeries(
+ distribution=script.distribution)
+ self.assertIn(distroseries.name, script.getSuites())
+
+ def test_getPockets(self):
+ distro = self.makeDistro()
+ distroseries = self.factory.makeDistroSeries(distribution=distro)
+ package = self.factory.makeSuiteSourcePackage(distroseries)
+ script = self.makeScript(distro)
+ os.makedirs(os.path.join(script.config.distsroot, package.suite))
+ self.assertEqual([package.suite], script.getPockets())
+
+ def test_writeAptContentsConf_writes_header(self):
+ self.makeContentArchive()
+ distro = self.makeDistro()
+ script = self.makeScript(distro)
+ script.writeAptContentsConf([], [])
+ apt_contents_conf = file(
+ "%s/%s-misc/apt-contents.conf"
+ % (script.content_archive, distro.name)).read()
+ self.assertIn('\nDefault\n{', apt_contents_conf)
+ self.assertIn(distro.name, apt_contents_conf)
+
+ def test_writeAptContentsConf_writes_suite_sections(self):
+ content_archive = self.makeContentArchive()
+ distro = self.makeDistro()
+ script = self.makeScript(distro)
+ suite = self.factory.getUniqueString('suite')
+ arch = self.factory.getUniqueString('arch')
+ script.writeAptContentsConf([suite], [arch])
+ apt_contents_conf = file(
+ "%s/%s-misc/apt-contents.conf"
+ % (script.content_archive, distro.name)).read()
+ self.assertIn('tree "dists/%s"\n' % suite, apt_contents_conf)
+ overrides_path = os.path.join(
+ content_archive, distro.name + "-contents",
+ distro.name + "-overrides")
+ self.assertIn('FileList "%s' % overrides_path, apt_contents_conf)
+ self.assertIn('Architectures "%s source";' % arch, apt_contents_conf)
+
+ def test_writeContentsTop(self):
+ content_archive = self.makeContentArchive()
+ distro = self.makeDistro()
+ script = self.makeScript(distro)
+ script.writeContentsTop()
+ contents_top = file(
+ "%s/%s-contents/%s-misc/Contents.top"
+ % (content_archive, distro.name, distro.name)).read()
+ self.assertIn("This file maps", contents_top)
+ self.assertIn(distro.title, contents_top)
+
+ def test_main(self):
+ self.makeContentArchive()
+ distro = self.makeDistro()
+ distroseries = self.factory.makeDistroSeries(distribution=distro)
+ processor = self.factory.makeProcessor()
+ das = self.factory.makeDistroArchSeries(
+ distroseries=distroseries, processorfamily=processor.family)
+ package = self.factory.makeSuiteSourcePackage(distroseries)
+ self.factory.makeSourcePackagePublishingHistory(
+ distroseries=distroseries, pocket=package.pocket)
+ self.factory.makeBinaryPackageBuild(
+ distroarchseries=das, pocket=package.pocket,
+ processor=processor)
+ suite = package.suite
+ script = self.makeScript(distro)
+ os.makedirs(os.path.join(script.config.distsroot, package.suite))
+ self.assertNotEqual([], script.getPockets())
+ fake_overrides(script, distroseries)
+ script.main()
+ self.assertTrue(file_exists(os.path.join(
+ script.config.distsroot, suite,
+ "Contents-%s.gz" % das.architecturetag)))
=== added file 'scripts/generate-contents-files.py'
--- scripts/generate-contents-files.py 1970-01-01 00:00:00 +0000
+++ scripts/generate-contents-files.py 2011-04-19 15:01:04 +0000
@@ -0,0 +1,19 @@
+#!/usr/bin/python -S
+#
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Master distro publishing script."""
+
+import _pythonpath
+
+from canonical.config import config
+from lp.archivepublisher.scripts.generate_contents_files import (
+ GenerateContentsFiles,
+ )
+
+
+if __name__ == '__main__':
+ script = GenerateContentsFiles(
+ "generate-contents", dbuser=config.archivepublisher.dbuser)
+ script.lock_and_run()
Follow ups
