launchpad-reviewers team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

Aaron Bentley has proposed merging lp:~abentley/launchpad/stable-test-failures into lp:launchpad.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)
Related bugs:
  Bug #926322 in Launchpad itself: "test suite fails on oneiric"
  https://bugs.launchpad.net/launchpad/+bug/926322

For more details, see:
https://code.launchpad.net/~abentley/launchpad/stable-test-failures/+merge/91504

= Summary =
Fix bug #926322: test suite fails on oneiric

== Proposed fix ==
Ignore SHA512 lines in tests.

Convert mlist-sync.py to a buildout script so that invoking it requires neither the PYTHONPATH nor the PATH

== Pre-implementation notes ==
Discusssed mlist-sync.py solution with gary.

== Implementation details ==
The fix for TestFTPArchive.test_generateConfig, is simple: skip SHA512 lines, since they may or may not be present.

The mlist-sync bug was subtle; mlist-sync was being invoked with PYTHONPATH, but not python interpreter, specified.  On Oneiric, this caused it to launch Python 2.7 with a PYTHONPATH intended for 2.6.  This ultimately led to python2.6's ssl.py to attempt to import from _ssl, but since _ssl is provided by libpython, it was attempting to import from libpython2.7, and at least one symbol that libpython2.6 has was not provided.  This failure cascaded up to http, which chose not to define HTTPSConnection as a result, and then to httplib2 which breaks if HTTPSConnection is defined.

The solution was to stop specifying the PYTHONPATH, by converting mlist-sync into a buildout script template.  Testing showed that PATH could also be removed.

== Tests ==
bin/test -t test_mlist_sync -t test_generateConfig

== Demo and Q/A ==
None


= Launchpad lint =

Checking for conflicts and issues in changed files.

Linting changed files:
  .bzrignore
  lib/lp/archivepublisher/tests/test_ftparchive.py
  buildout-templates/scripts/mlist-sync.py.in
  Makefile
  lib/lp/services/mailman/tests/test_mlist_sync.py
-- 
https://code.launchpad.net/~abentley/launchpad/stable-test-failures/+merge/91504
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~abentley/launchpad/stable-test-failures into lp:launchpad.

=== modified file '.bzrignore'
--- .bzrignore	2011-11-20 23:37:23 +0000
+++ .bzrignore	2012-02-03 20:57:20 +0000
@@ -76,3 +76,4 @@
 lib/canonical/launchpad/icing/yui
 .emacs.desktop
 callgrind.out.*
+scripts/mlist-sync.py

=== modified file 'Makefile'
--- Makefile	2012-01-26 00:23:55 +0000
+++ Makefile	2012-02-03 20:57:20 +0000
@@ -267,7 +267,8 @@
 	$(PY) cronscripts/merge-proposal-jobs.py -v
 
 run: build inplace stop
-	bin/run -r librarian,google-webservice,memcached,rabbitmq,txlongpoll -i $(LPCONFIG)
+	bin/run -r librarian,google-webservice,memcached,rabbitmq,txlongpoll \
+	-i $(LPCONFIG)
 
 run-testapp: LPCONFIG=testrunner-appserver
 run-testapp: build inplace stop
@@ -284,8 +285,8 @@
 
 run_all: build inplace stop
 	bin/run \
-	 -r librarian,sftp,forker,mailman,codebrowse,google-webservice,memcached,rabbitmq,txlongpoll \
-	 -i $(LPCONFIG)
+	 -r librarian,sftp,forker,mailman,codebrowse,google-webservice,\
+	memcached,rabbitmq,txlongpoll -i $(LPCONFIG)
 
 run_codebrowse: build
 	BZR_PLUGIN_PATH=bzrplugins $(PY) scripts/start-loggerhead.py -f
@@ -374,6 +375,7 @@
 	$(RM) .installed.cfg
 	$(RM) _pythonpath.py
 	$(RM) -r yui/*
+	$(RM) scripts/mlist-sync.py
 
 clean_logs:
 	$(RM) logs/thread*.request

=== added directory 'buildout-templates/scripts'
=== renamed file 'scripts/mlist-sync.py' => 'buildout-templates/scripts/mlist-sync.py.in'
--- scripts/mlist-sync.py	2012-01-01 03:13:08 +0000
+++ buildout-templates/scripts/mlist-sync.py.in	2012-02-03 20:57:20 +0000
@@ -1,10 +1,19 @@
-#!/usr/bin/python -S
-#
-# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
+#!${buildout:executable} -S
+
+# Copyright 2009-2012 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Sync Mailman data from one Launchpad to another."""
 
+# Initialize our paths.
+${python-relative-path-setup}
+
+import sys
+sys.path.insert(0, ${scripts:parts-directory|path-repr})
+import site
+
+# Run the script.
+
 # XXX BarryWarsaw 2008-02-12:
 # Things this script does NOT do correctly.
 #

=== modified file 'lib/lp/archivepublisher/tests/test_ftparchive.py'
--- lib/lp/archivepublisher/tests/test_ftparchive.py	2012-01-20 15:42:44 +0000
+++ lib/lp/archivepublisher/tests/test_ftparchive.py	2012-02-03 20:57:20 +0000
@@ -44,6 +44,11 @@
     return re.subn(r'(?sm)^Checksums-.*?(?=^[^ ])', '', text)[0]
 
 
+def skip_sha512(text):
+    """Ignore SHA512 lines, which are present only in newer distroseries."""
+    return re.sub('SHA512: [0-9a-f]*\n', '', text)
+
+
 class SamplePublisher:
     """Publisher emulation test class."""
 
@@ -312,7 +317,8 @@
         # regressions.
         fa.runApt(apt_conf)
         self._verifyFile("Packages",
-            os.path.join(self._distsdir, "hoary-test", "main", "binary-i386"))
+            os.path.join(self._distsdir, "hoary-test", "main", "binary-i386"),
+            skip_sha512)
         self._verifyEmpty(
             os.path.join(
                 self._distsdir, "hoary-test", "main", "debian-installer",

=== modified file 'lib/lp/services/mailman/tests/test_mlist_sync.py'
--- lib/lp/services/mailman/tests/test_mlist_sync.py	2012-01-01 02:58:52 +0000
+++ lib/lp/services/mailman/tests/test_mlist_sync.py	2012-02-03 20:57:20 +0000
@@ -12,7 +12,6 @@
     PIPE,
     Popen,
     )
-import sys
 import tempfile
 
 from Mailman import mm_cfg
@@ -87,9 +86,7 @@
              self.host_name, source_dir),
             stdout=PIPE, stderr=PIPE,
             cwd=config.root,
-            env=dict(LPCONFIG=DatabaseFunctionalLayer.appserver_config_name,
-                     PYTHONPATH=os.pathsep.join(sys.path),
-                     PATH=os.environ.get('PATH')))
+            env=dict(LPCONFIG=DatabaseFunctionalLayer.appserver_config_name))
         stdout, stderr = proc.communicate()
         return proc.returncode, stderr