launchpad-dev team mailing list archive

[Ian Booth <ian.booth@xxxxxxxxxxxxx>]

Hi all

I've been doing a bit of investigation on running the Launchpad test
suite in a canonistack instance. Andrew Glen-Young has been awesome in
helping me get access to an instance to work with - thanks.

tl;dr; - it's looking pretty good (3 simple issues to fix)

Tests with failures:

lp.archivepublisher.tests.test_ftparchive.TestFTPArchive.test_generateConfig
lp.archiveuploader.tests.test_uploadprocessor.TestUploadProcessor.testLZMADebUpload
lp.services.mailman.tests.test_mlist_sync.TestMListSync.test_staging_sync
lp.services.mailman.tests.test_mlist_sync.TestMListSync.test_staging_sync_list_without_team
lp.services.mailman.tests.test_mlist_sync.TestMListSync.test_staging_sync_with_team_address
Total: 17029 tests, 5 failures, 0 errors in 210 minutes 43.985 seconds.

-----------------------

Firstly, here's how I prepared the blank instance for a test run. I ran
up an oneiric instance provided by Andrew with type medium:

m1.medium:
- Memory: 4096MB
- VCPUS: 2
- Additional Storage: 40GB

I hacked together 3 scripts, based on snippets extracted from various
bits of our python stuff used to configure ec2:

rootsetup.sh (run as root)
setup.sh
databasesetup.sh

The above are bash scripts - I find these easier to write and debug
rather than how our ec2 stuff is written by creating a connection and
shoving bash commands down the pipe to be executed. I scp'ed these
scripts to the blank instance and ran them. I could have also used sftp.

Then I simply ran the tests: bin/test -vv | tee test_results.log

rootsetup.sh - installs all the required launchpad dependencies using
apt. We could do this and then snapshot the image and same a bit of time
next time but (I assume) because it's running in the DC, it's really
very fast.

setup.sh - pulls the lp branch to test. I just used trunk.

Quick look at the failures:

1. test_generateConfig

Failure due to running on oneric, extra line of text in output:
SHA512: fb8dc6eab7c520e.......

Simply need to update the test for version > lucid

2. testLZMADebUpload

Not sure, didn't look very hard
Traceback (most recent call last):
  File
"/var/launchpad/test/lib/lp/archiveuploader/tests/test_uploadprocessor.py",
line 1428, in testLZMADebUpload
    % queue_items.count())
AssertionError: 0 != 1: Expected one 'bar' item in the queue, actually
got 0.

Fails on Python 2.7 as well.

3. TestMListSync failures

All the same root cause - a Python / egg version incompatibility. Tests
run with Python 2.6

<snip>
    import httplib2
  File
"/var/launchpad/test/eggs/httplib2-0.6.0-py2.6.egg/httplib2/__init__.py", line
752, in <module>
    class HTTPSConnectionWithTimeout(httplib.HTTPSConnection):
    AttributeError: 'module' object has no attribute 'HTTPSConnection'

But if I start python (2.7) from the command line:
import httplib
dir(httplib)	
['ACCEPTED', ......, 'HTTPSConnection', ...]

Running with Python 2.7 fixes the above issue, but there's now 1
different failure:

  File
"/var/launchpad/test/lib/lp/services/mailman/tests/test_mlist_sync.py",
line 142, in test_staging_sync_list_without_team
    self.assertEqual(list_summary, self.getListInfo())
AssertionError: !=:
reference = [('team-1',
  'lists.launchpad.dev',
  'http://lists.launchpad.dev/mailman/',
  'team-1@xxxxxxxxxxxxxxxxxxx')]
actual = [('no-team',
  'lists.launchpad.dev',
  'http://lists.launchpad.dev/mailman/',
  'no-team@xxxxxxxxxxxxxxxxxxx not found'),
 ('team-1',
  'lists.launchpad.dev',
  'http://lists.launchpad.dev/mailman/',
  'team-1@xxxxxxxxxxxxxxxxxxx')]


TODO:

- productise the simple scripts I used to set everything up, and add in
the missing stuff to submit to pqm etc. I expect we could abstract out
the EC2Instance object and create a CanonistackInstance and use the
existing harness stuff with either instance. There's a bit of ec2 stuff
hardcoded in there and also a command or two that is not required for
canonistack that would need looking at also.

- finish making our tests run on Python 2.7 (which is something we
really must do anyway before precise) and most of the few errors I noted
above will go away

- create a new base image after rootsetup.sh is run to (slightly) reduce
the rampup time before tests actually start running

I probably won't be the one to do the above - Curtis will be most
unhappy if I don't do disclosure/sharing stuff. But I'm happy to work
with whoever we decide should pick up the ball to get to the next step.
If we decide we can wait a bit, I could well some more in my spare time,
but as I said, I'm meant to be on the disclosure feature and can't
devote too much work time to it.

Attachment: databasesetup.sh
Description: application/shellscript

Attachment: rootsetup.sh
Description: application/shellscript

Attachment: setup.sh
Description: application/shellscript