launchpad-dev team mailing list archive

[Jonathan Lange <jml@xxxxxxxxxxxxx>]

On Thu, Nov 18, 2010 at 2:43 PM, Danilo Šegan <danilo@xxxxxxxxxxxxx> wrote:
> Hi all,
>
> I've tried a few times to land my branch on db-devel using "ec2 land",
> but can't go past the following failure:
>
> У сре, 17. 11 2010. у 18:28 +0000, Danilo Segan пише:
>> Tests started at approximately 2010-11-17 17:34:04.826732
>> Source: bzr+ssh://bazaar.launchpad.net/~danilo/launchpad/drop-variants r9971
>> Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/ r9977
>>
>> 1461 tests run in 0:54:03.946610, 1 failures, 0 errors
>>
>> Failing tests
>> -------------
>>   lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout
>>
>> ======================================================================
>> FAILURE: lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout (subunit.RemotedTestCase)
>> ----------------------------------------------------------------------
>> _StringException: Text attachment: traceback
>> ------------
>> Traceback (most recent call last):
>> _StringException: Text attachment: traceback
>> ------------
>> Traceback (most recent call last):
>>   File "/var/launchpad/tmp/eggs/testtools-0.9.8_r128-py2.6.egg/testtools/runtest.py", line 143, in _run_user
>>     return fn(*args)
>>   File "/var/launchpad/tmp/eggs/testtools-0.9.8_r128-py2.6.egg/testtools/testcase.py", line 539, in _run_test_method
>>     return self._get_test_method()()
>>   File "/var/launchpad/test/lib/lp/services/job/tests/test_runner.py", line 390, in test_timeout
>>     self.assertEqual(1, len(runner.completed_jobs))
>>   File "/var/launchpad/tmp/eggs/testtools-0.9.8_r128-py2.6.egg/testtools/testcase.py", line 309, in assertEqual
>>     self.assertThat(observed, matcher)
>>   File "/var/launchpad/tmp/eggs/testtools-0.9.8_r128-py2.6.egg/testtools/testcase.py", line 385, in assertThat
>>     % (matchee, matcher, mismatch.describe()))
>> AssertionError: Match failed. Matchee: "2"
>> Matcher: Equals(1)
>> Difference: 1 != 2
>
>
> Looking at the full log (attached), it seems to be a problem handling
> the connection lost error after this test.  The test passes locally for
> me, though.  Could it be that our ec2 image needs the update of
> testtools as well due to the same issue we had locally (no version
> change, or was it for something else?).
>

Pasting the error for the thing that crashes the test suite:

Traceback (most recent call last):
  File "/var/launchpad/test/bin/test", line 275, in <module>
    testrunner.run([])
  File "/var/launchpad/tmp/eggs/zope.testing-3.9.4_p3-py2.6.egg/zope/testing/testrunner/__init__.py",
line 32, in run
    failed = run_internal(defaults, args, script_parts=script_parts)
  File "/var/launchpad/tmp/eggs/zope.testing-3.9.4_p3-py2.6.egg/zope/testing/testrunner/__init__.py",
line 45, in run_internal
    runner.run()
  File "/var/launchpad/tmp/eggs/zope.testing-3.9.4_p3-py2.6.egg/zope/testing/testrunner/runner.py",
line 138, in run
    self.run_tests()
  File "/var/launchpad/tmp/eggs/zope.testing-3.9.4_p3-py2.6.egg/zope/testing/testrunner/runner.py",
line 219, in run_tests
    setup_layers, self.failures, self.errors)
  File "/var/launchpad/tmp/eggs/zope.testing-3.9.4_p3-py2.6.egg/zope/testing/testrunner/runner.py",
line 377, in run_layer
    return run_tests(options, tests, layer_name, failures, errors)
  File "/var/launchpad/tmp/eggs/zope.testing-3.9.4_p3-py2.6.egg/zope/testing/testrunner/runner.py",
line 309, in run_tests
    test(result)
  File "/usr/lib/python2.6/unittest.py", line 300, in __call__
    return self.run(*args, **kwds)
  File "/var/launchpad/test/lib/lp/testing/__init__.py", line 849, in run
    protocol.readFrom(fdread)
  File "/usr/lib/python2.6/dist-packages/subunit/__init__.py", line
521, in readFrom
    self.lostConnection()
  File "/usr/lib/python2.6/dist-packages/subunit/__init__.py", line
511, in lostConnection
    self._state.lostConnection()
  File "/usr/lib/python2.6/dist-packages/subunit/__init__.py", line
363, in lostConnection
    self._outcome_label())
  File "/usr/lib/python2.6/dist-packages/subunit/__init__.py", line
506, in _lostConnectionInTest
    self.client.addError(self._current_test, RemoteError(error_string))
  File "/usr/lib/python2.6/dist-packages/subunit/__init__.py", line
708, in RemoteError
    return (_StringException, _StringException(description), None)
  File "/var/launchpad/tmp/eggs/testtools-0.9.8_r128-py2.6.egg/testtools/testresult/real.py",
line 530, in __init__
    (string,))
TypeError: _StringException expects unicode, got "lost connection
during failure report of test
'lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout'"

Some thoughts:
  * we've been having some problems with testtools, subunit and
unicode recently, partly due to bugs in subunit, partly due to
undocumented strict version dependencies, and partly due to Python 3
being a colossal pain.

  * the first place to look when you see a problem like this is:
    * https://bugs.launchpad.net/subunit and
    * https://bugs.launchpad.net/testtools ... sadly, there's nothing there

  * your hypothesis is fairly easy to check: do the contents of
eggs/testtools-0.9.8_r128-py2.6.egg on an ec2 instance match the
contents on your working local environment?

  * this is arguably *also* a bug in ec2test/remote.py. Failures in
the runner itself ought to send out the full mail rather than the
processed summary. The reason it's generating a summary now is that it
has detected a failure in the suite, so maybe it's not such a bad bug.

  * The error is in ZopeTestInSubProcess, which I know nothing about.
    * Looking at the code, it's just passing in a file descriptor and
passing it to subunit

In summary:
  1. Check up the testtools version hypothesis
  2. It's quite possibly a bug in subunit. The fix would be to change
readFrom or one of the things it calls.

Last time I had to deploy a new version of subunit to our test
builders, it was a big pain. I hope it's better now.

Good luck,
jml