launchpad-dev team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

Francis gave a good answer, even if it did not work for him, of how to look at this interactively.

Unfortunately, in your branch, the yuixhr_fixture tests pass interactively, and fail within the test suite, so the intended interactive diagnostic path that he described does not work.  I bet it is because run-testapp begins with a different config than the test machinery does before it switches to the local appserver-layer config, and the earlier config leaks over somehow; I saw an instance of leaking earlier, and that might explain the difference.  I'll investigate that later.

All that said, the logs I mentioned namelessly last time (logs/test-appserver-layer.log) still give a pretty good hint at the problem.  It showed that the error was in fact within the publication machinery as I had guessed--in endRequest specifically.  Here's one example of the same repeated error that I saw.

2011-10-30T01:10:58 ERROR root Exception during task
Traceback (most recent call last):
  File "/home/gary/launchpad/deps/eggs/zope.server-3.6.1-py2.6.egg/zope/server/taskthreads
.py", line 48, in handlerThread
    task.service()
  File "/home/gary/launchpad/deps/eggs/zope.server-3.6.1-py2.6.egg/zope/server/serverchann
elbase.py", line 207, in service
    task.service()
  File "/home/gary/launchpad/deps/eggs/zope.server-3.6.1-py2.6.egg/zope/server/http/httpta
sk.py", line 71, in service
    self.channel.server.executeRequest(self)
  File "/home/gary/launchpad/deps/eggs/zc.zservertracelog-1.1.5-py2.6.egg/zc/zservertracel
og/tracelog.py", line 114, in executeRequest
    response = self.application(env, start_response)
  File "/home/gary/launchpad/deps/eggs/zope.app.wsgi-3.6.0-py2.6.egg/zope/app/wsgi/__init_
_.py", line 58, in __call__
    request = publish(request, handle_errors=handle_errors)
  File "/home/gary/launchpad/deps/eggs/zope.publisher-3.12.0-py2.6.egg/zope/publisher/publ
ish.py", line 155, in publish
    publication.endRequest(request, obj)
  File "/home/gary/launchpad/lp/workspace/lib/canonical/launchpad/webapp/publication.py", 
line 769, in endRequest
    sys.exc_info(), request)
  File "/home/gary/launchpad/lp/workspace/lib/canonical/launchpad/webapp/errorlog.py", lin
e 363, in raising
    if self._oops_config.publish(report) is None:
  File "/home/gary/launchpad/deps/eggs/oops-0.0.9-py2.6.egg/oops/config.py", line 184, in 
publish
    id = publisher(report)
  File "/home/gary/launchpad/deps/eggs/oops_amqp-0.0.3-py2.6.egg/oops_amqp/publisher.py", 
line 88, in __call__
    channel = self.get_channel()
  File "/home/gary/launchpad/deps/eggs/oops_amqp-0.0.3-py2.6.egg/oops_amqp/publisher.py", 
line 65, in get_channel
    self.channels.channel = self.connection_factory().channel()
  File "/home/gary/launchpad/lp/workspace/lib/lp/services/messaging/rabbit.py", line 75, i
n connect
    raise MessagingUnavailable("Incomplete configuration")
MessagingUnavailable: Incomplete configuration

So...maybe error_queue_key needs to be set somewhere else?  Or maybe we need to do a dance like the one in TestCaptureOopsNoRabbit to make sure that rabbit is not used in yuixhr?  I bet changes to yuixhr might need to be mirrored for the appserver layer.

I'm not sure what would be better, making sure rabbit is set up correctly, or making sure that rabbit is not used at all.  What do you think?

In any case, you can see where yuixhr sets itself up in runlaunchpad.py's start_testapp.  Please only make surgical changes here--I have three pending branches with changes to yuixhr test machinery, and I'd prefer not to have to resolve big conflicts.  This section is one of them that has big changes.

I'm going to stop the weekend exploration now.  I'm happy to help more later, though this coming week is jam packed with interviews for me so time is tight (which is why I took the time to do this exploration now).

(I attempted a quick hack fix for the heck of it, without really looking closely at your branch.  FWIW, this did not help:

=== modified file 'lib/canonical/launchpad/scripts/runlaunchpad.py'
--- lib/canonical/launchpad/scripts/runlaunchpad.py	2011-10-14 02:59:48 +0000
+++ lib/canonical/launchpad/scripts/runlaunchpad.py	2011-10-29 20:23:34 +0000
@@ -355,6 +355,7 @@
         installFakeConnect,
         uninstallFakeConnect,
         )
+    from lp.testing.fixture import CaptureOops
     assert config.instance_name.startswith('testrunner-appserver'), (
         '%r does not start with "testrunner-appserver"' %
         config.instance_name)
@@ -383,9 +384,12 @@
         # Interactive tests always need this.  We let functional tests use
         # a local one too because of simplicity.
         LayerProcessController.startSMTPServer()
+        oops_fixture = CaptureOops()
+        oops_fixture.setUp()
     try:
         start_launchpad(argv, setup)
     finally:
+        oops_fixture.tearDown()
         LayerProcessController.stopSMTPServer()
         LibrarianLayer.tearDown()
         DatabaseLayer.tearDown()

)

Gary


On Oct 28, 2011, at 6:04 AM, Robert Collins wrote:

> On Fri, Oct 28, 2011 at 2:05 AM, Gary Poster <gary.poster@xxxxxxxxxxxxx> wrote:
>> I'm not here--I'm typing this on my phone--but my guess is that your branch messes with the publication machinery in such a way as to break the yuixhr stuff. The "unknown error" suggests something lowlevel like that. My first guess would be that something is not set up that needs to be set up in the associated runlaunchpad variant for yuixhr.
> 
> RabbitMQ would be the thing then I guess, but I thought that was on
> for most things now (and if its not, the amqp oops code is meant to
> degrade gracefully - is there some way t get debug output on it?).
> Note that all the other yuixhr tests work ok so I would be surprised
> if yuihr is fundamentally borked.
> 
>> I'd be happy to help look at this when I return.  That could conceivably be tomorrow but is more likely to be Monday.  I'd look at the logs when I ran the tests interactively as my first step. Maybe that would let you tackle it?
> 
> What do you mean by that? {Sorry for noob question, but I'm allowed
> one a week :P}
> 
> -Rob