launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #24103
[Merge] ~cjwatson/launchpad:zope.testbrowser.wsgi into launchpad:master
Colin Watson has proposed merging ~cjwatson/launchpad:zope.testbrowser.wsgi into launchpad:master.
Commit message:
Convert to zope.testbrowser.wsgi
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/375102
This is the way forward for use of zope.testbrowser: the old approach based on wsgi_intercept was deprecated in 4.0.0 and removed in 5.0.0.
This also removes most, though not quite all, of our use of zope.app.testing: we no longer use its HTTPCaller, instead using a slight variant of code from zope.app.wsgi.testlayer.
I've added a variety of small workarounds to adjust the test WSGI application's behaviour to be as close to that previously implemented in zope.app.testing as possible. In most cases, my goal was to keep tests unchanged, although a small number of test changes were unavoidable.
There's still some more refactoring needed here that was too large for this branch. wsgi_intercept remains in place until we've rearranged lazr.restful's testing helpers to use WebTest or something similar instead. FunctionalLayer still uses the old style of functional test setup, and should be refactored using zope.app.wsgi.testlayer. Using zope.app.wsgi.testlayer will also allow zope.testbrowser.wsgi.Browser to find the test WSGI application by itself rather than needing it to be passed explicitly.
I included some small preliminary commits to fix up tests where the new framework is stricter, and to fix a non-obvious landmine in YUITestLayer due to the strange way that zope.testrunner handles layer inheritance.
--
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:zope.testbrowser.wsgi into launchpad:master.
diff --git a/constraints.txt b/constraints.txt
index d133f3d..0e9c765 100644
--- a/constraints.txt
+++ b/constraints.txt
@@ -148,7 +148,8 @@ zope.app.appsetup==3.15.0
zope.app.debug==3.4.1
zope.app.http==3.9.0
zope.app.publication==3.12.0
-zope.app.wsgi==3.10.0
+#zope.app.wsgi==3.10.0
+zope.app.wsgi==3.15.0
#zope.testbrowser==3.10.4
zope.testbrowser[wsgi]==4.0.4
diff --git a/lib/lp/app/stories/basics/max-batch-size.txt b/lib/lp/app/stories/basics/max-batch-size.txt
index d5e6de5..0954057 100644
--- a/lib/lp/app/stories/basics/max-batch-size.txt
+++ b/lib/lp/app/stories/basics/max-batch-size.txt
@@ -5,8 +5,7 @@ batching have a maximum on the batch size. For example, requesting 1000
products will display a page telling the users that the batch is too
large and what is the current maximum.
- >>> from zope.app.testing.testbrowser import Browser
- >>> anon_browser = Browser()
+ >>> anon_browser.handleErrors = True
>>> anon_browser.open('http://launchpad.test/projects/+all?start=0&batch=1000')
Traceback (most recent call last):
...
diff --git a/lib/lp/app/stories/basics/xx-developerexceptions.txt b/lib/lp/app/stories/basics/xx-developerexceptions.txt
index a63c448..86e7d1e 100644
--- a/lib/lp/app/stories/basics/xx-developerexceptions.txt
+++ b/lib/lp/app/stories/basics/xx-developerexceptions.txt
@@ -96,9 +96,9 @@ unregister the adapter.
>>> error_view_fixture.cleanUp()
-= HTTPCaller handle_errors =
+= http handle_errors =
-The HTTPCaller instance accepts the handle_errors parameter in case you
+lp.testing.pages.http accepts the handle_errors parameter in case you
want to see tracebacks instead of error pages.
>>> print http(r"""
diff --git a/lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt b/lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt
index 2290a52..191c3e1 100644
--- a/lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt
+++ b/lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt
@@ -61,6 +61,7 @@ treat all empty-equivalent values equally.
... Authorization: Basic test@xxxxxxxxxxxxx:test
... Referer: https://launchpad.test/
... Content-Type: multipart/form-data; boundary=---------------------------2051078912280543729816242321
+ ...
... -----------------------------2051078912280543729816242321
... Content-Disposition: form-data; name="field.title"
...
diff --git a/lib/lp/services/database/doc/storm-store-reset.txt b/lib/lp/services/database/doc/storm-store-reset.txt
index 990f9c1..564feb6 100644
--- a/lib/lp/services/database/doc/storm-store-reset.txt
+++ b/lib/lp/services/database/doc/storm-store-reset.txt
@@ -16,7 +16,6 @@ we rely on that to find out whether or not to reset stores.
>>> import threading
>>> from lp.registry.model.person import Person
>>> from lp.services.database.interfaces import IStore
- >>> from lp.testing.pages import UnstickyCookieHTTPCaller
>>> logout()
>>> alive_items = None
>>> thread_name = None
diff --git a/lib/lp/services/profile/profiling.txt b/lib/lp/services/profile/profiling.txt
index 208fd1f..852a222 100644
--- a/lib/lp/services/profile/profiling.txt
+++ b/lib/lp/services/profile/profiling.txt
@@ -18,7 +18,10 @@ variable.
The pagetests profiler is created by the layer during its setUp.
- >>> from lp.testing.layers import PageTestLayer
+ >>> from lp.testing.layers import (
+ ... PageTestLayer,
+ ... wsgi_application,
+ ... )
(Save the existing configuration.)
@@ -39,15 +42,14 @@ The pagetests profiler is created by the layer during its setUp.
>>> len(PageTestLayer.profiler.getstats())
0
-The layer also replaces the standard HTTPCaller.__call__ by a wrapper
-that takes care of profiling (among other things).
+The layer also adds WSGI middleware that takes care of profiling (among
+other things).
- >>> from zope.app.testing.functional import HTTPCaller
+ >>> from lp.testing.pages import http
# We need to close the default interaction.
>>> logout()
- >>> http = HTTPCaller()
>>> response = http('GET / HTTP/1.0')
>>> profile_count = len(PageTestLayer.profiler.getstats())
>>> profile_count > 0
@@ -55,8 +57,8 @@ that takes care of profiling (among other things).
Requests made with a testbrowser will also be profiled.
- >>> from zope.app.testing.testbrowser import Browser
- >>> browser = Browser()
+ >>> from zope.testbrowser.wsgi import Browser
+ >>> browser = Browser(wsgi_app=wsgi_application)
>>> browser.open('http://launchpad.test/')
>>> len(PageTestLayer.profiler.getstats()) > profile_count
True
diff --git a/lib/lp/services/webapp/doc/test_adapter_timeout.txt.disabled b/lib/lp/services/webapp/doc/test_adapter_timeout.txt.disabled
index e6647cf..aca50bb 100644
--- a/lib/lp/services/webapp/doc/test_adapter_timeout.txt.disabled
+++ b/lib/lp/services/webapp/doc/test_adapter_timeout.txt.disabled
@@ -16,19 +16,22 @@ timed out.
First we create a view that will provoke a TimeoutError, a view for the
exception, and a time machine.
- >>> from zope.publisher.browser import BrowserView
- >>> from zope.app.testing.testbrowser import Browser
>>> from textwrap import dedent
+
+ >>> from storm.exceptions import TimeoutError
+ >>> import zope.component
+ >>> from zope.interface import Interface, implementer
+ >>> from zope.publisher.browser import BrowserView
>>> from zope.publisher.interfaces.browser import (
... IBrowserRequest, IBrowserView, IBrowserPublisher)
- >>> from zope.interface import Interface, implementer
>>> from zope.security.checker import CheckerPublic, MultiChecker
- >>> import zope.component
+
+ >>> from lp.services.config import config
>>> import lp.services.webapp.adapter
>>> from lp.services.webapp.interfaces import (
... IStoreSelector, MAIN_STORE, MASTER_FLAVOR)
- >>> from lp.services.config import config
- >>> from storm.exceptions import TimeoutError
+ >>> from lp.testing.pages import setupBrowser
+
>>> config.push('set_timeout', dedent('''
... [database]
... db_statement_timeout = 5000
@@ -82,8 +85,7 @@ Now we actually demonstrate the behaviour. The view did raise a TimeoutError.
>>> from lp.testing.fixture import CaptureOops
>>> capture = CaptureOops()
>>> capture.setUp()
- >>> browser = Browser()
- >>> browser.handleErrors = False
+ >>> browser = setupBrowser()
>>> browser.open('http://launchpad.test/doom_test')
Traceback (most recent call last):
...
diff --git a/lib/lp/services/webapp/doc/webapp-publication.txt b/lib/lp/services/webapp/doc/webapp-publication.txt
index a8ad43a..2c2c608 100644
--- a/lib/lp/services/webapp/doc/webapp-publication.txt
+++ b/lib/lp/services/webapp/doc/webapp-publication.txt
@@ -458,16 +458,14 @@ less nice.
>>> logger.setLevel(logging.CRITICAL)
>>> logout()
- >>> from zope.app.testing.functional import HTTPCaller
- >>> http = HTTPCaller()
+ >>> from lp.testing.pages import http
>>> print http("GET / HTTP/1.1\n"
... "Host: nosuchhost.launchpad.test")
HTTP/1.1 404 Not Found
...
>>> print http("GET /foo/bar HTTP/1.1\n"
- ... "Host: xmlrpc.launchpad.test\n"
- ... "X-zope-handle-errors: False")
+ ... "Host: xmlrpc.launchpad.test")
HTTP/1.1 405 Method Not Allowed
Allow: POST
...
diff --git a/lib/lp/services/webapp/tests/login.txt b/lib/lp/services/webapp/tests/login.txt
index 1f52f85..ec6eeed 100644
--- a/lib/lp/services/webapp/tests/login.txt
+++ b/lib/lp/services/webapp/tests/login.txt
@@ -8,6 +8,7 @@ link, they'll be sent to the OP to authenticate.
# Set handleErrors to True so that the Unauthorized exception is handled
# by the publisher and we get redirected to the +login page.
+ >>> from lp.testing.browser import Browser
>>> from lp.testing.layers import BaseLayer
>>> root_url = BaseLayer.appserver_root_url()
>>> browser = Browser()
diff --git a/lib/lp/services/webapp/tests/test_haproxy.py b/lib/lp/services/webapp/tests/test_haproxy.py
index 45f21b7..5c83a17 100644
--- a/lib/lp/services/webapp/tests/test_haproxy.py
+++ b/lib/lp/services/webapp/tests/test_haproxy.py
@@ -8,8 +8,6 @@ __all__ = []
from textwrap import dedent
-from zope.app.testing.functional import HTTPCaller
-
from lp.services.config import config
from lp.services.database.policy import (
DatabaseBlockedPolicy,
@@ -19,6 +17,7 @@ from lp.services.webapp import haproxy
from lp.services.webapp.servers import LaunchpadTestRequest
from lp.testing import TestCase
from lp.testing.layers import FunctionalLayer
+from lp.testing.pages import http
class HAProxyIntegrationTest(TestCase):
@@ -26,18 +25,17 @@ class HAProxyIntegrationTest(TestCase):
def setUp(self):
TestCase.setUp(self)
- self.http = HTTPCaller()
self.original_flag = haproxy.going_down_flag
self.addCleanup(haproxy.set_going_down_flag, self.original_flag)
def test_HAProxyStatusView_all_good_returns_200(self):
- result = self.http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
+ result = http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
self.assertEqual(200, result.getStatus())
def test_authenticated_HAProxyStatusView_works(self):
# We don't use authenticated requests, but this keeps us from
# generating oopses.
- result = self.http(
+ result = http(
u'GET /+haproxy HTTP/1.0\n'
u'Authorization: Basic Zm9vLmJhckBjYW5vbmljYWwuY29tOnRlc3Q=\n',
handle_errors=False)
@@ -45,7 +43,7 @@ class HAProxyIntegrationTest(TestCase):
def test_HAProxyStatusView_going_down_returns_500(self):
haproxy.set_going_down_flag(True)
- result = self.http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
+ result = http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
self.assertEqual(500, result.getStatus())
def test_haproxy_url_uses_DatabaseBlocked_policy(self):
@@ -67,5 +65,5 @@ class HAProxyIntegrationTest(TestCase):
'''))
self.addCleanup(config.pop, 'change_haproxy_status_code')
haproxy.set_going_down_flag(True)
- result = self.http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
+ result = http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
self.assertEqual(499, result.getStatus())
diff --git a/lib/lp/services/webapp/tests/test_login.py b/lib/lp/services/webapp/tests/test_login.py
index eaa0d3e..3d93b8c 100644
--- a/lib/lp/services/webapp/tests/test_login.py
+++ b/lib/lp/services/webapp/tests/test_login.py
@@ -38,11 +38,11 @@ from testtools.matchers import (
Equals,
MatchesListwise,
)
-from zope.app.testing.testbrowser import Browser as TestBrowser
from zope.component import getUtility
from zope.security.management import newInteraction
from zope.security.proxy import removeSecurityProxy
from zope.session.interfaces import ISession
+from zope.testbrowser.wsgi import Browser as TestBrowser
from lp.registry.interfaces.person import IPerson
from lp.services.database.interfaces import (
@@ -81,6 +81,7 @@ from lp.testing.layers import (
AppServerLayer,
DatabaseFunctionalLayer,
FunctionalLayer,
+ wsgi_application,
)
from lp.testing.pages import (
extract_text,
@@ -731,7 +732,7 @@ class TestMissingServerShowsNiceErrorPage(TestCase):
fixture.replacement = OpenIDLoginThatFailsDiscovery
self.useFixture(fixture)
- browser = TestBrowser()
+ browser = TestBrowser(wsgi_app=wsgi_application)
self.assertRaises(HTTPError,
browser.open, 'http://launchpad.test/+login')
self.assertEqual('503 Service Unavailable',
diff --git a/lib/lp/services/webservice/stories/xx-service.txt b/lib/lp/services/webservice/stories/xx-service.txt
index ee15cd4..24c7931 100644
--- a/lib/lp/services/webservice/stories/xx-service.txt
+++ b/lib/lp/services/webservice/stories/xx-service.txt
@@ -104,13 +104,12 @@ request, with the OAuth consumer name being equal to the User-Agent.
None
>>> logout()
- >>> from zope.app.testing.functional import HTTPCaller
+ >>> from lp.testing.pages import http
>>> def request_with_user_agent(agent, url="/devel"):
... if agent is None:
... agent_string = ''
... else:
... agent_string = '\nUser-Agent: %s' % agent
- ... http = HTTPCaller()
... request = ("GET %s HTTP/1.1\n"
... "Host: api.launchpad.test"
... "%s\n\n") % (url, agent_string)
diff --git a/lib/lp/soyuz/browser/tests/test_publishing.py b/lib/lp/soyuz/browser/tests/test_publishing.py
index 7df3736..24d41dd 100644
--- a/lib/lp/soyuz/browser/tests/test_publishing.py
+++ b/lib/lp/soyuz/browser/tests/test_publishing.py
@@ -12,7 +12,6 @@ from testtools.matchers import (
Contains,
MatchesAll,
)
-from zope.app.testing.functional import HTTPCaller
from zope.component import getUtility
from zope.publisher.interfaces import NotFound
from zope.security.interfaces import Unauthorized
@@ -37,6 +36,7 @@ from lp.testing import (
TestCaseWithFactory,
)
from lp.testing.layers import LaunchpadFunctionalLayer
+from lp.testing.pages import http
from lp.testing.sampledata import ADMIN_EMAIL
@@ -182,7 +182,7 @@ class TestSourcePackagePublishingHistoryNavigation(TestCaseWithFactory):
canonical_url(spph, path_only_if_possible=True)
+ '/+files/changelog')
logout()
- response = str(HTTPCaller()("GET %s HTTP/1.1\n\n" % redir_url))
+ response = str(http("GET %s HTTP/1.1\n\n" % redir_url))
self.assertThat(
response,
MatchesAll(
diff --git a/lib/lp/testing/doc/pagetest-helpers.txt b/lib/lp/testing/doc/pagetest-helpers.txt
index 21b6e8f..0676eb0 100644
--- a/lib/lp/testing/doc/pagetest-helpers.txt
+++ b/lib/lp/testing/doc/pagetest-helpers.txt
@@ -69,11 +69,11 @@ Using Raw HTTP Requests
-----------------------
Altough testbrowser is very convenient, sometimes more control over the
-request is needed. For these cases, there is a HTTPCaller instance
-available under 'http' that can be used to send raw HTTP request.
+request is needed. For these cases, there is a function available under
+'http' that can be used to send raw HTTP request.
>>> test.globs['http']
- <...HTTPCaller...>
+ <function http ...>
Helper Routines for Testing Page Content
diff --git a/lib/lp/testing/layers.py b/lib/lp/testing/layers.py
index 984e87b..c38af8e 100644
--- a/lib/lp/testing/layers.py
+++ b/lib/lp/testing/layers.py
@@ -56,6 +56,7 @@ __all__ = [
from cProfile import Profile
import datetime
import errno
+from functools import partial
import gc
import logging
import os
@@ -79,12 +80,16 @@ from fixtures import (
MonkeyPatch,
)
import psycopg2
+from six.moves.urllib.parse import quote
from storm.zope.interfaces import IZStorm
import transaction
+from webob.request import environ_from_url as orig_environ_from_url
import wsgi_intercept
from wsgi_intercept import httplib2_intercept
-from zope.app.publication.httpfactory import chooseClasses
-import zope.app.testing.functional
+from zope.app.publication.httpfactory import (
+ chooseClasses,
+ HTTPPublicationRequestFactory,
+ )
from zope.app.testing.functional import (
FunctionalTestSetup,
ZopePublication,
@@ -141,6 +146,7 @@ from lp.services.webapp.interfaces import IOpenLaunchBag
from lp.services.webapp.servers import (
LaunchpadAccessLogger,
register_launchpad_request_publication_factories,
+ wsgi_native_string,
)
import lp.services.webapp.session
from lp.testing import (
@@ -153,7 +159,6 @@ from lp.testing.pgsql import PgTestSetup
from lp.testing.smtpd import SMTPController
-orig__call__ = zope.app.testing.functional.HTTPCaller.__call__
COMMA = ','
WAIT_INTERVAL = datetime.timedelta(seconds=180)
@@ -988,7 +993,7 @@ class LaunchpadLayer(LibrarianLayer, MemcachedLayer, RabbitMQLayer):
"DELETE FROM SessionData")
-def wsgi_application(environ, start_response):
+def raw_wsgi_application(environ, start_response):
"""This is a wsgi application for Zope functional testing.
We use it with wsgi_intercept, which is itself mostly interesting
@@ -1007,6 +1012,9 @@ def wsgi_application(environ, start_response):
# recognize. httplib2 usually takes care of this, but we've
# bypassed that code in our test environment.
environ['REQUEST_METHOD'] = environ['REQUEST_METHOD'].upper()
+ # zope.app.testing.functional.HTTPCaller used to set this, but WebTest
+ # doesn't. However, some tests rely on it.
+ environ.setdefault('REMOTE_ADDR', wsgi_native_string('127.0.0.1'))
# Now we do the proper dance to get the desired request. This is an
# almalgam of code from zope.app.testing.functional.HTTPCaller and
# zope.publisher.paste.Application.
@@ -1029,6 +1037,17 @@ def wsgi_application(environ, start_response):
return response.consumeBodyIter()
+_wsgi_application_middlewares = []
+
+
+def wsgi_application(environ, start_response):
+ """As `raw_wsgi_application`, but possibly wrapped in middleware."""
+ app = raw_wsgi_application
+ for middleware in reversed(_wsgi_application_middlewares):
+ app = middleware(app)
+ return app(environ, start_response)
+
+
class FunctionalLayer(BaseLayer):
"""Loads the Zope3 component architecture in appserver mode."""
@@ -1059,10 +1078,24 @@ class FunctionalLayer(BaseLayer):
'api.launchpad.test', 80, lambda: wsgi_application)
httplib2_intercept.install()
+ # webob.request.environ_from_url defaults to HTTP/1.0, which is
+ # somewhat unhelpful and breaks some tests (due to e.g. differences
+ # in status codes used for redirections). Patch this to default to
+ # HTTP/1.1 instead.
+ def environ_from_url_http11(path):
+ env = orig_environ_from_url(path)
+ env['SERVER_PROTOCOL'] = 'HTTP/1.1'
+ return env
+
+ FunctionalLayer._environ_from_url_http11 = MonkeyPatch(
+ 'webob.request.environ_from_url', environ_from_url_http11)
+ FunctionalLayer._environ_from_url_http11.setUp()
+
@classmethod
@profiled
def tearDown(cls):
FunctionalLayer.isSetUp = False
+ FunctionalLayer._environ_from_url_http11.cleanUp()
wsgi_intercept.remove_wsgi_intercept('localhost', 80)
wsgi_intercept.remove_wsgi_intercept('api.launchpad.test', 80)
httplib2_intercept.uninstall()
@@ -1544,18 +1577,18 @@ class MockHTTPTask:
request_data = MockHTTPRequestParser()
channel = MockHTTPServerChannel()
- def __init__(self, response, first_line):
- self.request = response._request
+ def __init__(self, first_line, request, response_status, response_headers):
+ self.request = request
# We have no way of knowing when the task started, so we use
# the current time here. That shouldn't be a problem since we don't
# care about that for our tests anyway.
self.start_time = time.time()
- self.status = response.getStatus()
+ self.status = response_status
# When streaming files (see lib/zope/publisher/httpresults.txt)
# the 'Content-Length' header is missing. When it happens we set
# 'bytes_written' to an obviously invalid value. This variable is
# used for logging purposes, see webapp/servers.py.
- content_length = response.getHeader('Content-Length')
+ content_length = response_headers.get('Content-Length')
if content_length is not None:
self.bytes_written = int(content_length)
else:
@@ -1567,6 +1600,60 @@ class MockHTTPTask:
return self.request._orig_env
+class ProfilingMiddleware:
+ """Middleware to profile WSGI responses."""
+
+ def __init__(self, app, profiler=None):
+ self.app = app
+ self.profiler = profiler
+
+ def __call__(self, environ, start_response):
+ if self.profiler is not None:
+ start_response = partial(self.profiler.runcall, start_response)
+ return self.app(environ, start_response)
+
+
+class AccessLoggingMiddleware:
+ """Middleware to log page hits."""
+
+ def __init__(self, app, access_logger):
+ self.app = app
+ self.access_logger = access_logger
+
+ def __call__(self, environ, start_response):
+ response_status_string = []
+ response_headers_list = []
+
+ def wrap_start_response(status, headers, exc_info=None):
+ response_status_string.append(status)
+ response_headers_list.extend(headers)
+ return start_response(status, headers, exc_info)
+
+ request = HTTPPublicationRequestFactory(None)(
+ environ['wsgi.input'], environ)
+ # Reconstruct the first line of the request. This isn't completely
+ # accurate, but saving it in such a way that we can get at it from
+ # here is gratuitously annoying. This is similar to parts of
+ # wsgiref.util.request_uri, but with slightly more lenient quoting.
+ url = quote(
+ environ.get('SCRIPT_NAME', '') + environ.get('PATH_INFO', ''),
+ safe='/+')
+ if environ.get('QUERY_STRING'):
+ url += '?' + environ['QUERY_STRING']
+ first_line = '%s %s %s' % (
+ request.method, url, environ['SERVER_PROTOCOL'].rstrip('\n'))
+ entries = []
+ for entry in self.app(environ, wrap_start_response):
+ yield entry
+ entries.append(entry)
+ response_status = int(response_status_string[0].split(' ', 1)[0])
+ # Reversed so that the first of any given header wins. This isn't
+ # very accurate, but is good enough for test middleware.
+ response_headers = dict(reversed(response_headers_list))
+ self.access_logger.log(MockHTTPTask(
+ first_line, request, response_status, response_headers))
+
+
class PageTestLayer(LaunchpadFunctionalLayer, BingServiceLayer):
"""Environment for page tests.
"""
@@ -1585,29 +1672,22 @@ class PageTestLayer(LaunchpadFunctionalLayer, BingServiceLayer):
logger.logger.setLevel(logging.INFO)
access_logger = LaunchpadAccessLogger(logger)
- def my__call__(obj, request_string, handle_errors=True, form=None):
- """Call HTTPCaller.__call__ and log the page hit."""
- if PageTestLayer.profiler:
- response = PageTestLayer.profiler.runcall(
- orig__call__, obj, request_string,
- handle_errors=handle_errors, form=form)
- else:
- response = orig__call__(
- obj, request_string, handle_errors=handle_errors,
- form=form)
- first_line = request_string.strip().splitlines()[0]
- access_logger.log(MockHTTPTask(response._response, first_line))
- return response
-
- PageTestLayer.orig__call__ = (
- zope.app.testing.functional.HTTPCaller.__call__)
- zope.app.testing.functional.HTTPCaller.__call__ = my__call__
+ PageTestLayer._profiling_middleware = partial(
+ ProfilingMiddleware, profiler=PageTestLayer.profiler)
+ PageTestLayer._access_logging_middleware = partial(
+ AccessLoggingMiddleware, access_logger=access_logger)
+ _wsgi_application_middlewares.extend([
+ PageTestLayer._profiling_middleware,
+ PageTestLayer._access_logging_middleware,
+ ])
@classmethod
@profiled
def tearDown(cls):
- zope.app.testing.functional.HTTPCaller.__call__ = (
- PageTestLayer.orig__call__)
+ _wsgi_application_middlewares.remove(
+ PageTestLayer._access_logging_middleware)
+ _wsgi_application_middlewares.remove(
+ PageTestLayer._profiling_middleware)
if PageTestLayer.profiler:
PageTestLayer.profiler.dump_stats(
os.environ.get('PROFILE_PAGETESTS_REQUESTS'))
@@ -2005,6 +2085,26 @@ class TwistedAppServerLayer(TwistedLaunchpadZopelessLayer):
class YUITestLayer(FunctionalLayer):
"""The layer for all YUITests cases."""
+ @classmethod
+ @profiled
+ def setUp(cls):
+ pass
+
+ @classmethod
+ @profiled
+ def tearDown(cls):
+ pass
+
+ @classmethod
+ @profiled
+ def testSetUp(cls):
+ pass
+
+ @classmethod
+ @profiled
+ def testTearDown(cls):
+ pass
+
class YUIAppServerLayer(MemcachedLayer):
"""The layer for all YUIAppServer test cases."""
diff --git a/lib/lp/testing/pages.py b/lib/lp/testing/pages.py
index e7c2fc3..311feac 100644
--- a/lib/lp/testing/pages.py
+++ b/lib/lp/testing/pages.py
@@ -10,9 +10,9 @@ __metaclass__ = type
from contextlib import contextmanager
from datetime import datetime
import doctest
+from io import BytesIO
from itertools import chain
import os
-import pdb
import pprint
import re
import unittest
@@ -45,15 +45,16 @@ from contrib.oauth import (
from lazr.restful.testing.webservice import WebServiceCaller
import six
import transaction
-from zope.app.testing.functional import (
- HTTPCaller,
- SimpleCookie,
- )
-from zope.app.testing.testbrowser import Browser
+from webtest import TestRequest
+from zope.app.wsgi.testlayer import FakeResponse as _FakeResponse
from zope.component import getUtility
from zope.security.management import setSecurityPolicy
from zope.security.proxy import removeSecurityProxy
from zope.session.interfaces import ISession
+from zope.testbrowser.wsgi import (
+ AuthorizationMiddleware,
+ Browser,
+ )
from lp.app.interfaces.launchpad import ILaunchpadCelebrities
from lp.registry.errors import NameAlreadyTaken
@@ -82,7 +83,10 @@ from lp.testing import (
)
from lp.testing.dbuser import dbuser
from lp.testing.factory import LaunchpadObjectFactory
-from lp.testing.layers import PageTestLayer
+from lp.testing.layers import (
+ PageTestLayer,
+ wsgi_application,
+ )
from lp.testing.systemdocs import (
LayeredDocFileSuite,
stop,
@@ -96,44 +100,53 @@ SAMPLEDATA_ACCESS_SECRETS = {
}
-class UnstickyCookieHTTPCaller(HTTPCaller):
- """HTTPCaller subclass that do not carry cookies across requests.
+class FakeResponse(_FakeResponse):
+ """A fake response for use in tests.
- HTTPCaller propogates cookies between subsequent requests.
- This is a nice feature, except it triggers a bug in Launchpad where
- sending both Basic Auth and cookie credentials raises an exception
- (Bug 39881).
+ This is like `zope.app.wsgi.FakeResponse`, but does a better job of
+ emulating `zope.app.testing.functional` by using the request's
+ `SERVER_PROTOCOL` in the response.
"""
- def __init__(self, *args, **kw):
- if kw.get('debug'):
- self._debug = True
- del kw['debug']
- else:
- self._debug = False
- HTTPCaller.__init__(self, *args, **kw)
+ def __init__(self, response, request):
+ self.response = response
+ self.request = request
- def __call__(self, *args, **kw):
- if self._debug:
- pdb.set_trace()
- try:
- return HTTPCaller.__call__(self, *args, **kw)
- finally:
- self.resetCookies()
+ def getOutput(self):
+ output = super(FakeResponse, self).getOutput()
+ protocol = self.request.environ['SERVER_PROTOCOL']
+ if not isinstance(protocol, bytes):
+ protocol = protocol.encode('ISO-8859-1')
+ return (
+ (b'%s %s\n' % (protocol, self.response.status)) +
+ output.split(b'\n', 1)[1])
- def chooseRequestClass(self, method, path, environment):
- """See `HTTPCaller`.
+ __str__ = getOutput
- This version adds the 'PATH_INFO' variable to the environment,
- because some of our factories expects it.
- """
- if 'PATH_INFO' not in environment:
- environment = dict(environment)
- environment['PATH_INFO'] = path
- return HTTPCaller.chooseRequestClass(self, method, path, environment)
- def resetCookies(self):
- self.cookies = SimpleCookie()
+def http(string, handle_errors=True):
+ """Make a test HTTP request.
+
+ This is like `zope.app.wsgi.testlayer.http`, but it forces `SERVER_NAME`
+ and `SERVER_PORT` to be set according to the HTTP Host header. Left to
+ itself, `zope.app.wsgi.testlayer.http` will (via WebOb) set
+ `SERVER_PORT` to 80, which confuses
+ `VirtualHostRequestPublicationFactory.canHandle`.
+ """
+ if not isinstance(string, bytes):
+ string = string.encode('UTF-8')
+ request = TestRequest.from_file(BytesIO(string.lstrip()))
+ request.environ['wsgi.handleErrors'] = handle_errors
+ if 'HTTP_HOST' in request.environ:
+ if ':' in request.environ['HTTP_HOST']:
+ host, port = request.environ['HTTP_HOST'].split(':', 1)
+ else:
+ host = request.environ['HTTP_HOST']
+ port = 80
+ request.environ['SERVER_NAME'] = host
+ request.environ['SERVER_PORT'] = int(port)
+ response = request.get_response(AuthorizationMiddleware(wsgi_application))
+ return FakeResponse(response, request)
class LaunchpadWebServiceCaller(WebServiceCaller):
@@ -148,7 +161,8 @@ class LaunchpadWebServiceCaller(WebServiceCaller):
:param handle_errors: Should errors raise exception or be handled by
the publisher. Default is to let the publisher handle them.
- Other parameters are passed to the HTTPCaller used to make the calls.
+ Other parameters are passed to the WebServiceCaller used to make the
+ calls.
"""
if oauth_consumer_key is not None and oauth_access_key is not None:
# XXX cjwatson 2016-01-25: Callers should be updated to pass
@@ -678,7 +692,7 @@ def setupBrowser(auth=None):
string of the form 'Basic email:password' for an authenticated user.
:return: A `Browser` object.
"""
- browser = Browser()
+ browser = Browser(wsgi_app=AuthorizationMiddleware(wsgi_application))
# Set up our Browser objects with handleErrors set to False, since
# that gives a tracebacks instead of unhelpful error messages.
browser.handleErrors = False
@@ -814,7 +828,7 @@ def permissive_security_policy(dbuser_name=None):
# unconditional.
def setUpGlobs(test, future=False):
test.globs['transaction'] = transaction
- test.globs['http'] = UnstickyCookieHTTPCaller()
+ test.globs['http'] = http
test.globs['webservice'] = LaunchpadWebServiceCaller(
'launchpad-library', 'salgado-change-anything')
test.globs['public_webservice'] = LaunchpadWebServiceCaller(
diff --git a/lib/lp/testing/xmlrpc.py b/lib/lp/testing/xmlrpc.py
index 5fdd450..04a161d 100644
--- a/lib/lp/testing/xmlrpc.py
+++ b/lib/lp/testing/xmlrpc.py
@@ -11,7 +11,6 @@ from cStringIO import StringIO
import httplib
import xmlrpclib
-from zope.app.testing.functional import HTTPCaller
from zope.security.management import (
endInteraction,
queryInteraction,
@@ -21,6 +20,7 @@ from lp.services.webapp.interaction import (
get_current_principal,
setupInteraction,
)
+from lp.testing.pages import http
class _FakeSocket(object):
@@ -35,8 +35,8 @@ class _FakeSocket(object):
return StringIO(self._output)
-class HTTPCallerHTTPConnection(httplib.HTTPConnection):
- """A HTTPConnection which talks to HTTPCaller instead of a real server.
+class TestHTTPConnection(httplib.HTTPConnection):
+ """A HTTPConnection which talks to http() instead of a real server.
Only the methods called by xmlrpclib are overridden.
"""
@@ -44,30 +44,26 @@ class HTTPCallerHTTPConnection(httplib.HTTPConnection):
_data_to_send = ''
_response = None
- def __init__(self, host):
- httplib.HTTPConnection.__init__(self, host)
- self.caller = HTTPCaller()
-
def connect(self):
"""No need to connect."""
pass
def send(self, data):
- """Send the request to HTTPCaller."""
- # We don't send it to HTTPCaller yet, we store the data and sends
+ """Send the request to http()."""
+ # We don't send it to http() yet; we store the data and send
# everything at once when the client requests a response.
self._data_to_send += data
def _zope_response(self):
"""Get the response."""
current_principal = None
- # End and save the current interaction, since HTTPCaller creates
+ # End and save the current interaction, since http() creates
# its own interaction.
if queryInteraction():
current_principal = get_current_principal()
endInteraction()
if self._response is None:
- self._response = self.caller(self._data_to_send)
+ self._response = http(self._data_to_send)
# Restore the interaction to what it was before.
setupInteraction(current_principal)
return self._response
@@ -81,9 +77,9 @@ class HTTPCallerHTTPConnection(httplib.HTTPConnection):
class XMLRPCTestTransport(xmlrpclib.Transport):
- """An XMLRPC Transport which sends the requests to HTTPCaller."""
+ """An XMLRPC Transport which sends the requests to http()."""
def make_connection(self, host):
- """Return our custom HTTPCaller HTTPConnection."""
+ """Return our custom http() HTTPConnection."""
host, self._extra_headers, x509 = self.get_host_info(host)
- return HTTPCallerHTTPConnection(host)
+ return TestHTTPConnection(host)
diff --git a/lib/lp/testopenid/stories/logging-in.txt b/lib/lp/testopenid/stories/logging-in.txt
index 43ce1f1..dc3089e 100644
--- a/lib/lp/testopenid/stories/logging-in.txt
+++ b/lib/lp/testopenid/stories/logging-in.txt
@@ -13,8 +13,8 @@ portion of the authentication process:
>>> from openid.store.memstore import MemoryStore
>>> from lp.testopenid.testing.helpers import (
... complete_from_browser, make_identifier_select_endpoint,
- ... PublisherFetcher)
- >>> setDefaultFetcher(PublisherFetcher())
+ ... ZopeFetcher)
+ >>> setDefaultFetcher(ZopeFetcher())
The authentication process is started by the relying party issuing a
checkid_setup request, sending the user to Launchpad:
diff --git a/lib/lp/testopenid/testing/helpers.py b/lib/lp/testopenid/testing/helpers.py
index bed30cb..c35c8f1 100644
--- a/lib/lp/testopenid/testing/helpers.py
+++ b/lib/lp/testopenid/testing/helpers.py
@@ -8,22 +8,21 @@ __all__ = [
'complete_from_browser',
'EchoView',
'make_identifier_select_endpoint',
- 'PublisherFetcher',
+ 'ZopeFetcher',
]
-import socket
from StringIO import StringIO
-import sys
-import urllib2
from openid import fetchers
from openid.consumer.discover import (
OPENID_IDP_2_0_TYPE,
OpenIDServiceEndpoint,
)
-from zope.app.testing.testbrowser import PublisherConnection
+from six.moves.urllib.error import HTTPError
+from zope.testbrowser.wsgi import Browser
from lp.services.webapp import LaunchpadView
+from lp.testing.layers import wsgi_application
from lp.testopenid.interfaces.server import get_server_url
@@ -39,44 +38,23 @@ class EchoView(LaunchpadView):
return out.getvalue()
-# Grabbed from zope.testbrowser 3.7.0a1, as more recent
-# PublisherHTTPHandlers are for mechanize, so python-openid breaks.
-class PublisherHTTPHandler(urllib2.HTTPHandler):
- """Special HTTP handler to use the Zope Publisher."""
-
- def http_request(self, req):
- # look at data and set content type
- if req.has_data():
- data = req.get_data()
- if isinstance(data, dict):
- req.add_data(data['body'])
- req.add_unredirected_header('Content-type',
- data['content-type'])
- return urllib2.AbstractHTTPHandler.do_request_(self, req)
-
- https_request = http_request
-
- def http_open(self, req):
- """Open an HTTP connection having a ``urllib2`` request."""
- # Here we connect to the publisher.
- if sys.version_info > (2, 6) and not hasattr(req, 'timeout'):
- # Workaround mechanize incompatibility with Python
- # 2.6. See: LP #280334
- req.timeout = socket._GLOBAL_DEFAULT_TIMEOUT
- return self.do_open(PublisherConnection, req)
-
- https_open = http_open
-
-
-class PublisherFetcher(fetchers.Urllib2Fetcher):
- """An `HTTPFetcher` that passes requests on to the Zope publisher."""
- def __init__(self):
- super(PublisherFetcher, self).__init__()
- self.opener = urllib2.build_opener(PublisherHTTPHandler)
-
- def urlopen(self, request):
- request.add_header('X-zope-handle-errors', True)
- return self.opener.open(request)
+class ZopeFetcher(fetchers.HTTPFetcher):
+ """An `HTTPFetcher` based on zope.testbrowser."""
+
+ def fetch(self, url, body=None, headers=None):
+ browser = Browser(wsgi_app=wsgi_application)
+ if headers is not None:
+ for key, value in headers.items():
+ browser.addHeader(key, value)
+ browser.addHeader('X-Zope-Handle-Errors', 'True')
+ try:
+ browser.open(url, data=body)
+ except HTTPError as e:
+ status = e.code
+ else:
+ status = 200
+ return fetchers.HTTPResponse(
+ browser.url, status, browser.headers, browser.contents)
def complete_from_browser(consumer, browser):
diff --git a/setup.py b/setup.py
index 90c4424..ed2da29 100644
--- a/setup.py
+++ b/setup.py
@@ -233,6 +233,8 @@ setup(
'txpkgupload',
'virtualenv-tools3',
'wadllib',
+ 'WebOb',
+ 'WebTest',
'WSGIProxy2',
'z3c.pt',
'z3c.ptcompat',
