launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #06807
[Merge] lp:~wgrant/launchpad/explode-spec-exploder into lp:launchpad
William Grant has proposed merging lp:~wgrant/launchpad/explode-spec-exploder into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~wgrant/launchpad/explode-spec-exploder/+merge/98560
Remove the obsolete spec wiki notification exploder. As described in bug #54005, the wikis stopped sending mailing to it nearly 2 years ago.
--
https://code.launchpad.net/~wgrant/launchpad/explode-spec-exploder/+merge/98560
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~wgrant/launchpad/explode-spec-exploder into lp:launchpad.
=== removed file 'lib/lp/blueprints/doc/spec-mail-exploder.txt'
--- lib/lp/blueprints/doc/spec-mail-exploder.txt 2011-12-29 05:29:36 +0000
+++ lib/lp/blueprints/doc/spec-mail-exploder.txt 1970-01-01 00:00:00 +0000
@@ -1,270 +0,0 @@
-The Spec Mail Exploder
-======================
-
-There's an address, notifications@xxxxxxxxxxxxxxxxxxx, which looks at
-all incoming email to see if it looks like a change notifications from a
-Moin wiki. If it receive such notification, it will look up the relevant
-Launchpad specification, and re-send the mail to the spec's related
-people.
-
-First, let's take a look at how a Moin change notification looks like:
-
- >>> from lp.services.mail.tests.helpers import read_test_message
- >>> moin_change = read_test_message('moin-change.txt')
-
-It contains some headers, which are of no use to us, and the body is
-base64 encoded.
-
- >>> print str(moin_change) #doctest: -NORMALIZE_WHITESPACE
- From webmaster@xxxxxxxxxx Mon Mar 20 10:31:59 2006
- Return-path: <webmaster@xxxxxxxxxx>
- Content-Type: text/plain; charset="utf-8"
- MIME-Version: 1.0
- Content-Transfer-Encoding: base64
- From: Launchpad Wiki <webmaster@xxxxxxxxxx>
- To: Launchpad Wiki <webmaster@xxxxxxxxxx>
- Date: Mon, 20 Mar 2006 10:26:28 -0000
- Message-ID: <20060320102628.17391.34951@xxxxxxxxxxxxxxxxx>
- Subject: [Launchpad Wiki] Update of "MediaIntegrityCheck" by ...
- Status: RO
- Content-Length: 1220
- <BLANKLINE>
- RGVhciBXaWtpIHVzZXIsDQo...
-
-Let's take a look at the decode body, since we'll have to use it in
-order to extract any useful information about the wiki page.
-
- >>> print moin_change.get_payload(decode=True)
- Dear Wiki user,
- <BLANKLINE>
- You have subscribed to a wiki page or wiki category on "Ubuntu Wiki"
- for change notification.
- <BLANKLINE>
- The following page has been changed by JamesHenstridge:
- https://wiki.ubuntu.com/MediaIntegrityCheck
- <BLANKLINE>
- The comment on the change is:
- A comment about the change
- <BLANKLINE>
- -----------------------------------------------------------------...
- ...
-
-
-The Mail Handler
-----------------
-
-The mail handler that handles mail on notifications@xxxxxxxxxxxxxxxxxxx
-is BlueprintHandler.
-
- >>> from lp.services.config import config
- >>> from lp.services.mail.handlers import mail_handlers
- >>> handler = mail_handlers.get(config.launchpad.specs_domain)
- >>> handler is not None
- True
-
-It has a helper method to help associate a URL with a specification. We
-use this method instead of simply ISpecificationSet.getByURL() since the
-Ubuntu wikis are special, they have more than one domain name mapping to
-the same wiki. So if we start with the correct URL, of course we get the
-specification with that URL.
-
- >>> spec = handler._getSpecByURL(
- ... 'https://wiki.ubuntu.com/MediaIntegrityCheck')
- >>> spec.specurl
- u'https://wiki.ubuntu.com/MediaIntegrityCheck'
-
-But if someone would edit the same wiki page on wiki.edubuntu.org, we
-would want the same spec's subscribers to be notified about it.
-
- >>> spec = handler._getSpecByURL(
- ... 'https://wiki.edubuntu.org/MediaIntegrityCheck')
- >>> spec.specurl
- u'https://wiki.ubuntu.com/MediaIntegrityCheck'
-
-And the same with wiki.kubuntu.org URLs.
-
- >>> spec = handler._getSpecByURL(
- ... 'https://wiki.kubuntu.org/MediaIntegrityCheck')
- >>> spec.specurl
- u'https://wiki.ubuntu.com/MediaIntegrityCheck'
-
-If no matching specification is found, None is returned.
-
- >>> spec = handler._getSpecByURL(
- ... 'https://wiki.kubuntu.org/NonExistant')
- >>> spec is None
- True
-
-We use a logger to see what the handler does:
-
- >>> from lp.services.scripts import log
- >>> OLD_LOG_LEVEL = log._log.getEffectiveLevel()
- >>> log._log.setLevel(10)
-
-Now, if we pass the email to it, we can see that it finds the correct
-spec, and mails it to the related people.
-
- >>> handler.process(
- ... moin_change, 'notifications@xxxxxxxxxxxxxxxxxxx', log=log)
- True
-
- >>> log._log.setLevel(OLD_LOG_LEVEL)
-
- >>> from lp.blueprints.interfaces.specification import ISpecificationSet
- >>> spec = getUtility(ISpecificationSet).getByURL(
- ... 'https://wiki.ubuntu.com/MediaIntegrityCheck')
- >>> spec.notificationRecipientAddresses()
- ['test@xxxxxxxxxxxxx']
-
-Let's look at the email that got sent:
-
- >>> import transaction
- >>> from lp.services.mail import stub
- >>> transaction.commit()
- >>> len(stub.test_emails)
- 1
-
-The email got sent to all related people:
-
- >>> from_addr, to_addrs, raw_message = stub.test_emails.pop()
- >>> to_addrs == spec.notificationRecipientAddresses()
- True
-
-We practically bounced the message, we didn't change anything about it,
-except for setting the Sender header to our bounce address:
-
- >>> import email
- >>> sent_msg = email.message_from_string(raw_message)
- >>> sent_body = sent_msg.get_payload(decode=True)
- >>> sent_body == moin_change.get_payload(decode=True)
- True
-
- >>> sent_msg['To'] == moin_change['To']
- True
-
- >>> sent_msg['From'] == moin_change['From']
- True
-
- >>> sent_msg['Subject'] == moin_change['Subject']
- True
-
- >>> sent_msg['Sender'] == config.canonical.bounce_address
- True
-
-If we get a change notification about a spec which doesn't exist in
-Launchpad, nothing happens.
-
- >>> log._log.setLevel(10)
- >>> moin_change = read_test_message('moin-change-nonexistant.txt')
- >>> handler.process(
- ... moin_change, 'notifications@xxxxxxxxxxxxxxxxxxx', log=log)
- True
-
- >>> print getUtility(ISpecificationSet).getByURL(
- ... 'https://wiki.ubuntu.com/NonExistant')
- None
-
-In order to prevent loops, we set the X-Loop header:
-
- >>> sent_msg['X-Loop']
- 'notifications@xxxxxxxxxxxxxxxxxxx'
-
-So that if we someone re-sends the notification we sent to us, we'll
-simply emit a warning and drop the email.
-
- >>> class FakeFileAlias:
- ... http_url = 'http://librarian/foo.txt'
- >>> del sent_msg['Sender']
- >>> sent_msg['Sender'] = 'webmaster@xxxxxxxxxx'
- >>> handler.process(
- ... sent_msg, 'notifications@xxxxxxxxxxxxxxxxxxx',
- ... filealias=FakeFileAlias(), log=log)
- WARNING...:Got back a notification we sent: http://librarian/foo.txt
- True
-
- >>> log._log.setLevel(OLD_LOG_LEVEL)
-
-No emails were sent:
-
- >>> transaction.commit()
- >>> len(stub.test_emails)
- 0
-
-To prevent bad things happening if someone subscribes this address to
-specifications for example, we ignore all email that are sent from
-Launchpad:
-
- >>> del sent_msg['X-Loop']
- >>> del sent_msg['Sender']
- >>> sent_msg['Sender'] = config.canonical.bounce_address
- >>> log._log.setLevel(10)
- >>> handler.process(
- ... sent_msg, 'notifications@xxxxxxxxxxxxxxxxxxx',
- ... filealias=FakeFileAlias(), log=log)
- WARNING...:We received an email from Launchpad: http://librarian/foo.txt
- True
-
- >>> log._log.setLevel(OLD_LOG_LEVEL)
-
-Again, no emails were sent:
-
- >>> transaction.commit()
- >>> len(stub.test_emails)
- 0
-
-Let's pass a notification from wiki.kubuntu.org regarding
-MediaIntegrityCheck to see that the correct specification will be found,
-and that the notification will be forwarded to the specification's
-subscribers:
-
- >>> kubuntu_change = read_test_message('moin-change-kubuntu.txt')
- >>> log._log.setLevel(10)
- >>> handler.process(
- ... kubuntu_change, 'notifications@xxxxxxxxxxxxxxxxxxx', log=log)
- True
-
- >>> log._log.setLevel(OLD_LOG_LEVEL)
-
- >>> transaction.commit()
- >>> len(stub.test_emails)
- 1
-
- >>> from_addr, to_addrs, raw_message = stub.test_emails.pop()
- >>> to_addrs == spec.notificationRecipientAddresses()
- True
-
-Lastly, let's simulate sending a moin notification, and use handleMail()
-instead to ensure that handler above handles the email. This will make
-sure that the handler is setup properly to handle unknown users, since
-webmaster@xxxxxxxxxx doesn't belong to any Person in Launchpad:
-
- >>> from lp.registry.interfaces.person import IPersonSet
- >>> getUtility(IPersonSet).getByEmail('webmaster@xxxxxxxxxx') is None
- True
-
- >>> moin_change = read_test_message('moin-change.txt')
- >>> moin_change['X-Launchpad-Original-To'] = "notifications@%s" % (
- ... config.launchpad.specs_domain)
- >>> moin_change['Sender'] = 'webmaster@xxxxxxxxxx'
-
- >>> from lp.services.mail.incoming import handleMail
- >>> from lp.services.mail.sendmail import sendmail
- >>> sendmail(moin_change, bulk=False)
- '...'
-
- >>> transaction.commit()
- >>> handleMail()
-
- >>> transaction.commit()
- >>> len(stub.test_emails)
- 1
-
- >>> from_addr, to_addrs, raw_message = stub.test_emails.pop()
- >>> to_addrs == spec.notificationRecipientAddresses()
- True
-
- >>> sent_msg = email.message_from_string(raw_message)
- >>> sent_msg['Message-Id'] == moin_change['Message-Id']
- True
-
-
=== removed file 'lib/lp/blueprints/mail/handler.py'
--- lib/lp/blueprints/mail/handler.py 2012-01-01 02:58:52 +0000
+++ lib/lp/blueprints/mail/handler.py 1970-01-01 00:00:00 +0000
@@ -1,126 +0,0 @@
-# Copyright 2010-2011 Canonical Ltd. This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Handle incoming Blueprints email."""
-
-__metaclass__ = type
-__all__ = [
- "BlueprintHandler",
- ]
-
-import re
-from urlparse import urlunparse
-
-from zope.component import getUtility
-from zope.interface import implements
-
-from lp.blueprints.interfaces.specification import ISpecificationSet
-from lp.services.config import config
-from lp.services.mail.helpers import get_main_body
-from lp.services.mail.interfaces import IMailHandler
-from lp.services.mail.sendmail import sendmail
-from lp.services.webapp import urlparse
-
-
-MOIN_URL_RE = re.compile(r'(https?://[^ \r\n]+)')
-
-
-def get_spec_url_from_moin_mail(moin_text):
- """Extract a specification URL from Moin change notification."""
- if not isinstance(moin_text, basestring):
- return None
- match = MOIN_URL_RE.search(moin_text)
- if match:
- return match.group(1)
- else:
- return None
-
-
-class BlueprintHandler:
- """Handles emails sent to specs.launchpad.net."""
-
- implements(IMailHandler)
-
- allow_unknown_users = True
-
- _spec_changes_address = re.compile(r'^notifications@.*')
-
- # The list of hosts where the Ubuntu wiki is located. We could do a
- # more general solution, but this kind of setup is unusual, and it
- # will be mainly the Ubuntu and Launchpad wikis that will use this
- # notification forwarder.
- UBUNTU_WIKI_HOSTS = [
- 'wiki.ubuntu.com', 'wiki.edubuntu.org', 'wiki.kubuntu.org']
-
- def _getSpecByURL(self, url):
- """Returns a spec that is associated with the URL.
-
- It takes into account that the same Ubuntu wiki is on three
- different hosts.
- """
- scheme, host, path, params, query, fragment = urlparse(url)
- if host in self.UBUNTU_WIKI_HOSTS:
- for ubuntu_wiki_host in self.UBUNTU_WIKI_HOSTS:
- possible_url = urlunparse(
- (scheme, ubuntu_wiki_host, path, params, query,
- fragment))
- spec = getUtility(ISpecificationSet).getByURL(possible_url)
- if spec is not None:
- return spec
- else:
- return getUtility(ISpecificationSet).getByURL(url)
-
- def get_spec_url_from_email(self, signed_msg):
- """Return the first url found in the email body."""
- mail_body = get_main_body(signed_msg)
- return get_spec_url_from_moin_mail(mail_body)
-
- def process(self, signed_msg, to_addr, filealias=None, log=None):
- """See IMailHandler."""
- match = self._spec_changes_address.match(to_addr)
- if not match:
- # We handle only spec-changes at the moment.
- return False
- our_address = "notifications@%s" % config.launchpad.specs_domain
- # Check for emails that we sent.
- xloop = signed_msg['X-Loop']
- if xloop and our_address in signed_msg.get_all('X-Loop'):
- if log and filealias:
- log.warning(
- 'Got back a notification we sent: %s' %
- filealias.http_url)
- return True
- # Check for emails that Launchpad sent us.
- if signed_msg['Sender'] == config.canonical.bounce_address:
- if log and filealias:
- log.warning('We received an email from Launchpad: %s'
- % filealias.http_url)
- return True
- # When sending the email, the sender will be set so that it's
- # clear that we're the one sending the email, not the original
- # sender.
- del signed_msg['Sender']
-
- spec_url = self.get_spec_url_from_email(signed_msg)
- if spec_url is not None:
- if log is not None:
- log.debug('Found a spec URL: %s' % spec_url)
- spec = self._getSpecByURL(spec_url)
- if spec is not None:
- if log is not None:
- log.debug('Found a corresponding spec: %s' % spec.name)
- # Add an X-Loop header, in order to prevent mail loop.
- signed_msg.add_header('X-Loop', our_address)
- notification_addresses = spec.notificationRecipientAddresses()
- if log is not None:
- log.debug(
- 'Sending notification to: %s' %
- ', '.join(notification_addresses))
- sendmail(signed_msg, to_addrs=notification_addresses)
-
- elif log is not None:
- log.debug(
- "Didn't find a corresponding spec for %s" % spec_url)
- elif log is not None:
- log.debug("Didn't find a specification URL")
- return True
=== removed file 'lib/lp/blueprints/mail/tests/test_handler.py'
--- lib/lp/blueprints/mail/tests/test_handler.py 2012-01-01 02:58:52 +0000
+++ lib/lp/blueprints/mail/tests/test_handler.py 1970-01-01 00:00:00 +0000
@@ -1,83 +0,0 @@
-# Copyright 2011 Canonical Ltd. This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Testing the Blueprint email handler."""
-
-__metaclass__ = type
-
-from testtools.matchers import (
- Equals,
- Is,
- )
-
-from lp.blueprints.mail.handler import (
- BlueprintHandler,
- get_spec_url_from_moin_mail,
- )
-from lp.testing import (
- TestCase,
- TestCaseWithFactory,
- )
-from lp.testing.layers import DatabaseFunctionalLayer
-
-
-class TestGetSpecUrlFromMoinMail(TestCase):
- """Tests for get_spec_url_from_moin_mail."""
-
- def test_invalid_params(self):
- # Only strings and unicode are OK.
- self.assertThat(get_spec_url_from_moin_mail(None), Is(None))
- self.assertThat(get_spec_url_from_moin_mail(42), Is(None))
- self.assertThat(get_spec_url_from_moin_mail(object()), Is(None))
-
- def test_missing_urls(self):
- # Strings with missing URLs also return None
- self.assertThat(
- get_spec_url_from_moin_mail('nothing here'),
- Is(None))
-
- def test_string_contains_url(self):
- body = """
- Testing a big string
-
- An url, http://example.com/foo in a string
- """
- self.assertThat(
- get_spec_url_from_moin_mail(body),
- Equals('http://example.com/foo'))
-
- def test_two_urls(self):
- # Given two urls, only the first is returned.
- body = """
- Testing two urls:
- http://example.com/first
- http://example.com/second
- """
- self.assertThat(
- get_spec_url_from_moin_mail(body),
- Equals('http://example.com/first'))
-
- def test_unicode(self):
- # Unicode strings and urls are fine.
- body = u"""
- Testing unicode:
- http://example.com/\N{SNOWMAN}
- """
- self.assertThat(
- get_spec_url_from_moin_mail(body),
- Equals(u'http://example.com/\N{SNOWMAN}'))
-
-
-class TestBlueprintEmailHandler(TestCaseWithFactory):
-
- layer = DatabaseFunctionalLayer
-
- def test_find_url_in_multipart_message(self):
- """Multipart email is common, and we should be able to handle it."""
- message = self.factory.makeSignedMessage(
- body="An url http://example.com/foo in the body.",
- attachment_contents="Nothing here either")
- handler = BlueprintHandler()
- self.assertThat(
- handler.get_spec_url_from_email(message),
- Equals('http://example.com/foo'))
=== modified file 'lib/lp/blueprints/tests/test_doc.py'
--- lib/lp/blueprints/tests/test_doc.py 2011-12-22 11:12:50 +0000
+++ lib/lp/blueprints/tests/test_doc.py 2012-03-21 02:58:23 +0000
@@ -5,30 +5,13 @@
Run the doctests and pagetests.
"""
-import logging
import os
-from lp.services.mail.tests.test_doc import ProcessMailLayer
from lp.services.testing import build_test_suite
-from lp.testing.systemdocs import (
- LayeredDocFileSuite,
- setUp,
- tearDown,
- )
here = os.path.dirname(os.path.realpath(__file__))
-special = {
- 'spec-mail-exploder.txt': LayeredDocFileSuite(
- "../doc/spec-mail-exploder.txt",
- setUp=setUp, tearDown=tearDown,
- layer=ProcessMailLayer,
- stdout_logging=True,
- stdout_logging_level=logging.WARNING),
- }
-
-
def test_suite():
- return build_test_suite(here, special)
+ return build_test_suite(here)
=== modified file 'lib/lp/services/mail/handlers.py'
--- lib/lp/services/mail/handlers.py 2012-01-01 02:58:52 +0000
+++ lib/lp/services/mail/handlers.py 2012-03-21 02:58:23 +0000
@@ -7,7 +7,6 @@
]
from lp.answers.mail.handler import AnswerTrackerHandler
-from lp.blueprints.mail.handler import BlueprintHandler
from lp.bugs.mail.handler import MaloneHandler
from lp.code.mail.codehandler import CodeHandler
from lp.services.config import config
@@ -18,7 +17,6 @@
DEFAULT = (
(config.launchpad.bugs_domain, MaloneHandler),
- (config.launchpad.specs_domain, BlueprintHandler),
(config.answertracker.email_domain, AnswerTrackerHandler),
# XXX flacoste 2007-04-23 Backward compatibility for old domain.
# We probably want to remove it in the future.
