launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #05074
[Merge] lp:~wgrant/launchpad/separate-zopeless-mail into lp:launchpad
William Grant has proposed merging lp:~wgrant/launchpad/separate-zopeless-mail into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~wgrant/launchpad/separate-zopeless-mail/+merge/76923
This branch splits Zopeless mail delivery out of the rest of the obsolete Zopeless infrastructure, and renames it to something hopefully a bit less misleading.
Since Zopeless no longer exists except as a compatibility layer, the environments seen by scripts and appservers have largely converged. Except for mail delivery: initZopeless overrides lp.services.mail.sendmail to bypass normal transactional Zope mail delivery, instead delivering immediately directly via SMTP. Bug #29744 asks for this to converge as well, but some scripts still depend on this immediate delivery behaviour.
This branch untangles immediate delivery and ZopelessTransactionManager. For compatibility, all non-test initZopeless callsites now call set_immediate_mail_delivery(True) -- they will hopefully eventually be gradually removed as part of the bug #29744 migration.
The 'zopeless' config section has been renamed as well, as it pertains entirely to mail configuration. Only one production config mentions it: dogfood sets zopeless.send_mail = True, which is the default anyway -- I intend to remove it before landing this.
Suggestions for better names than 'immediate mail delivery' are very welcome.
--
https://code.launchpad.net/~wgrant/launchpad/separate-zopeless-mail/+merge/76923
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~wgrant/launchpad/separate-zopeless-mail into lp:launchpad.
=== modified file 'configs/development/launchpad-lazr.conf'
--- configs/development/launchpad-lazr.conf 2011-09-21 14:38:47 +0000
+++ configs/development/launchpad-lazr.conf 2011-09-26 00:35:27 +0000
@@ -353,7 +353,7 @@
[vhost.vostok]
hostname: vostok.dev
-[zopeless]
+[immediate_mail]
# XXX sinzui 2008-03-26:
# A development box should never send email to the outer world,
# so disable that here. note that the testrunner config inherits
=== modified file 'configs/testrunner-appserver/launchpad-lazr.conf'
--- configs/testrunner-appserver/launchpad-lazr.conf 2011-04-05 23:36:31 +0000
+++ configs/testrunner-appserver/launchpad-lazr.conf 2011-09-26 00:35:27 +0000
@@ -55,7 +55,7 @@
[vhost.feeds]
rooturl: http://feeds.launchpad.dev:8085/
-[zopeless]
+[immediate_mail]
# BarryWarsaw 04-Dec-2008: AppServerLayer tests should send email to the fake
# SMTP server that the layer starts up, so that they can be collected and
# tested.
=== modified file 'daemons/buildd-manager.tac'
--- daemons/buildd-manager.tac 2011-09-18 09:49:40 +0000
+++ daemons/buildd-manager.tac 2011-09-26 00:35:27 +0000
@@ -8,15 +8,19 @@
from twisted.scripts.twistd import ServerOptions
from twisted.web import server
-from lp.buildmaster.manager import BuilddManager
-from lp.services.twistedsupport.loggingsupport import RotatableFileLogObserver
from canonical.config import config
from canonical.database.sqlbase import ZopelessTransactionManager
from canonical.launchpad.daemons import readyservice
from canonical.launchpad.scripts import execute_zcml_for_scripts
+from lp.buildmaster.manager import BuilddManager
+from lp.services.mail.sendmail import set_immediate_mail_delivery
+from lp.services.twistedsupport.loggingsupport import RotatableFileLogObserver
execute_zcml_for_scripts()
ZopelessTransactionManager.initZopeless(dbuser='buildd_manager')
+# XXX wgrant 2011-09-24 bug=29744: initZopeless used to do this.
+# Should be removed from callsites verified to not need it.
+set_immediate_mail_delivery(True)
options = ServerOptions()
options.parseOptions()
=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf 2011-09-21 14:38:47 +0000
+++ lib/canonical/config/schema-lazr.conf 2011-09-26 00:35:27 +0000
@@ -2037,10 +2037,10 @@
root: /var/tmp/testkeyserver
-# Configuration specific for code that is running in the Zopeless
-# environment. Hopefully this section will disappear when the
-# Zope and Zopeless environments grow closer.
-[zopeless]
+# Immediate mail delivery configuration, for scripts that bypass Zope
+# queued mail delivery. Will disappear once scripts are updated to use
+# normal queued delivery.
+[immediate_mail]
# datatype: port_number
smtp_port: 25
=== modified file 'lib/canonical/testing/layers.py'
--- lib/canonical/testing/layers.py 2011-09-22 19:59:40 +0000
+++ lib/canonical/testing/layers.py 2011-09-26 00:35:27 +0000
@@ -140,6 +140,7 @@
IMailBox,
TestMailBox,
)
+from lp.services.mail.sendmail import set_immediate_mail_delivery
import lp.services.mail.stub
from lp.services.memcache.client import memcache_client_factory
from lp.services.osutils import kill_by_pidfile
@@ -1535,6 +1536,10 @@
raise LayerIsolationError(
"Last test using Zopeless failed to tearDown correctly")
ZopelessTransactionManager.initZopeless(dbuser='launchpad_main')
+ # XXX wgrant 2011-09-24 bug=29744: initZopeless used to do this.
+ # Tests that still need it should eventually set this directly,
+ # so the whole layer is not polluted.
+ set_immediate_mail_delivery(True)
# Connect Storm
reconnect_stores()
@@ -1548,6 +1553,11 @@
"Failed to uninstall ZopelessTransactionManager")
# LaunchpadScriptLayer will disconnect the stores for us.
+ # XXX wgrant 2011-09-24 bug=29744: uninstall used to do this.
+ # Tests that still need immediate delivery should eventually do
+ # this directly.
+ set_immediate_mail_delivery(False)
+
@classmethod
@profiled
def commit(cls):
=== modified file 'lib/lp/services/job/runner.py'
--- lib/lp/services/job/runner.py 2011-09-18 07:15:22 +0000
+++ lib/lp/services/job/runner.py 2011-09-26 00:35:27 +0000
@@ -64,7 +64,10 @@
LeaseHeld,
SuspendJobException,
)
-from lp.services.mail.sendmail import MailController
+from lp.services.mail.sendmail import (
+ MailController,
+ set_immediate_mail_delivery,
+ )
from lp.services.scripts.base import LaunchpadCronScript
from lp.services.twistedsupport import run_reactor
@@ -370,6 +373,9 @@
scripts.execute_zcml_for_scripts(use_web_security=False)
signal(SIGHUP, handler)
ZopelessTransactionManager.initZopeless(dbuser=cls.dbuser)
+ # XXX wgrant 2011-09-24 bug=29744: initZopeless used to do this.
+ # Should be removed from callsites verified to not need it.
+ set_immediate_mail_delivery(True)
@staticmethod
def __exit__(exc_type, exc_val, exc_tb):
=== modified file 'lib/lp/services/mail/sendmail.py'
--- lib/lp/services/mail/sendmail.py 2011-09-18 07:22:34 +0000
+++ lib/lp/services/mail/sendmail.py 2011-09-26 00:35:27 +0000
@@ -21,6 +21,7 @@
'get_msgid',
'MailController',
'sendmail',
+ 'set_immediate_mail_delivery',
'simple_sendmail',
'simple_sendmail_from_person',
'validate_message',
@@ -51,7 +52,6 @@
from zope.sendmail.interfaces import IMailDelivery
from canonical.config import config
-from canonical.database.sqlbase import ZopelessTransactionManager
from lp.app import versioninfo
from lp.services.encoding import is_ascii_only
from lp.services.mail.stub import TestMailer
@@ -64,6 +64,7 @@
Charset.add_charset('utf-8', Charset.SHORTEST, Charset.QP, 'utf-8')
Charset.add_alias('utf8', 'utf-8')
+
def do_paranoid_email_content_validation(from_addr, to_addrs, subject, body):
"""Validate various bits of the email.
@@ -161,6 +162,20 @@
return format_address(person.displayname, email_address)
+_immediate_mail_delivery = False
+
+
+def set_immediate_mail_delivery(enabled):
+ """Enable or disable immediate mail delivery.
+
+ Mail is by default queued until the transaction is committed. But if
+ a script requires that mail violate transactions, immediate mail
+ delivery can be enabled.
+ """
+ global _immediate_mail_delivery
+ _immediate_mail_delivery = enabled
+
+
def simple_sendmail(from_addr, to_addrs, subject, body, headers=None,
bulk=True):
"""Send an email from from_addr to to_addrs with the subject and body
@@ -422,11 +437,11 @@
raw_message = message.as_string()
message_detail = message['Subject']
- if ZopelessTransactionManager._installed is not None:
- # Zopeless email sending is not unit tested, and won't be.
- # The zopeless specific stuff is pretty simple though so this
+ if _immediate_mail_delivery:
+ # Immediate email delivery is not unit tested, and won't be.
+ # The immediate-specific stuff is pretty simple though so this
# should be fine.
- # TODO: Store a timeline action for zopeless mail.
+ # TODO: Store a timeline action for immediate mail.
if config.isTestRunner():
# when running in the testing environment, store emails
@@ -436,12 +451,13 @@
# For debugging, from process-one-mail, just print it.
sys.stdout.write(raw_message)
else:
- if config.zopeless.send_email:
+ if config.immediate_mail.send_email:
# Note that we simply throw away dud recipients. This is fine,
# as it emulates the Z3 API which doesn't report this either
# (because actual delivery is done later).
smtp = SMTP(
- config.zopeless.smtp_host, config.zopeless.smtp_port)
+ config.immediate_mail.smtp_host,
+ config.immediate_mail.smtp_port)
# The "MAIL FROM" is set to the bounce address, to behave in a
# way similar to mailing list software.
=== modified file 'lib/lp/services/scripts/base.py'
--- lib/lp/services/scripts/base.py 2011-09-18 08:28:20 +0000
+++ lib/lp/services/scripts/base.py 2011-09-26 00:35:27 +0000
@@ -51,6 +51,7 @@
install_feature_controller,
make_script_feature_controller,
)
+from lp.services.mail.sendmail import set_immediate_mail_delivery
from lp.services.scripts.interfaces.scriptactivity import IScriptActivitySet
@@ -322,6 +323,10 @@
self._init_zca(use_web_security=use_web_security)
self._init_db(isolation=isolation)
+ # XXX wgrant 2011-09-24 bug=29744: initZopeless used to do this.
+ # Should be called directly by scripts that actually need it.
+ set_immediate_mail_delivery(True)
+
date_started = datetime.datetime.now(UTC)
profiler = None
if self.options.profile:
=== modified file 'scripts/bug-import.py'
--- scripts/bug-import.py 2010-12-16 14:53:36 +0000
+++ scripts/bug-import.py 2011-09-26 00:35:27 +0000
@@ -48,7 +48,7 @@
# don't send email
send_email_data = """
- [zopeless]
+ [immediate_mail]
send_email: False
"""
config.push('send_email_data', send_email_data)
=== modified file 'scripts/mlist-import.py'
--- scripts/mlist-import.py 2010-04-27 19:48:39 +0000
+++ scripts/mlist-import.py 2011-09-26 00:35:27 +0000
@@ -71,7 +71,7 @@
# switch. Notifications are disabled by default because they can
# cause huge amounts to be sent to the team owner.
send_email_config = """
- [zopeless]
+ [immediate_mail]
send_email: %s
""" % self.options.notifications
config.push('send_email_config', send_email_config)
