launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #08435
[Merge] lp:~sinzui/launchpad/queue-project-email into lp:launchpad
Curtis Hovey has proposed merging lp:~sinzui/launchpad/queue-project-email into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #721438 in Launchpad itself: "Automatically email projects with license/content issues"
https://bugs.launchpad.net/launchpad/+bug/721438
For more details, see:
https://code.launchpad.net/~sinzui/launchpad/queue-project-email/+merge/108235
Pre-implementation: abentley, wgrant
Launchpad needs a daily process that queues the jobs to send emails
about expiring commercial licenses.
--------------------------------------------------------------------
RULES
* Using daily-builds as a guide
* Create a cronscript that gets each kind of product job source
* It calls each job source to make the needed jobs.
* Maybe the job sources need to be called in a specific order.
QA
* Verify database/schema/security.py --no-revoke was run to
add the commercialsubscription table to the product-job user.
* Ask a WebOps to run ./cronscripts/daily_product_jobs.py -vvv
* Verify that the script reports that 14 or more jobs were created.
* Verify that a ThirtyDay... job were created for numerink.
* Verify that an Expired... job were created for sausalito.
* Verify that sausalito is deactivated.
LINT
cronscripts/daily_product_jobs.py
database/schema/security.cfg
lib/lp/registry/interfaces/productjob.py
lib/lp/registry/model/productjob.py
lib/lp/registry/tests/test_productjob.py
lib/lp/services/config/schema-lazr.conf
TEST
./bin/test -vvc lp.registry.tests.test_productjob
IMPLEMENTATION
Created a cronscript that calls ProductJobManager.createAllDailyJobs().
I discovered that I neglected to add the product-job user to the config.
The queries to find expiring products needed access to the
commercialsubscription which I added via security.cfg.
cronscripts/daily_product_jobs.py
database/schema/security.cfg
lib/lp/services/config/schema-lazr.conf
I extracted the create() method from the three commercial job sources to
a base interface and added getExpiringProducts(). The ProductJobManager
requires that all the commercial jobs to have the same contracts.
lib/lp/registry/interfaces/productjob.py
I added the ProductJobManager to orchestra job creation. This will grow
to handle "other/open source" and "other/I don't know" cases in the
future. ProductJobManager does the primary work for the cronscript. I
added getExpiringProducts() to the mixin class used by the commercial
jobs, then updated each class to provide _get_expiration_dates() that
returns the set of dates that constrain the search. In the tests I add
CommercialHelpers which is used by old and new tests to ensure the job
classes, the manager, and the cronscript has a common setup to ensure
they operate under the same conditions.
lib/lp/registry/model/productjob.py
lib/lp/registry/tests/test_productjob.py
--
https://code.launchpad.net/~sinzui/launchpad/queue-project-email/+merge/108235
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~sinzui/launchpad/queue-project-email into lp:launchpad.
=== added file 'cronscripts/daily_product_jobs.py'
--- cronscripts/daily_product_jobs.py 1970-01-01 00:00:00 +0000
+++ cronscripts/daily_product_jobs.py 2012-05-31 20:24:32 +0000
@@ -0,0 +1,37 @@
+#!/usr/bin/python -S
+#
+# Copyright 2012 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+"""Request jobs to update products and send emails."""
+
+__metaclass__ = type
+
+import _pythonpath
+
+import transaction
+
+from lp.registry.model.productjob import ProductJobManager
+from lp.services.config import config
+from lp.services.scripts.base import LaunchpadCronScript
+from lp.services.webapp.errorlog import globalErrorUtility
+
+
+class RequestProductJobs(LaunchpadCronScript):
+ """Create `ProductJobs` for products that need updating."""
+
+ def __init__(self):
+ name = 'daily_product_jobs'
+ dbuser = config.product_job.dbuser
+ LaunchpadCronScript.__init__(self, name, dbuser)
+
+ def main(self):
+ globalErrorUtility.configure(self.name)
+ manager = ProductJobManager(self.logger)
+ job_count = manager.createAllDailyJobs()
+ self.logger.info('Requested %d total product jobs.' % job_count)
+ transaction.commit()
+
+
+if __name__ == '__main__':
+ script = RequestProductJobs()
+ script.lock_and_run()
=== modified file 'database/schema/security.cfg'
--- database/schema/security.cfg 2012-05-31 01:57:07 +0000
+++ database/schema/security.cfg 2012-05-31 20:24:32 +0000
@@ -2327,6 +2327,7 @@
[product-job]
groups=script
public.account = SELECT
+public.commercialsubscription = SELECT
public.emailaddress = SELECT
public.job = SELECT, INSERT, UPDATE
public.person = SELECT
=== modified file 'lib/lp/registry/interfaces/productjob.py'
--- lib/lp/registry/interfaces/productjob.py 2012-05-24 20:25:54 +0000
+++ lib/lp/registry/interfaces/productjob.py 2012-05-31 20:24:32 +0000
@@ -17,7 +17,10 @@
'IThirtyDayCommercialExpirationJobSource',
]
-from zope.interface import Attribute
+from zope.interface import (
+ Attribute,
+ Interface,
+ )
from zope.schema import (
Int,
Object,
@@ -127,35 +130,43 @@
"""
+class ExpirationSource(Interface):
+
+ def create(product, reviewer):
+ """Create a new job.
+
+ :param product: An IProduct.
+ :param reviewer: The user or agent sending the email.
+ """
+
+ def getExpiringProducts():
+ """Return the products that require a job to update them.
+
+ The products returned can passed to create() to make the job.
+ The products have a commercial subscription that expires within
+ the job's effective date range. The products returned do not
+ have a recent job; once the job is created, the product is
+ excluded from the future calls to this method.
+ """
+
+
class ISevenDayCommercialExpirationJob(IProductNotificationJob):
"""A job that sends an email about an expiring commercial subscription."""
-class ISevenDayCommercialExpirationJobSource(IProductNotificationJobSource):
+class ISevenDayCommercialExpirationJobSource(IProductNotificationJobSource,
+ ExpirationSource):
"""An interface for creating `ISevenDayCommercialExpirationJob`s."""
- def create(product, reviewer):
- """Create a new `ISevenDayCommercialExpirationJob`.
-
- :param product: An IProduct.
- :param reviewer: The user or agent sending the email.
- """
-
class IThirtyDayCommercialExpirationJob(IProductNotificationJob):
"""A job that sends an email about an expiring commercial subscription."""
-class IThirtyDayCommercialExpirationJobSource(IProductNotificationJobSource):
+class IThirtyDayCommercialExpirationJobSource(IProductNotificationJobSource,
+ ExpirationSource):
"""An interface for creating `IThirtyDayCommercialExpirationJob`s."""
- def create(product, reviewer):
- """Create a new `IThirtyDayCommercialExpirationJob`.
-
- :param product: An IProduct.
- :param reviewer: The user or agent sending the email.
- """
-
class ICommercialExpiredJob(IProductNotificationJob):
"""A job that sends an email about an expired commercial subscription.
@@ -166,12 +177,6 @@
"""
-class ICommercialExpiredJobSource(IProductNotificationJobSource):
+class ICommercialExpiredJobSource(IProductNotificationJobSource,
+ ExpirationSource):
"""An interface for creating `IThirtyDayCommercialExpirationJob`s."""
-
- def create(product, reviewer):
- """Create a new `ICommercialExpiredJob`.
-
- :param product: An IProduct.
- :param reviewer: The user or agent sending the email.
- """
=== modified file 'lib/lp/registry/model/productjob.py'
--- lib/lp/registry/model/productjob.py 2012-05-24 21:26:57 +0000
+++ lib/lp/registry/model/productjob.py 2012-05-31 20:24:32 +0000
@@ -6,14 +6,24 @@
__metaclass__ = type
__all__ = [
'ProductJob',
+ 'ProductJobManager',
'CommercialExpiredJob',
'SevenDayCommercialExpirationJob',
'ThirtyDayCommercialExpirationJob',
]
+from datetime import (
+ datetime,
+ timedelta,
+ )
+from pytz import utc
from lazr.delegates import delegates
import simplejson
-from storm.expr import And
+from storm.expr import (
+ And,
+ Not,
+ Select,
+ )
from storm.locals import (
Int,
Reference,
@@ -26,6 +36,7 @@
)
from zope.security.proxy import removeSecurityProxy
+from lp.app.interfaces.launchpad import ILaunchpadCelebrities
from lp.registry.enums import ProductJobType
from lp.registry.interfaces.person import IPersonSet
from lp.registry.interfaces.product import (
@@ -44,6 +55,7 @@
IThirtyDayCommercialExpirationJob,
IThirtyDayCommercialExpirationJobSource,
)
+from lp.registry.model.commercialsubscription import CommercialSubscription
from lp.registry.model.product import Product
from lp.services.config import config
from lp.services.database.decoratedresultset import DecoratedResultSet
@@ -69,6 +81,43 @@
from lp.services.webapp.publisher import canonical_url
+class ProductJobManager:
+ """Creates jobs for product that need updating or notification."""
+
+ def __init__(self, logger):
+ self.logger = logger
+
+ def createAllDailyJobs(self):
+ """Create jobs for all products that have timed updates.
+
+ :return: The count of jobs that were created.
+ """
+ reviewer = getUtility(ILaunchpadCelebrities).janitor
+ total = 0
+ total += self.createDailyJobs(CommercialExpiredJob, reviewer)
+ total += self.createDailyJobs(
+ SevenDayCommercialExpirationJob, reviewer)
+ total += self.createDailyJobs(
+ ThirtyDayCommercialExpirationJob, reviewer)
+ return total
+
+ def createDailyJobs(self, job_class, reviewer):
+ """Create jobs for products that have timed updates.
+
+ :param job_class: A JobSource class that provides `ExpirationSource`.
+ :param reviewer: The user that is creating the job.
+ :return: The count of jobs that were created.
+ """
+ total = 0
+ for product in job_class.getExpiringProducts():
+ self.logger.debug(
+ 'Creating a %s for %s' %
+ (job_class.__class__.__name__, product.name))
+ job_class.create(product, reviewer)
+ total += 1
+ return total
+
+
class ProductJob(StormBase):
"""Base class for product jobs."""
@@ -305,12 +354,32 @@
@classmethod
def create(cls, product, reviewer):
- """Create a job."""
+ """See `ExpirationSourceMixin`."""
subject = cls._subject_template % product.name
return super(CommericialExpirationMixin, cls).create(
product, cls._email_template_name, subject, reviewer,
reply_to_commercial=True)
+ @classmethod
+ def getExpiringProducts(cls):
+ """See `ExpirationSourceMixin`."""
+ earliest_date, latest_date, past_date = cls._get_expiration_dates()
+ recent_jobs = And(
+ ProductJob.job_type == cls.class_job_type,
+ ProductJob.job_id == Job.id,
+ Job.date_created > past_date,
+ )
+ conditions = [
+ Product.active == True,
+ CommercialSubscription.productID == Product.id,
+ CommercialSubscription.date_expires >= earliest_date,
+ CommercialSubscription.date_expires < latest_date,
+ Not(Product.id.is_in(Select(
+ ProductJob.product_id,
+ tables=[ProductJob, Job], where=recent_jobs))),
+ ]
+ return IStore(Product).find(Product, *conditions)
+
@cachedproperty
def message_data(self):
"""See `IProductNotificationJob`."""
@@ -332,6 +401,13 @@
classProvides(ISevenDayCommercialExpirationJobSource)
class_job_type = ProductJobType.COMMERCIAL_EXPIRATION_7_DAYS
+ @staticmethod
+ def _get_expiration_dates():
+ now = datetime.now(utc)
+ in_seven_days = now + timedelta(days=7)
+ seven_days_ago = now - timedelta(days=7)
+ return now, in_seven_days, seven_days_ago
+
class ThirtyDayCommercialExpirationJob(CommericialExpirationMixin,
ProductNotificationJob):
@@ -341,6 +417,15 @@
classProvides(IThirtyDayCommercialExpirationJobSource)
class_job_type = ProductJobType.COMMERCIAL_EXPIRATION_30_DAYS
+ @staticmethod
+ def _get_expiration_dates():
+ now = datetime.now(utc)
+ # Avoid overlay with the seven day notification.
+ in_twenty_three_days = now + timedelta(days=7)
+ in_thirty_days = now + timedelta(days=30)
+ thirty_days_ago = now - timedelta(days=30)
+ return in_twenty_three_days, in_thirty_days, thirty_days_ago
+
class CommercialExpiredJob(CommericialExpirationMixin, ProductNotificationJob):
"""A job that sends an email about an expired commercial subscription."""
@@ -353,6 +438,13 @@
_subject_template = (
'The commercial subscription for %s in Launchpad expired')
+ @staticmethod
+ def _get_expiration_dates():
+ now = datetime.now(utc)
+ ten_years_ago = now - timedelta(days=3650)
+ thirty_days_ago = now - timedelta(days=30)
+ return ten_years_ago, now, thirty_days_ago
+
@property
def _is_proprietary(self):
"""Does the product have a proprietary licence?"""
=== modified file 'lib/lp/registry/tests/test_productjob.py'
--- lib/lp/registry/tests/test_productjob.py 2012-05-24 20:25:54 +0000
+++ lib/lp/registry/tests/test_productjob.py 2012-05-31 20:24:32 +0000
@@ -9,7 +9,7 @@
datetime,
timedelta,
)
-
+import transaction
import pytz
from zope.component import getUtility
from zope.interface import (
@@ -39,6 +39,7 @@
from lp.registry.model.productjob import (
ProductJob,
ProductJobDerived,
+ ProductJobManager,
ProductNotificationJob,
CommercialExpiredJob,
SevenDayCommercialExpirationJob,
@@ -51,11 +52,96 @@
from lp.testing.layers import (
DatabaseFunctionalLayer,
LaunchpadZopelessLayer,
+ ZopelessAppServerLayer,
)
from lp.testing.mail_helpers import pop_notifications
+from lp.services.log.logger import BufferLogger
+from lp.services.scripts.tests import run_script
from lp.services.webapp.publisher import canonical_url
+class CommercialHelpers:
+
+ def make_expiring_product(self, date_expires, job_class=None):
+ product = self.factory.makeProduct(
+ licenses=[License.OTHER_PROPRIETARY])
+ removeSecurityProxy(
+ product.commercial_subscription).date_expires = date_expires
+ if job_class:
+ reviewer = getUtility(ILaunchpadCelebrities).janitor
+ job_class.create(product, reviewer)
+ return product
+
+ def make_test_products(self):
+ products = {}
+ now = datetime.now(pytz.utc)
+ products['approved'] = self.factory.makeProduct(licenses=[License.MIT])
+ products['expired'] = self.make_expiring_product(now - timedelta(1))
+ products['expired_with_job'] = self.make_expiring_product(
+ now - timedelta(1), job_class=CommercialExpiredJob)
+ products['seven'] = self.make_expiring_product(now + timedelta(6))
+ products['seven_with_job'] = self.make_expiring_product(
+ now + timedelta(6), job_class=SevenDayCommercialExpirationJob)
+ products['thirty'] = self.make_expiring_product(now + timedelta(29))
+ products['thirty_with_job'] = self.make_expiring_product(
+ now + timedelta(29), job_class=ThirtyDayCommercialExpirationJob)
+ products['sixty'] = self.make_expiring_product(now + timedelta(60))
+ return products
+
+
+class ProductJobManagerTestCase(TestCaseWithFactory, CommercialHelpers):
+ """Test case for the ProductJobManager class."""
+ layer = DatabaseFunctionalLayer
+
+ @staticmethod
+ def make_manager():
+ logger = BufferLogger()
+ return ProductJobManager(logger)
+
+ def test_init(self):
+ # The logger was set.
+ manager = self.make_manager()
+ self.assertIsInstance(manager.logger, BufferLogger)
+
+ def test_createAllDailyJobs(self):
+ # 3 kinds of commercial expiration jobs are created.
+ self.make_test_products()
+ manager = self.make_manager()
+ self.assertEqual(3, manager.createAllDailyJobs())
+ log = manager.logger.getLogBuffer()
+ self.assertIn(CommercialExpiredJob.__class__.__name__, log)
+ self.assertIn(SevenDayCommercialExpirationJob.__class__.__name__, log)
+ self.assertIn(ThirtyDayCommercialExpirationJob.__class__.__name__, log)
+
+ def test_createDailyJobs(self):
+ # Commercial expiration jobs are created.
+ test_products = self.make_test_products()
+ manager = self.make_manager()
+ reviewer = self.factory.makePerson()
+ total = manager.createDailyJobs(CommercialExpiredJob, reviewer)
+ self.assertEqual(1, total)
+ self.assertIn(
+ 'DEBUG Creating a %s for %s' %
+ (CommercialExpiredJob.__class__.__name__,
+ test_products['expired'].name),
+ manager.logger.getLogBuffer())
+
+
+class DailyProductJobsTestCase(TestCaseWithFactory, CommercialHelpers):
+ """Test case for the ProductJobManager class."""
+ layer = ZopelessAppServerLayer
+
+ def test_run(self):
+ # The script called ProductJobManager.createAllDailyJobs().
+ # This test uses the same setup as
+ # ProductJobManagerTestCase.test_createAllDailyJobs
+ self.make_test_products()
+ transaction.commit()
+ retcode, stdout, stderr = run_script(
+ 'cronscripts/daily_product_jobs.py', [])
+ self.assertIn('Requested 3 total product jobs.', stderr)
+
+
class ProductJobTestCase(TestCaseWithFactory):
"""Test case for basic ProductJob class."""
@@ -388,11 +474,12 @@
'Launchpad <noreply@xxxxxxxxxxxxx>', notifications[0]['From'])
-class CommericialExpirationMixin:
+class CommericialExpirationMixin(CommercialHelpers):
layer = DatabaseFunctionalLayer
EXPIRE_SUBSCRIPTION = False
+ EXPECTED_PRODUCT = None
def make_notification_data(self, licenses=[License.MIT]):
product = self.factory.makeProduct(licenses=licenses)
@@ -402,6 +489,14 @@
reviewer = getUtility(ILaunchpadCelebrities).janitor
return product, reviewer
+ def test_getExpiringProducts(self):
+ # Get the products with an expiring commercial subscription in
+ # the job type's date range that do not already have a recent job.
+ test_products = self.make_test_products()
+ products = list(self.JOB_CLASS.getExpiringProducts())
+ self.assertEqual(1, len(products))
+ self.assertEqual(test_products[self.EXPECTED_PRODUCT], products[0])
+
def test_create(self):
# Create an instance of an commercial expiration job that stores
# the notification information.
@@ -459,32 +554,53 @@
TestCaseWithFactory):
"""Test case for the SevenDayCommercialExpirationJob class."""
+ EXPECTED_PRODUCT = 'seven'
JOB_INTERFACE = ISevenDayCommercialExpirationJob
JOB_SOURCE_INTERFACE = ISevenDayCommercialExpirationJobSource
JOB_CLASS = SevenDayCommercialExpirationJob
JOB_CLASS_TYPE = ProductJobType.COMMERCIAL_EXPIRATION_7_DAYS
+ def test_get_expiration_dates(self):
+ dates = SevenDayCommercialExpirationJob._get_expiration_dates()
+ earliest_date, latest_date, past_date = dates
+ self.assertEqual(timedelta(days=7), latest_date - earliest_date)
+ self.assertEqual(timedelta(days=14), latest_date - past_date)
+
class ThirtyDayCommercialExpirationJobTestCase(CommericialExpirationMixin,
TestCaseWithFactory):
"""Test case for the SevenDayCommercialExpirationJob class."""
+ EXPECTED_PRODUCT = 'thirty'
JOB_INTERFACE = IThirtyDayCommercialExpirationJob
JOB_SOURCE_INTERFACE = IThirtyDayCommercialExpirationJobSource
JOB_CLASS = ThirtyDayCommercialExpirationJob
JOB_CLASS_TYPE = ProductJobType.COMMERCIAL_EXPIRATION_30_DAYS
+ def test_get_expiration_dates(self):
+ dates = ThirtyDayCommercialExpirationJob._get_expiration_dates()
+ earliest_date, latest_date, past_date = dates
+ self.assertEqual(timedelta(days=23), latest_date - earliest_date)
+ self.assertEqual(timedelta(days=60), latest_date - past_date)
+
class CommercialExpiredJobTestCase(CommericialExpirationMixin,
TestCaseWithFactory):
"""Test case for the CommercialExpiredJob class."""
EXPIRE_SUBSCRIPTION = True
+ EXPECTED_PRODUCT = 'expired'
JOB_INTERFACE = ICommercialExpiredJob
JOB_SOURCE_INTERFACE = ICommercialExpiredJobSource
JOB_CLASS = CommercialExpiredJob
JOB_CLASS_TYPE = ProductJobType.COMMERCIAL_EXPIRED
+ def test_get_expiration_dates(self):
+ dates = CommercialExpiredJob._get_expiration_dates()
+ earliest_date, latest_date, past_date = dates
+ self.assertEqual(timedelta(days=3650), latest_date - earliest_date)
+ self.assertEqual(timedelta(days=30), latest_date - past_date)
+
def test_is_proprietary_open_source(self):
product, reviewer = self.make_notification_data(licenses=[License.MIT])
job = CommercialExpiredJob.create(product, reviewer)
=== modified file 'lib/lp/services/config/schema-lazr.conf'
--- lib/lp/services/config/schema-lazr.conf 2012-05-23 17:27:49 +0000
+++ lib/lp/services/config/schema-lazr.conf 2012-05-31 20:24:32 +0000
@@ -1472,6 +1472,12 @@
dbuser: productreleasefinder
+[product_job]
+# The database user which will be used by this process.
+# datatype: string
+dbuser: product-job
+
+
[profiling]
# When set to True, the user is allowed to request profiles be run for
# either individual pages or for every request sent.
Follow ups
