launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #04847
[Merge] lp:~henninge/launchpad/abel-broken-hwdb-reports into lp:launchpad
Henning Eggers has proposed merging lp:~henninge/launchpad/abel-broken-hwdb-reports into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #838495 in Launchpad itself: "hwdb is breaking rapidly and generating 600+Kb oopses"
https://bugs.launchpad.net/launchpad/+bug/838495
For more details, see:
https://code.launchpad.net/~henninge/launchpad/abel-broken-hwdb-reports/+merge/73697
= Summary =
Create a new script which is mostly a copy of process-hwdb-submissions
but processes "INVALID" submissions to see if they are still invalid.
Also adds a parameter to process-hwdb-submissions to suppress warnings
(see bug 838495)
== Proposed fix ==
Factor out the selection of submissions for processing into its own
method. For the new script make it return INVALID entries instead of
SUBMITTED.
Add a "start" parameter to be able to pick up work where it was left.
process-hwdb-submissions did not have to worry about this because it
always changed the status away from SUBMITTED so it would never be
led to process a submission twice.
Add the --warning command line option and pass it through to
_logWarning.
== Tests ==
I added some tests to verify that the differntiated submission selection
work. For this I had to update the factory to allow the setting of the
status of a submission
bin/test -vvcm lp.hardwaredb.scripts.tests.test_hwdbsubmissions
bin/text -vvcm canonical.launchpad.scripts.tests.test_hwdb_submission_processing
= Launchpad lint =
Checking for conflicts and issues in changed files.
Linting changed files:
cronscripts/process-hwdb-submissions.py
lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py
lib/lp/hardwaredb/scripts/tests/__init__.py
lib/lp/hardwaredb/scripts/tests/test_hwdbsubmissions.py
lib/canonical/launchpad/scripts/tests/test_hwdb_submission_parser.py
lib/lp/testing/factory.py
scripts/reprocess-hwdb-submissions.py
lib/lp/hardwaredb/scripts/hwdbsubmissions.py
./cronscripts/process-hwdb-submissions.py
24: '_pythonpath' imported but unused
./lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py
485: local variable 'test' is assigned to but never used
527: local variable 'test' is assigned to but never used
640: local variable 'test' is assigned to but never used
678: local variable 'test' is assigned to but never used
1697: local variable 'UDI_SSB' is assigned to but never used
2098: local variable 'properties' is assigned to but never used
4097: local variable 'parent' is assigned to but never used
5106: local variable 'device_set' is assigned to but never used
162: E301 expected 1 blank line, found 0
179: E202 whitespace before '}'
189: E202 whitespace before '}'
251: E301 expected 1 blank line, found 0
255: E202 whitespace before '}'
579: E202 whitespace before ']'
614: E202 whitespace before ']'
655: E202 whitespace before ']'
929: E301 expected 1 blank line, found 2
1929: E202 whitespace before '}'
2012: E202 whitespace before '}'
2115: E202 whitespace before '}'
2143: E202 whitespace before ')'
2185: E202 whitespace before '}'
3167: E222 multiple spaces after operator
3180: E222 multiple spaces after operator
3322: E202 whitespace before '}'
3546: E202 whitespace before '}'
3731: E202 whitespace before '}'
4253: E202 whitespace before ')'
4370: E301 expected 1 blank line, found 2
4731: E202 whitespace before ']'
4986: E202 whitespace before ']'
5256: E301 expected 1 blank line, found 0
5259: E301 expected 1 blank line, found 0
5331: E301 expected 1 blank line, found 0
5444: E301 expected 1 blank line, found 0
5472: E301 expected 1 blank line, found 0
./lib/canonical/launchpad/scripts/tests/test_hwdb_submission_parser.py
12: redefinition of unused 'etree' from line 10
307: E202 whitespace before ']'
496: E301 expected 1 blank line, found 0
536: E301 expected 1 blank line, found 0
562: E301 expected 1 blank line, found 0
2260: E301 expected 1 blank line, found 2
2265: E301 expected 1 blank line, found 0
2298: E301 expected 1 blank line, found 0
2317: E301 expected 1 blank line, found 0
2336: E301 expected 1 blank line, found 0
2355: E301 expected 1 blank line, found 0
2374: E301 expected 1 blank line, found 0
2482: E202 whitespace before ')'
2487: E301 expected 1 blank line, found 0
2505: E202 whitespace before ')'
2510: E301 expected 1 blank line, found 0
2530: E301 expected 1 blank line, found 0
2533: E301 expected 1 blank line, found 0
2549: E301 expected 1 blank line, found 0
2571: E301 expected 1 blank line, found 0
2592: E301 expected 1 blank line, found 0
./scripts/reprocess-hwdb-submissions.py
24: '_pythonpath' imported but unused
./lib/lp/hardwaredb/scripts/hwdbsubmissions.py
26: redefinition of unused 'etree' from line 24
372: local variable 'property_name' is assigned to but never used
2789: local variable 'path' is assigned to but never used
2801: local variable 'path' is assigned to but never used
3036: local variable 'error' is assigned to but never used
125: E302 expected 2 blank lines, found 1
971: E221 multiple spaces before operator
2662: E202 whitespace before '}'
3060: E202 whitespace before ')'
3106: E302 expected 2 blank lines, found 1
--
https://code.launchpad.net/~henninge/launchpad/abel-broken-hwdb-reports/+merge/73697
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~henninge/launchpad/abel-broken-hwdb-reports into lp:launchpad.
=== modified file 'cronscripts/process-hwdb-submissions.py'
--- cronscripts/process-hwdb-submissions.py 2010-04-27 19:48:39 +0000
+++ cronscripts/process-hwdb-submissions.py 2011-09-01 16:59:33 +0000
@@ -35,6 +35,9 @@
self.parser.add_option(
'-m', '--max-submissions',
help='Limit the number of submissions which will be processed.')
+ self.parser.add_option(
+ '-w', '--warnings', action="store_true", default=False,
+ help='Include warnings.')
def main(self):
max_submissions = self.options.max_submissions
@@ -51,7 +54,8 @@
'--max_submissions must be a positive integer.')
return
- process_pending_submissions(self.txn, self.logger, max_submissions)
+ process_pending_submissions(
+ self.txn, self.logger, max_submissions, self.options.warnings)
if __name__ == '__main__':
script = HWDBSubmissionProcessor(
=== modified file 'lib/canonical/launchpad/scripts/tests/test_hwdb_submission_parser.py'
--- lib/canonical/launchpad/scripts/tests/test_hwdb_submission_parser.py 2011-08-29 20:11:27 +0000
+++ lib/canonical/launchpad/scripts/tests/test_hwdb_submission_parser.py 2011-09-01 16:59:33 +0000
@@ -925,7 +925,7 @@
All "method substitutes" return a valid result.
"""
- def __init__(self, logger=None):
+ def __init__(self, logger=None, record_warnings=True):
super(self.__class__, self).__init__(logger)
self.hal_result = 'parsed HAL data'
self.processors_result = 'parsed processor data'
@@ -1346,7 +1346,7 @@
All "method substitutes" return a valid result.
"""
- def __init__(self, logger=None):
+ def __init__(self, logger=None, record_warnings=True):
SubmissionParser.__init__(self, logger)
self.summary_result = 'parsed summary'
self.hardware_result = 'parsed hardware'
=== modified file 'lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py'
--- lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py 2011-08-29 20:11:27 +0000
+++ lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py 2011-09-01 16:59:33 +0000
@@ -4444,6 +4444,24 @@
'provide bus, vendor ID, product ID or product name: None None '
'None None /devices/pci0000:00/0000:00:1d.7/usb1/1-1/1-1:1.0')
+ def test_warnings_not_suppressed(self):
+ """Logging of warnings can be allowed."""
+ parser = SubmissionParser(self.log)
+ parser.submission_key = "log_with_warnings"
+ parser._logWarning("This message is logged.")
+ self.assertWarningMessage(
+ parser.submission_key, "This message is logged.")
+
+ def test_warnings_suppressed(self):
+ """Logging of warnings can be suppressed."""
+ number_of_existing_log_messages = len(self.handler.records)
+ parser = SubmissionParser(self.log, record_warnings=False)
+ parser.submission_key = "log_without_warnings"
+ parser._logWarning("This message is not logged.")
+ # No new warnings are recorded
+ self.assertEqual(
+ number_of_existing_log_messages, len(self.handler.records))
+
def test_device_id(self):
"""Each UdevDevice has a property 'id'."""
device = UdevDevice(None, self.root_device)
=== modified file 'lib/lp/hardwaredb/scripts/hwdbsubmissions.py'
--- lib/lp/hardwaredb/scripts/hwdbsubmissions.py 2011-08-31 16:38:48 +0000
+++ lib/lp/hardwaredb/scripts/hwdbsubmissions.py 2011-09-01 16:59:33 +0000
@@ -11,6 +11,8 @@
__all__ = [
'SubmissionParser',
'process_pending_submissions',
+ 'ProcessingLoopForPendingSubmissions',
+ 'ProcessingLoopForReprocessingBadSubmissions',
]
import bz2
@@ -32,6 +34,7 @@
from zope.component import getUtility
from zope.interface import implements
+from zope.security.proxy import removeSecurityProxy
from canonical.lazr.xml import RelaxNGValidator
@@ -49,6 +52,7 @@
IHWVendorIDSet,
IHWVendorNameSet,
)
+from lp.hardwaredb.model.hwdb import HWSubmission
from lp.app.interfaces.launchpad import ILaunchpadCelebrities
from canonical.launchpad.interfaces.looptuner import ITunableLoop
from canonical.launchpad.utilities.looptuner import LoopTuner
@@ -121,7 +125,7 @@
class SubmissionParser(object):
"""A Parser for the submissions to the hardware database."""
- def __init__(self, logger=None):
+ def __init__(self, logger=None, record_warnings=True):
if logger is None:
logger = getLogger()
self.logger = logger
@@ -137,6 +141,7 @@
self._setMainSectionParsers()
self._setHardwareSectionParsers()
self._setSoftwareSectionParsers()
+ self.record_warnings = record_warnings
def _logError(self, message, submission_key, create_oops=True):
"""Log `message` for an error in submission submission_key`."""
@@ -149,6 +154,8 @@
def _logWarning(self, message, warning_id=None):
"""Log `message` for a warning in submission submission_key`."""
+ if not self.record_warnings:
+ return
if warning_id is None:
issue_warning = True
elif warning_id not in self._logged_warnings:
@@ -2936,12 +2943,12 @@
return self.udev['id']
-class ProcessingLoop(object):
+class ProcessingLoopBase(object):
"""An `ITunableLoop` for processing HWDB submissions."""
implements(ITunableLoop)
- def __init__(self, transaction, logger, max_submissions):
+ def __init__(self, transaction, logger, max_submissions, record_warnings):
self.transaction = transaction
self.logger = logger
self.max_submissions = max_submissions
@@ -2949,6 +2956,7 @@
self.invalid_submissions = 0
self.finished = False
self.janitor = getUtility(ILaunchpadCelebrities).janitor
+ self.record_warnings = record_warnings
def _validateSubmission(self, submission):
submission.status = HWSubmissionProcessingStatus.PROCESSED
@@ -2971,16 +2979,16 @@
error_utility.raising(info, request)
self.logger.error('%s (%s)' % (error_explanation, request.oopsid))
+ def getUnprocessedSubmissions(self, chunk_size):
+ raise NotImplementedError
+
def __call__(self, chunk_size):
"""Process a batch of yet unprocessed HWDB submissions."""
# chunk_size is a float; we compare it below with an int value,
# which can lead to unexpected results. Since it is also used as
# a limit for an SQL query, convert it into an integer.
chunk_size = int(chunk_size)
- submissions = getUtility(IHWSubmissionSet).getByStatus(
- HWSubmissionProcessingStatus.SUBMITTED,
- user=self.janitor
- )[:chunk_size]
+ submissions = self.getUnprocessedSubmissions(chunk_size)
# Listify the submissions, since we'll have to loop over each
# one anyway. This saves a COUNT query for getting the number of
# submissions
@@ -2997,7 +3005,7 @@
# loop.
for submission in submissions:
try:
- parser = SubmissionParser(self.logger)
+ parser = SubmissionParser(self.logger, self.record_warnings)
success = parser.processSubmission(submission)
if success:
self._validateSubmission(submission)
@@ -3043,21 +3051,76 @@
break
self.transaction.commit()
-def process_pending_submissions(transaction, logger, max_submissions=None):
+
+class ProcessingLoopForPendingSubmissions(ProcessingLoopBase):
+
+ def getUnprocessedSubmissions(self, chunk_size):
+ submissions = getUtility(IHWSubmissionSet).getByStatus(
+ HWSubmissionProcessingStatus.SUBMITTED,
+ user=self.janitor
+ )[:chunk_size]
+ submissions = list(submissions)
+ return submissions
+
+
+class ProcessingLoopForReprocessingBadSubmissions(ProcessingLoopBase):
+
+ def __init__(self, start, transaction, logger,
+ max_submissions, record_warnings):
+ super(ProcessingLoopForReprocessingBadSubmissions, self).__init__(
+ transaction, logger, max_submissions, record_warnings)
+ self.start = start
+
+ def getUnprocessedSubmissions(self, chunk_size):
+ submissions = getUtility(IHWSubmissionSet).getByStatus(
+ HWSubmissionProcessingStatus.INVALID, user=self.janitor)
+ submissions = removeSecurityProxy(submissions).find(
+ HWSubmission.id >= self.start)
+ submissions = list(submissions[:chunk_size])
+ if len(submissions) > 0:
+ self.start = submissions[-1].id + 1
+ return submissions
+
+
+def process_pending_submissions(transaction, logger, max_submissions=None,
+ record_warnings=True):
"""Process pending submissions.
Parse pending submissions, store extracted data in HWDB tables and
mark them as either PROCESSED or INVALID.
"""
- loop = ProcessingLoop(transaction, logger, max_submissions)
- # It is hard to predict how long it will take to parse a submission.
- # we don't want to last a DB transaction too long but we also
- # don't want to commit more often than necessary. The LoopTuner
- # handles this for us. The loop's run time will be approximated to
- # 2 seconds, but will never handle more than 50 submissions.
- loop_tuner = LoopTuner(
- loop, 2, minimum_chunk_size=1, maximum_chunk_size=50)
- loop_tuner.run()
- logger.info(
- 'Processed %i valid and %i invalid HWDB submissions'
- % (loop.valid_submissions, loop.invalid_submissions))
+ loop = ProcessingLoopForPendingSubmissions(
+ transaction, logger, max_submissions, record_warnings)
+ # It is hard to predict how long it will take to parse a submission.
+ # we don't want to last a DB transaction too long but we also
+ # don't want to commit more often than necessary. The LoopTuner
+ # handles this for us. The loop's run time will be approximated to
+ # 2 seconds, but will never handle more than 50 submissions.
+ loop_tuner = LoopTuner(
+ loop, 2, minimum_chunk_size=1, maximum_chunk_size=50)
+ loop_tuner.run()
+ logger.info(
+ 'Processed %i valid and %i invalid HWDB submissions'
+ % (loop.valid_submissions, loop.invalid_submissions))
+
+def reprocess_invalid_submissions(start, transaction, logger,
+ max_submissions=None, record_warnings=True):
+ """Reprocess invalid submissions.
+
+ Parse submissions that have been marked as invalid. A newer
+ variant of the parser might be able to process them.
+ """
+ loop = ProcessingLoopForReprocessingBadSubmissions(
+ start, transaction, logger, max_submissions, record_warnings)
+ # It is hard to predict how long it will take to parse a submission.
+ # we don't want to last a DB transaction too long but we also
+ # don't want to commit more often than necessary. The LoopTuner
+ # handles this for us. The loop's run time will be approximated to
+ # 2 seconds, but will never handle more than 50 submissions.
+ loop_tuner = LoopTuner(
+ loop, 2, minimum_chunk_size=1, maximum_chunk_size=50)
+ loop_tuner.run()
+ logger.info(
+ 'Processed %i valid and %i invalid HWDB submissions'
+ % (loop.valid_submissions, loop.invalid_submissions))
+ logger.info('last processed: %i' % loop.start)
=== added directory 'lib/lp/hardwaredb/scripts/tests'
=== added file 'lib/lp/hardwaredb/scripts/tests/__init__.py'
=== added file 'lib/lp/hardwaredb/scripts/tests/test_hwdbsubmissions.py'
--- lib/lp/hardwaredb/scripts/tests/test_hwdbsubmissions.py 1970-01-01 00:00:00 +0000
+++ lib/lp/hardwaredb/scripts/tests/test_hwdbsubmissions.py 2011-09-01 16:59:33 +0000
@@ -0,0 +1,115 @@
+# Copyright 2011 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for hwdbsubmissions script."""
+
+__metaclass__ = type
+
+
+from canonical.testing.layers import LaunchpadScriptLayer
+from lp.hardwaredb.interfaces.hwdb import HWSubmissionProcessingStatus
+from lp.hardwaredb.scripts.hwdbsubmissions import (
+ ProcessingLoopForPendingSubmissions,
+ ProcessingLoopForReprocessingBadSubmissions,
+ )
+from lp.testing import TestCaseWithFactory
+
+
+class TestProcessingLoops(TestCaseWithFactory):
+ layer = LaunchpadScriptLayer
+
+ def _makePendingSubmissionsLoop(self):
+ """Parameters don't matter for these tests."""
+ return ProcessingLoopForPendingSubmissions(None, None, 0, False)
+
+ def test_PendingSubmissions_submitted_found(self):
+ # The PendingSubmissions loop finds submitted entries.
+ submission = self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.SUBMITTED)
+ loop = self._makePendingSubmissionsLoop()
+ # The sample data already contains one submission which we ignore.
+ submissions = loop.getUnprocessedSubmissions(2)
+ self.assertEqual([submission], submissions[1:])
+
+ def test_PendingSubmissions_processed_not_found(self):
+ # The PendingSubmissions loop ignores processed entries.
+ submission = self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.PROCESSED)
+ loop = self._makePendingSubmissionsLoop()
+ # The sample data already contains one submission which we ignore.
+ submissions = loop.getUnprocessedSubmissions(2)
+ self.assertEqual([], submissions[1:])
+ self.assertNotEqual([submission], submissions)
+
+ def test_PendingSubmissions_invalid_not_found(self):
+ # The PendingSubmissions loop ignores invalid entries.
+ submission = self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.INVALID)
+ loop = self._makePendingSubmissionsLoop()
+ # The sample data already contains one submission which we ignore.
+ submissions = loop.getUnprocessedSubmissions(2)
+ self.assertEqual([], submissions[1:])
+ self.assertNotEqual([submission], submissions)
+
+ def test_PendingSubmissions_respects_chunk_size(self):
+ # Only the requested number of entries are returned.
+ self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.SUBMITTED)
+ self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.SUBMITTED)
+ loop = self._makePendingSubmissionsLoop()
+ # The sample data already contains one submission.
+ submissions = loop.getUnprocessedSubmissions(2)
+ self.assertEqual(2, len(submissions))
+
+ def _makeBadSubmissionsLoop(self, start=0):
+ """Parameters don't matter for these tests."""
+ return ProcessingLoopForReprocessingBadSubmissions(
+ start, None, None, 0, False)
+
+ def test_BadSubmissions_invalid_found(self):
+ # The BadSubmissions loop finds invalid entries.
+ submission = self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.INVALID)
+ loop = self._makeBadSubmissionsLoop()
+ submissions = loop.getUnprocessedSubmissions(2)
+ self.assertEqual([submission], submissions)
+
+ def test_BadSubmissions_processed_not_found(self):
+ # The BadSubmissions loop ignores processed entries.
+ self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.PROCESSED)
+ loop = self._makeBadSubmissionsLoop()
+ submissions = loop.getUnprocessedSubmissions(2)
+ self.assertEqual([], submissions)
+
+ def test_BadSubmissions_submitted_not_found(self):
+ # The BadSubmissions loop ignores submitted entries.
+ self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.SUBMITTED)
+ loop = self._makeBadSubmissionsLoop()
+ submissions = loop.getUnprocessedSubmissions(2)
+ self.assertEqual([], submissions)
+
+ def test_BadSubmissions_respects_chunk_size(self):
+ # Only the requested number of entries are returned.
+ self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.INVALID)
+ self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.INVALID)
+ loop = self._makeBadSubmissionsLoop()
+ # The sample data already contains one submission.
+ submissions = loop.getUnprocessedSubmissions(1)
+ self.assertEqual(1, len(submissions))
+
+ def test_BadSubmissions_respects_start(self):
+ # It is possible to request a start id. Previous entries are ignored.
+ submission1 = self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.INVALID)
+ submission2 = self.factory.makeHWSubmission(
+ status=HWSubmissionProcessingStatus.INVALID)
+ self.assertTrue(submission1.id < submission2.id)
+ loop = self._makeBadSubmissionsLoop(submission2.id)
+ # The sample data already contains one submission.
+ submissions = loop.getUnprocessedSubmissions(2)
+ self.assertEqual([submission2], submissions)
=== modified file 'lib/lp/testing/factory.py'
--- lib/lp/testing/factory.py 2011-08-31 17:40:09 +0000
+++ lib/lp/testing/factory.py 2011-09-01 16:59:33 +0000
@@ -3690,8 +3690,7 @@
date_uploaded=UTC_NOW,
scheduleddeletiondate=None,
ancestor=None,
- **kwargs
- ):
+ **kwargs):
"""Make a `SourcePackagePublishingHistory`.
:param sourcepackagerelease: The source package release to publish
@@ -4095,7 +4094,7 @@
emailaddress=u'test@xxxxxxxxxxxxx',
distroarchseries=None, private=False,
contactable=False, system=None,
- submission_data=None):
+ submission_data=None, status=None):
"""Create a new HWSubmission."""
if date_created is None:
date_created = datetime.now(pytz.UTC)
@@ -4116,11 +4115,15 @@
format = HWSubmissionFormat.VERSION_1
submission_set = getUtility(IHWSubmissionSet)
- return submission_set.createSubmission(
+ submission = submission_set.createSubmission(
date_created, format, private, contactable,
submission_key, emailaddress, distroarchseries,
raw_submission, filename, filesize, system)
+ if status is not None:
+ removeSecurityProxy(submission).status = status
+ return submission
+
def makeHWSubmissionDevice(self, submission, device, driver, parent,
hal_device_id):
"""Create a new HWSubmissionDevice."""
=== added file 'scripts/reprocess-hwdb-submissions.py'
--- scripts/reprocess-hwdb-submissions.py 1970-01-01 00:00:00 +0000
+++ scripts/reprocess-hwdb-submissions.py 2011-09-01 16:59:33 +0000
@@ -0,0 +1,79 @@
+#!/usr/bin/python -S
+#
+# Copyright 2009 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+# pylint: disable-msg=W0403
+
+"""
+Cron job that parses pending HWDB submissions.
+
+
+Options:
+ -m, --max-submissions: (optional) The maximum number of submissions
+ which will be processed.
+
+This script iterates over the HWDB submissions with the status
+SUBMITTED, beginning with the oldest submissions, populate the
+HWDB tables with the data from these submissions.
+
+Properly processed submissions are set to the status PROCESSED;
+submissions that cannot be processed are set to the status INVALID.
+"""
+
+import _pythonpath
+
+from lp.services.scripts.base import LaunchpadScript
+from lp.hardwaredb.scripts.hwdbsubmissions import (
+ reprocess_invalid_submissions)
+
+
+class HWDBSubmissionProcessor(LaunchpadScript):
+
+ def add_my_options(self):
+ """See `LaunchpadScript`."""
+ self.parser.add_option(
+ '-m', '--max-submissions',
+ help='Limit the number of submissions which will be processed.')
+ self.parser.add_option(
+ '-w', '--warnings', action="store_true", default=False,
+ help='Include warnings.')
+ self.parser.add_option(
+ '-s', '--start',
+ help=('Process HWSubmission records having an id greater or '
+ 'equal than this value.'))
+
+ def main(self):
+ max_submissions = self.options.max_submissions
+ if max_submissions is not None:
+ try:
+ max_submissions = int(self.options.max_submissions)
+ except ValueError:
+ self.logger.error(
+ 'Invalid value for --max_submissions specified: %r.'
+ % max_submissions)
+ return
+ if max_submissions <= 0:
+ self.logger.error(
+ '--max_submissions must be a positive integer.')
+ return
+ if self.options.start is None:
+ self.logger.error('Option --start not specified.')
+ return
+ try:
+ start = int(self.options.start)
+ except ValueError:
+ self.logger.error('Option --start must have an integer value.')
+ return
+ if start < 0:
+ self.logger.error('--start must be a positive integer.')
+ return
+
+ reprocess_invalid_submissions(
+ start, self.txn, self.logger,
+ max_submissions, self.options.warnings)
+
+if __name__ == '__main__':
+ script = HWDBSubmissionProcessor(
+ 'hwdbsubmissions', dbuser='hwdb-submission-processor')
+ script.lock_and_run()
