launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #04594
[Merge] lp:~lifeless/python-oops/extraction into lp:python-oops
Robert Collins has proposed merging lp:~lifeless/python-oops/extraction into lp:python-oops.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~lifeless/python-oops/extraction/+merge/71510
Rearranging how the oops code is organised - stuff I had extracted so far is really less core, so I've moved it to lp:python-oops-datedir-repo; this merge:
- bumps to v4
- deletes the moved stuff
- adds the actual core - the Config object.
--
https://code.launchpad.net/~lifeless/python-oops/extraction/+merge/71510
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~lifeless/python-oops/extraction into lp:python-oops.
=== modified file 'README'
--- README 2011-08-10 06:08:53 +0000
+++ README 2011-08-15 06:49:23 +0000
@@ -47,6 +47,59 @@
of extensability and will ignore additional keys, and/or raise an error on keys
they expect but which contain unexpected data.
+Typical usage:
+
+* When initializing your script/app/server, create a Config object::
+
+ >>> from oops import Config
+ >>> factory = Config()
+
+* New report will be based on the template::
+
+ >>> factory.template
+ {}
+
+* You can edit the template report::
+
+ >>> factory.template['branch_nick'] = 'mybranch'
+ >>> factory.template['appname'] = 'demo'
+
+* You can supply a callback (for instance, to capture your process memory usage
+when the oops is created)::
+
+ >>> mycallback = lambda report: None
+ >>> factory.on_create.append(mycallback)
+
+* Later on, when something has gone wrong and you want to create an OOPS
+report::
+
+ >>> report = factory.create()
+ >>> report
+ {'appname': 'demo', 'branch_nick': 'mybranch'}
+
+* And then send it off for storage::
+
+ >>> factory.publish(report)
+ []
+
+* Note that publish returns a list - each item in the list is the id allocated
+by the particular repository that recieved the report. (Id allocation is up
+to the repository). The last repository that received the report will have
+record its allocated id in the report for convenience.
+
+ >>> 'id' in report
+ False
+ >>> def demo_publish(report):
+ ... return 'id 1'
+ >>> factory.publishers.append(demo_publish)
+ >>> factory.publish(report)
+ ['id 1']
+ >>> report['id']
+ 'id 1'
+
+* The oops_datedir_repo package contains a local disk based repository which
+can be used as a publisher.
+
More coming soon.
Installation
=== modified file 'oops/__init__.py'
--- oops/__init__.py 2011-08-15 00:34:15 +0000
+++ oops/__init__.py 2011-08-15 06:49:23 +0000
@@ -25,8 +25,10 @@
# established at this point, and setup.py will use a version of next-$(revno).
# If the releaselevel is 'final', then the tarball will be major.minor.micro.
# Otherwise it is major.minor.micro~$(revno).
-__version__ = (0, 0, 3, 'beta', 0)
+__version__ = (0, 0, 4, 'beta', 0)
__all__ = [
+ 'Config'
]
+from oops.config import Config
=== added file 'oops/config.py'
--- oops/config.py 1970-01-01 00:00:00 +0000
+++ oops/config.py 2011-08-15 06:49:23 +0000
@@ -0,0 +1,122 @@
+# Copyright (c) 2010, 2011, Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""The primary interface for clients creating OOPS reports.
+
+Typical usage:
+
+* Configure the library::
+
+ >>> from oops import Config
+ >>> factory = Config()
+ >>> def demo_publish(report):
+ ... return 'id 1'
+ >>> factory.publishers.append(demo_publish)
+
+* Create a report::
+
+ >>> report = factory.create()
+
+* And then send it off for storage::
+
+ >>> factory.publish(report)
+ ['id 1']
+ >>> report
+ {'id': 'id 1'}
+
+* See the Config object pydoc for more information.
+
+OOPS reports a bson serializable dicts (providing a capable lowest common
+denominator - its safe to put binary data in them).
+
+Some well known keys used by Launchpad in its OOPS reports::
+
+* id: The name of this error report.
+* type: The type of the exception that occurred.
+* value: The value of the exception that occurred.
+* time: The time at which the exception occurred.
+* pageid: The identifier for the template/script that oopsed.
+* branch_nick: The branch nickname.
+* revno: The revision number of the branch.
+* tb_text: A text version of the traceback.
+* username: The user associated with the request.
+* url: The URL for the failed request.
+* req_vars: The request variables.
+* branch_nick: A name for the branch of code that was running when the report
+ was triggered.
+* revno: The revision that the branch was at.
+* Informational: A flag, True if the error wasn't fatal- if it was
+ 'informational'.
+"""
+
+
+__all__ = [
+ 'Config',
+ ]
+
+__metaclass__ = type
+
+
+class Config:
+ """The configuration for the OOPS system.
+
+ :ivar on_create: A list of callables to call when making a new report. Each
+ will be called in series with the new report, and its return value will
+ be ignored.
+ :ivar publishers: A list of callables to call when publishing a report.
+ Each will be called in series with the report to publish. Their return
+ value will be assigned to the reports 'id' key : if a publisher
+ allocates a different id than a prior publisher, only the last
+ publisher in the list will have its id present in the report at the
+ end. See the publish() method for more information.
+ """
+
+ def __init__(self):
+ self.on_create = []
+ self.template = {}
+ self.publishers = []
+
+ def create(self):
+ """Create an OOPS.
+
+ The current template used copied to make the new report, and it is
+ passed to all the on_create callbacks for population.
+
+ :return: A fresh OOPS.
+ """
+ result = dict(self.template)
+ [callback(result) for callback in self.on_create]
+ return result
+
+ def publish(self, report):
+ """Publish a report.
+
+ Each publisher is passed the report to publish. Publishers should
+ return the id they allocated-or-used for the report, which gets
+ automatically put into the report for them. All of the ids are also
+ returned to the caller, allowing them to handle the case when multiple
+ publishers allocated different ids. The id from the last publisher is
+ left in the report's id key as a convenience for the common case when
+ only one publisher is present.
+
+ :return: A list of the allocated ids.
+ """
+ result = []
+ for publisher in self.publishers:
+ id = publisher(report)
+ report['id'] = id
+ result.append(id)
+ return result
=== removed file 'oops/serializer_rfc822.py'
--- oops/serializer_rfc822.py 2011-08-15 00:34:15 +0000
+++ oops/serializer_rfc822.py 1970-01-01 00:00:00 +0000
@@ -1,198 +0,0 @@
-# Copyright (c) 2010, 2011, Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Read / Write an OOPS dict as an rfc822 formatted message.
-
-This style of OOPS format is very web server specific, not extensible - it
-should be considered deprecated.
-
-The reports this serializer handles always have the following variables:
-
-* id: The name of this error report.
-* type: The type of the exception that occurred.
-* value: The value of the exception that occurred.
-* time: The time at which the exception occurred.
-* pageid: The identifier for the template/script that oopsed.
-* branch_nick: The branch nickname.
-* revno: The revision number of the branch.
-* tb_text: A text version of the traceback.
-* username: The user associated with the request.
-* url: The URL for the failed request.
-* req_vars: The request variables.
-* branch_nick: A name for the branch of code that was running when the report
- was triggered.
-* revno: The revision that the branch was at.
-* Informational: A flag, True if the error wasn't fatal- if it was
- 'informational'.
-"""
-
-
-__all__ = [
- 'read',
- 'write',
- ]
-
-__metaclass__ = type
-
-import datetime
-import logging
-import rfc822
-import re
-import urllib
-
-import iso8601
-
-
-def read(fp):
- """Deserialize an OOPS from an RFC822 format message."""
- msg = rfc822.Message(fp)
- id = msg.getheader('oops-id')
- exc_type = msg.getheader('exception-type')
- exc_value = msg.getheader('exception-value')
- datestr = msg.getheader('date')
- if datestr is not None:
- date =iso8601.parse_date(msg.getheader('date'))
- else:
- date = None
- pageid = msg.getheader('page-id')
- username = msg.getheader('user')
- url = msg.getheader('url')
- duration = int(float(msg.getheader('duration', '-1')))
- informational = msg.getheader('informational')
- branch_nick = msg.getheader('branch')
- revno = msg.getheader('revision')
-
- # Explicitly use an iterator so we can process the file
- # sequentially. In most instances the iterator will actually
- # be the file object passed in because file objects should
- # support iteration.
- lines = iter(msg.fp)
-
- # Request variables until the first blank line.
- req_vars = []
- for line in lines:
- line = line.strip()
- if line == '':
- break
- key, value = line.split('=', 1)
- req_vars.append((urllib.unquote(key), urllib.unquote(value)))
-
- # Statements until the next blank line.
- statements = []
- for line in lines:
- line = line.strip()
- if line == '':
- break
- match = re.match(
- r'^(\d+)-(\d+)(?:@([\w-]+))?\s+(.*)', line)
- assert match is not None, (
- "Unable to interpret oops line: %s" % line)
- start, end, db_id, statement = match.groups()
- if db_id is not None:
- db_id = intern(db_id) # This string is repeated lots.
- statements.append(
- (int(start), int(end), db_id, statement))
-
- # The rest is traceback.
- tb_text = ''.join(lines)
-
- return dict(id=id, type=exc_type, value=exc_value, time=date,
- pageid=pageid, tb_text=tb_text, username=username, url=url,
- duration=duration, req_vars=req_vars, db_statements=statements,
- informational=informational, branch_nick=branch_nick, revno=revno)
-
-
-def _normalise_whitespace(s):
- """Normalise the whitespace in a string to spaces"""
- if s is None:
- return None # (used by the cast to %s to get 'None')
- return ' '.join(s.split())
-
-
-def _safestr(obj):
- if isinstance(obj, unicode):
- return obj.replace('\\', '\\\\').encode('ASCII',
- 'backslashreplace')
- # A call to str(obj) could raise anything at all.
- # We'll ignore these errors, and print something
- # useful instead, but also log the error.
- # We disable the pylint warning for the blank except.
- try:
- value = str(obj)
- except:
- logging.getLogger('oops.serializer_rfc822').exception(
- 'Error while getting a str '
- 'representation of an object')
- value = '<unprintable %s object>' % (
- str(type(obj).__name__))
- # Some str() calls return unicode objects.
- if isinstance(value, unicode):
- return _safestr(value)
- # encode non-ASCII characters
- value = value.replace('\\', '\\\\')
- value = re.sub(r'[\x80-\xff]',
- lambda match: '\\x%02x' % ord(match.group(0)), value)
- return value
-
-
-def to_chunks(report):
- """Returns a list of bytestrings making up the serialized oops."""
- chunks = []
- chunks.append('Oops-Id: %s\n' % _normalise_whitespace(report['id']))
- if 'type' in report:
- chunks.append(
- 'Exception-Type: %s\n' % _normalise_whitespace(report['type']))
- if 'value' in report:
- chunks.append(
- 'Exception-Value: %s\n' % _normalise_whitespace(report['value']))
- if 'time' in report:
- chunks.append('Date: %s\n' % report['time'].isoformat())
- if 'pageid' in report:
- chunks.append(
- 'Page-Id: %s\n' % _normalise_whitespace(report['pageid']))
- if 'branch_nick' in report:
- chunks.append('Branch: %s\n' % _safestr(report['branch_nick']))
- if 'revno' in report:
- chunks.append('Revision: %s\n' % report['revno'])
- if 'username' in report:
- chunks.append(
- 'User: %s\n' % _normalise_whitespace(report['username']))
- if 'url' in report:
- chunks.append('URL: %s\n' % _normalise_whitespace(report['url']))
- if 'duration' in report:
- chunks.append('Duration: %s\n' % report['duration'])
- if 'informational' in report:
- chunks.append('Informational: %s\n' % report['informational'])
- chunks.append('\n')
- safe_chars = ';/\\?:@&+$, ()*!'
- if 'req_vars' in report:
- for key, value in report['req_vars']:
- chunks.append('%s=%s\n' % (urllib.quote(key, safe_chars),
- urllib.quote(value, safe_chars)))
- chunks.append('\n')
- if 'db_statements' in report:
- for (start, end, database_id, statement) in report['db_statements']:
- chunks.append('%05d-%05d@%s %s\n' % (
- start, end, database_id, _normalise_whitespace(statement)))
- chunks.append('\n')
- if 'tb_text' in report:
- chunks.append(report['tb_text'])
- return chunks
-
-
-def write(report, output):
- """Write a report to a file."""
- output.writelines(to_chunks(report))
=== modified file 'oops/tests/__init__.py'
--- oops/tests/__init__.py 2011-08-10 06:08:53 +0000
+++ oops/tests/__init__.py 2011-08-15 06:49:23 +0000
@@ -21,8 +21,7 @@
def test_suite():
test_mod_names = [
- 'uniquefileallocator',
- 'serializer_rfc822',
+ 'config',
]
return TestLoader().loadTestsFromNames(
['oops.tests.test_' + name for name in test_mod_names])
=== added file 'oops/tests/test_config.py'
--- oops/tests/test_config.py 1970-01-01 00:00:00 +0000
+++ oops/tests/test_config.py 2011-08-15 06:49:23 +0000
@@ -0,0 +1,68 @@
+# Copyright (c) 2010, 2011, Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for the legacy rfc822 based [de]serializer."""
+
+__metaclass__ = type
+
+from functools import partial
+
+import testtools
+from testtools.matchers import Is
+
+from oops.config import Config
+
+
+class TestConfig(testtools.TestCase):
+
+ def test_init(self):
+ config = Config()
+ self.assertEqual([], config.on_create)
+ self.assertEqual([], config.publishers)
+ self.assertEqual({}, config.template)
+
+ def test_on_create_called(self):
+ calls = []
+ def capture(id, report):
+ calls.append((id, report))
+ config = Config()
+ config.on_create.append(partial(capture, '1'))
+ config.on_create.append(partial(capture, '2'))
+ report = config.create()
+ self.assertThat(report, Is(calls[0][1]))
+ self.assertThat(report, Is(calls[1][1]))
+ self.assertEqual([('1', {}), ('2', {})], calls)
+
+ def test_create_template(self):
+ config = Config()
+ config.template['base'] = True
+ report = config.create()
+ self.assertEqual({'base': True}, report)
+
+ def test_publish_calls_publishers(self):
+ calls = []
+ def pub_1(report):
+ return '1'
+ def pub_2(report):
+ self.assertEqual('1', report['id'])
+ return '2'
+ config = Config()
+ config.publishers.append(pub_1)
+ config.publishers.append(pub_2)
+ report = {}
+ self.assertEqual(['1', '2'], config.publish(report))
+ self.assertEqual({'id': '2'}, report)
+
=== removed file 'oops/tests/test_serializer_rfc822.py'
--- oops/tests/test_serializer_rfc822.py 2011-08-15 00:34:15 +0000
+++ oops/tests/test_serializer_rfc822.py 1970-01-01 00:00:00 +0000
@@ -1,271 +0,0 @@
-# Copyright (c) 2010, 2011, Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Tests for the legacy rfc822 based [de]serializer."""
-
-__metaclass__ = type
-
-import datetime
-import StringIO
-from textwrap import dedent
-
-from pytz import utc
-import testtools
-
-from oops.serializer_rfc822 import (
- read,
- to_chunks,
- write,
- )
-
-
-class TestParsing(testtools.TestCase):
-
- def test_read(self):
- """Test ErrorReport.read()."""
- fp = StringIO.StringIO(dedent("""\
- Oops-Id: OOPS-A0001
- Exception-Type: NotFound
- Exception-Value: error message
- Date: 2005-04-01T00:00:00+00:00
- Page-Id: IFoo:+foo-template
- User: Sample User
- URL: http://localhost:9000/foo
- Duration: 42
-
- HTTP_USER_AGENT=Mozilla/5.0
- HTTP_REFERER=http://localhost:9000/
- name%3Dfoo=hello%0Aworld
-
- 00001-00005@store_a SELECT 1
- 00005-00010@store_b SELECT 2
-
- traceback-text"""))
- report = read(fp)
- self.assertEqual(report['id'], 'OOPS-A0001')
- self.assertEqual(report['type'], 'NotFound')
- self.assertEqual(report['value'], 'error message')
- self.assertEqual(
- report['time'], datetime.datetime(2005, 4, 1, tzinfo=utc))
- self.assertEqual(report['pageid'], 'IFoo:+foo-template')
- self.assertEqual(report['tb_text'], 'traceback-text')
- self.assertEqual(report['username'], 'Sample User')
- self.assertEqual(report['url'], 'http://localhost:9000/foo')
- self.assertEqual(report['duration'], 42)
- self.assertEqual(len(report['req_vars']), 3)
- self.assertEqual(report['req_vars'][0], ('HTTP_USER_AGENT',
- 'Mozilla/5.0'))
- self.assertEqual(report['req_vars'][1], ('HTTP_REFERER',
- 'http://localhost:9000/'))
- self.assertEqual(report['req_vars'][2], ('name=foo', 'hello\nworld'))
- self.assertEqual(len(report['db_statements']), 2)
- self.assertEqual(
- report['db_statements'][0],
- (1, 5, 'store_a', 'SELECT 1'))
- self.assertEqual(
- report['db_statements'][1],
- (5, 10, 'store_b', 'SELECT 2'))
-
- def test_read_no_store_id(self):
- """Test ErrorReport.read() for old logs with no store_id."""
- fp = StringIO.StringIO(dedent("""\
- Oops-Id: OOPS-A0001
- Exception-Type: NotFound
- Exception-Value: error message
- Date: 2005-04-01T00:00:00+00:00
- Page-Id: IFoo:+foo-template
- User: Sample User
- URL: http://localhost:9000/foo
- Duration: 42
-
- HTTP_USER_AGENT=Mozilla/5.0
- HTTP_REFERER=http://localhost:9000/
- name%3Dfoo=hello%0Aworld
-
- 00001-00005 SELECT 1
- 00005-00010 SELECT 2
-
- traceback-text"""))
- report = read(fp)
- self.assertEqual(report['id'], 'OOPS-A0001')
- self.assertEqual(report['type'], 'NotFound')
- self.assertEqual(report['value'], 'error message')
- self.assertEqual(
- report['time'], datetime.datetime(2005, 4, 1, tzinfo=utc))
- self.assertEqual(report['pageid'], 'IFoo:+foo-template')
- self.assertEqual(report['tb_text'], 'traceback-text')
- self.assertEqual(report['username'], 'Sample User')
- self.assertEqual(report['url'], 'http://localhost:9000/foo')
- self.assertEqual(report['duration'], 42)
- self.assertEqual(len(report['req_vars']), 3)
- self.assertEqual(report['req_vars'][0], ('HTTP_USER_AGENT',
- 'Mozilla/5.0'))
- self.assertEqual(report['req_vars'][1], ('HTTP_REFERER',
- 'http://localhost:9000/'))
- self.assertEqual(report['req_vars'][2], ('name=foo', 'hello\nworld'))
- self.assertEqual(len(report['db_statements']), 2)
- self.assertEqual(report['db_statements'][0], (1, 5, None, 'SELECT 1'))
- self.assertEqual(report['db_statements'][1], (5, 10, None, 'SELECT 2'))
- self.assertFalse(report['informational'])
-
- def test_read_branch_nick_revno(self):
- """Test ErrorReport.read()."""
- fp = StringIO.StringIO(dedent("""\
- Oops-Id: OOPS-A0001
- Exception-Type: NotFound
- Exception-Value: error message
- Date: 2005-04-01T00:00:00+00:00
- Page-Id: IFoo:+foo-template
- User: Sample User
- URL: http://localhost:9000/foo
- Duration: 42
- Branch: mybranch
- Revision: 45
-
- HTTP_USER_AGENT=Mozilla/5.0
- HTTP_REFERER=http://localhost:9000/
- name%3Dfoo=hello%0Aworld
-
- 00001-00005@store_a SELECT 1
- 00005-00010@store_b SELECT 2
-
- traceback-text"""))
- report = read(fp)
- self.assertEqual(report['branch_nick'], 'mybranch')
- self.assertEqual(report['revno'], '45')
-
- def test_minimal_oops(self):
- # If we get a crazy-small oops, we can read it sensibly. Because there
- # is existing legacy code, all keys are filled in with None, [] rather
- # than being empty.
- fp = StringIO.StringIO(dedent("""\
- Oops-Id: OOPS-A0001
- """))
- report = read(fp)
- self.assertEqual(report['id'], 'OOPS-A0001')
- self.assertEqual(report['type'], None)
- self.assertEqual(report['value'], None)
- self.assertEqual(report['time'], None)
- self.assertEqual(report['pageid'], None)
- self.assertEqual(report['tb_text'], '')
- self.assertEqual(report['username'], None)
- self.assertEqual(report['url'], None)
- self.assertEqual(report['duration'], -1)
- self.assertEqual(len(report['req_vars']), 0)
- self.assertEqual(len(report['db_statements']), 0)
- self.assertFalse(report['informational'])
- self.assertEqual(report['branch_nick'], None)
- self.assertEqual(report['revno'], None)
-
-
-class TestSerializing(testtools.TestCase):
-
- def test_write_file(self):
- output = StringIO.StringIO()
- report = {
- 'id': 'OOPS-A0001',
- 'type': 'NotFound',
- 'value': 'error message',
- 'time': datetime.datetime(2005, 04, 01, 00, 00, 00, tzinfo=utc),
- 'pageid': 'IFoo:+foo-template',
- 'tb_text': 'traceback-text',
- 'username': 'Sample User',
- 'url': 'http://localhost:9000/foo',
- 'duration': 42,
- 'req_vars': [('HTTP_USER_AGENT', 'Mozilla/5.0'),
- ('HTTP_REFERER', 'http://localhost:9000/'),
- ('name=foo', 'hello\nworld')],
- 'db_statements': [(1, 5, 'store_a', 'SELECT 1'),
- (5, 10, 'store_b', 'SELECT\n2')],
- 'informational': False,
- 'branch_nick': 'mybranch',
- 'revno': '45',
- }
- write(report, output)
- self.assertEqual(output.getvalue(), dedent("""\
- Oops-Id: OOPS-A0001
- Exception-Type: NotFound
- Exception-Value: error message
- Date: 2005-04-01T00:00:00+00:00
- Page-Id: IFoo:+foo-template
- Branch: mybranch
- Revision: 45
- User: Sample User
- URL: http://localhost:9000/foo
- Duration: 42
- Informational: False
-
- HTTP_USER_AGENT=Mozilla/5.0
- HTTP_REFERER=http://localhost:9000/
- name%3Dfoo=hello%0Aworld
-
- 00001-00005@store_a SELECT 1
- 00005-00010@store_b SELECT 2
-
- traceback-text"""))
-
- def test_to_chunks(self):
- report = {
- 'id': 'OOPS-A0001',
- 'type': 'NotFound',
- 'value': 'error message',
- 'time': datetime.datetime(2005, 04, 01, 00, 00, 00, tzinfo=utc),
- 'pageid': 'IFoo:+foo-template',
- 'tb_text': 'traceback-text',
- 'username': 'Sample User',
- 'url': 'http://localhost:9000/foo',
- 'duration': 42,
- 'req_vars': [('HTTP_USER_AGENT', 'Mozilla/5.0'),
- ('HTTP_REFERER', 'http://localhost:9000/'),
- ('name=foo', 'hello\nworld')],
- 'db_statements': [(1, 5, 'store_a', 'SELECT 1'),
- (5, 10, 'store_b', 'SELECT\n2')],
- 'informational': False,
- 'branch_nick': 'mybranch',
- 'revno': '45',
- }
- self.assertEqual([
- "Oops-Id: OOPS-A0001\n",
- "Exception-Type: NotFound\n",
- "Exception-Value: error message\n",
- "Date: 2005-04-01T00:00:00+00:00\n",
- "Page-Id: IFoo:+foo-template\n",
- "Branch: mybranch\n",
- "Revision: 45\n",
- "User: Sample User\n",
- "URL: http://localhost:9000/foo\n";,
- "Duration: 42\n",
- "Informational: False\n",
- "\n",
- "HTTP_USER_AGENT=Mozilla/5.0\n",
- "HTTP_REFERER=http://localhost:9000/\n";,
- "name%3Dfoo=hello%0Aworld\n",
- "\n",
- "00001-00005@store_a SELECT 1\n",
- "00005-00010@store_b SELECT 2\n",
- "\n",
- "traceback-text",
- ],
- to_chunks(report))
-
- def test_minimal_oops(self):
- # An oops with just an id, though arguably crazy, is written
- # sensibly.
- report = {'id': 'OOPS-1234'}
- self.assertEqual([
- "Oops-Id: OOPS-1234\n",
- "\n"
- ], to_chunks(report))
=== removed file 'oops/tests/test_uniquefileallocator.py'
--- oops/tests/test_uniquefileallocator.py 2011-08-10 05:08:58 +0000
+++ oops/tests/test_uniquefileallocator.py 1970-01-01 00:00:00 +0000
@@ -1,161 +0,0 @@
-# Copyright (c) 2010, 2011, Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Tests for the unique file naming facility."""
-
-__metaclass__ = type
-
-import datetime
-import os
-import stat
-
-from fixtures import TempDir
-import pytz
-import testtools
-
-from oops.uniquefileallocator import UniqueFileAllocator
-
-
-UTC = pytz.timezone('UTC')
-
-
-class TestUniqueFileAllocator(testtools.TestCase):
-
- def setUp(self):
- super(TestUniqueFileAllocator, self).setUp()
- tempdir = self.useFixture(TempDir())
- self._tempdir = tempdir.path
-
- def test_setToken(self):
- namer = UniqueFileAllocator("/any-old/path/", 'OOPS', 'T')
- self.assertEqual('T', namer.get_log_infix())
-
- # Some scripts will append a string token to the prefix.
- namer.setToken('CW')
- self.assertEqual('TCW', namer.get_log_infix())
-
- # Some scripts run multiple processes and append a string number
- # to the prefix.
- namer.setToken('1')
- self.assertEqual('T1', namer.get_log_infix())
-
- def assertUniqueFileAllocator(self, namer, now, expected_id,
- expected_last_id, expected_suffix, expected_lastdir):
- logid, filename = namer.newId(now)
- self.assertEqual(logid, expected_id)
- self.assertEqual(filename,
- os.path.join(namer._output_root, expected_suffix))
- self.assertEqual(namer._last_serial, expected_last_id)
- self.assertEqual(namer._last_output_dir,
- os.path.join(namer._output_root, expected_lastdir))
-
- def test_newId(self):
- # TODO: This should return an id, fileobj instead of a file name, to
- # reduce races with threads that are slow to use what they asked for,
- # when combined with configuration changes causing disk scans. That
- # would also permit using a completely stubbed out file system,
- # reducing IO in tests that use UniqueFileAllocator (such as all the
- # pagetests in Launchpad. At that point an interface to obtain a
- # factory of UniqueFileAllocator's would be useful to parameterise the
- # entire test suite.
- namer = UniqueFileAllocator(self._tempdir, 'OOPS', 'T')
- # first name of the day
- self.assertUniqueFileAllocator(namer,
- datetime.datetime(2006, 04, 01, 00, 30, 00, tzinfo=UTC),
- 'OOPS-91T1', 1, '2006-04-01/01800.T1', '2006-04-01')
- # second name of the day
- self.assertUniqueFileAllocator(namer,
- datetime.datetime(2006, 04, 01, 12, 00, 00, tzinfo=UTC),
- 'OOPS-91T2', 2, '2006-04-01/43200.T2', '2006-04-01')
-
- # first name of the following day sets a new dir and the id starts
- # over.
- self.assertUniqueFileAllocator(namer,
- datetime.datetime(2006, 04, 02, 00, 30, 00, tzinfo=UTC),
- 'OOPS-92T1', 1, '2006-04-02/01800.T1', '2006-04-02')
-
- # Setting a token inserts the token into the filename.
- namer.setToken('YYY')
- logid, filename = namer.newId(
- datetime.datetime(2006, 04, 02, 00, 30, 00, tzinfo=UTC))
- self.assertEqual(logid, 'OOPS-92TYYY2')
-
- # Setting a type controls the log id:
- namer.setToken('')
- namer._log_type = "PROFILE"
- logid, filename = namer.newId(
- datetime.datetime(2006, 04, 02, 00, 30, 00, tzinfo=UTC))
- self.assertEqual(logid, 'PROFILE-92T3')
-
- # Native timestamps are not permitted - UTC only.
- now = datetime.datetime(2006, 04, 02, 00, 30, 00)
- self.assertRaises(ValueError, namer.newId, now)
-
- def test_changeErrorDir(self):
- """Test changing the log output dir."""
- namer = UniqueFileAllocator(self._tempdir, 'OOPS', 'T')
-
- # First an id in the original error directory.
- self.assertUniqueFileAllocator(namer,
- datetime.datetime(2006, 04, 01, 00, 30, 00, tzinfo=UTC),
- 'OOPS-91T1', 1, '2006-04-01/01800.T1', '2006-04-01')
-
- # UniqueFileAllocator uses the _output_root attribute to get the
- # current output directory.
- new_output_dir = self.useFixture(TempDir()).path
- namer._output_root = new_output_dir
-
- # Now an id on the same day, in the new directory.
- now = datetime.datetime(2006, 04, 01, 12, 00, 00, tzinfo=UTC)
- log_id, filename = namer.newId(now)
-
- # Since it's a new directory, with no previous logs, the id is 1
- # again, rather than 2.
- self.assertEqual(log_id, 'OOPS-91T1')
- self.assertEqual(namer._last_serial, 1)
- self.assertEqual(namer._last_output_dir,
- os.path.join(new_output_dir, '2006-04-01'))
-
- def test_findHighestSerial(self):
- namer = UniqueFileAllocator(self._tempdir, "OOPS", "T")
- # Creates the dir using now as the timestamp.
- output_dir = namer.output_dir()
- # write some files, in non-serial order.
- open(os.path.join(output_dir, '12343.T1'), 'w').close()
- open(os.path.join(output_dir, '12342.T2'), 'w').close()
- open(os.path.join(output_dir, '12345.T3'), 'w').close()
- open(os.path.join(output_dir, '1234567.T0010'), 'w').close()
- open(os.path.join(output_dir, '12346.A42'), 'w').close()
- open(os.path.join(output_dir, '12346.B100'), 'w').close()
- # The namer should figure out the right highest serial.
- self.assertEqual(namer._findHighestSerial(output_dir), 10)
-
- def test_output_dir_permission(self):
- # Set up default dir creation mode to rwx------.
- umask_permission = stat.S_IRWXG | stat.S_IRWXO
- old_umask = os.umask(umask_permission)
- namer = UniqueFileAllocator(self._tempdir, "OOPS", "T")
- output_dir = namer.output_dir()
- st = os.stat(output_dir)
- # Permission we want here is: rwxr-xr-x
- wanted_permission = (
- stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH |
- stat.S_IXOTH)
- # Get only the permission bits for this directory.
- dir_permission = stat.S_IMODE(st.st_mode)
- self.assertEqual(dir_permission, wanted_permission)
- # Restore the umask to the original value.
- ignored = os.umask(old_umask)
=== removed file 'oops/uniquefileallocator.py'
--- oops/uniquefileallocator.py 2011-08-10 05:08:58 +0000
+++ oops/uniquefileallocator.py 1970-01-01 00:00:00 +0000
@@ -1,213 +0,0 @@
-# Copyright (c) 2010, 2011, Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-
-"""Create uniquely named log files on disk."""
-
-
-__all__ = ['UniqueFileAllocator']
-
-__metaclass__ = type
-
-
-import datetime
-import errno
-import os.path
-import stat
-import threading
-
-import pytz
-
-
-UTC = pytz.utc
-
-# the section of the ID before the instance identifier is the
-# days since the epoch, which is defined as the start of 2006.
-epoch = datetime.datetime(2006, 01, 01, 00, 00, 00, tzinfo=UTC)
-
-
-class UniqueFileAllocator:
- """Assign unique file names to logs being written from an app/script.
-
- UniqueFileAllocator causes logs written from one process to be uniquely
- named. It is not safe for use in multiple processes with the same output
- root - each process must have a unique output root.
- """
-
- def __init__(self, output_root, log_type, log_subtype):
- """Create a UniqueFileAllocator.
-
- :param output_root: The root directory that logs should be placed in.
- :param log_type: A string to use as a prefix in the ID assigned to new
- logs. For instance, "OOPS".
- :param log_subtype: A string to insert in the generate log filenames
- between the day number and the serial. For instance "T" for
- "Testing".
- """
- self._lock = threading.Lock()
- self._output_root = output_root
- self._last_serial = 0
- self._last_output_dir = None
- self._log_type = log_type
- self._log_subtype = log_subtype
- self._log_token = ""
-
- def _findHighestSerialFilename(self, directory=None, time=None):
- """Find details of the last log present in the given directory.
-
- This function only considers logs with the currently
- configured log_subtype.
-
- One of directory, time must be supplied.
-
- :param directory: Look in this directory.
- :param time: Look in the directory that a log written at this time
- would have been written to. If supplied, supercedes directory.
- :return: a tuple (log_serial, log_filename), which will be (0,
- None) if no logs are found. log_filename is a usable path, not
- simply the basename.
- """
- if directory is None:
- directory = self.output_dir(time)
- prefix = self.get_log_infix()
- lastid = 0
- lastfilename = None
- for filename in os.listdir(directory):
- logid = filename.rsplit('.', 1)[1]
- if not logid.startswith(prefix):
- continue
- logid = logid[len(prefix):]
- if logid.isdigit() and (lastid is None or int(logid) > lastid):
- lastid = int(logid)
- lastfilename = filename
- if lastfilename is not None:
- lastfilename = os.path.join(directory, lastfilename)
- return lastid, lastfilename
-
- def _findHighestSerial(self, directory):
- """Find the last serial actually applied to disk in directory.
-
- The purpose of this function is to not repeat sequence numbers
- if the logging application is restarted.
-
- This method is not thread safe, and only intended to be called
- from the constructor (but it is called from other places in
- integration tests).
- """
- return self._findHighestSerialFilename(directory)[0]
-
- def getFilename(self, log_serial, time):
- """Get the filename for a given log serial and time."""
- log_subtype = self.get_log_infix()
- # TODO: Calling output_dir causes a global lock to be taken and a
- # directory scan, which is bad for performance. It would be better
- # to have a split out 'directory name for time' function which the
- # 'want to use this directory now' function can call.
- output_dir = self.output_dir(time)
- second_in_day = time.hour * 3600 + time.minute * 60 + time.second
- return os.path.join(
- output_dir, '%05d.%s%s' % (
- second_in_day, log_subtype, log_serial))
-
- def get_log_infix(self):
- """Return the current log infix to use in ids and file names."""
- return self._log_subtype + self._log_token
-
- def newId(self, now=None):
- """Returns an (id, filename) pair for use by the caller.
-
- The ID is composed of a short string to identify the Launchpad
- instance followed by an ID that is unique for the day.
-
- The filename is composed of the zero padded second in the day
- followed by the ID. This ensures that reports are in date order when
- sorted lexically.
- """
- if now is not None:
- now = now.astimezone(UTC)
- else:
- now = datetime.datetime.now(UTC)
- # We look up the error directory before allocating a new ID,
- # because if the day has changed, errordir() will reset the ID
- # counter to zero.
- self.output_dir(now)
- self._lock.acquire()
- try:
- self._last_serial += 1
- newid = self._last_serial
- finally:
- self._lock.release()
- subtype = self.get_log_infix()
- day_number = (now - epoch).days + 1
- log_id = '%s-%d%s%d' % (self._log_type, day_number, subtype, newid)
- filename = self.getFilename(newid, now)
- return log_id, filename
-
- def output_dir(self, now=None):
- """Find or make the directory to allocate log names in.
-
- Log names are assigned within subdirectories containing the date the
- assignment happened.
- """
- if now is not None:
- now = now.astimezone(UTC)
- else:
- now = datetime.datetime.now(UTC)
- date = now.strftime('%Y-%m-%d')
- result = os.path.join(self._output_root, date)
- if result != self._last_output_dir:
- self._lock.acquire()
- try:
- self._last_output_dir = result
- # make sure the directory exists
- try:
- os.makedirs(result)
- except OSError, e:
- if e.errno != errno.EEXIST:
- raise
- # Make sure the directory permission is set to: rwxr-xr-x
- permission = (
- stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP |
- stat.S_IROTH | stat.S_IXOTH)
- os.chmod(result, permission)
- # TODO: Note that only one process can do this safely: its not
- # cross-process safe, and also not entirely threadsafe:
- # another # thread that has a new log and hasn't written it
- # could then use that serial number. We should either make it
- # really safe, or remove the contention entirely and log
- # uniquely per thread of execution.
- self._last_serial = self._findHighestSerial(result)
- finally:
- self._lock.release()
- return result
-
- def listRecentReportFiles(self):
- now = datetime.datetime.now(UTC)
- yesterday = now - datetime.timedelta(days=1)
- directories = [self.output_dir(now), self.output_dir(yesterday)]
- for directory in directories:
- report_names = os.listdir(directory)
- for name in sorted(report_names, reverse=True):
- yield directory, name
-
- def setToken(self, token):
- """Append a string to the log subtype in filenames and log ids.
-
- :param token: a string to append..
- Scripts that run multiple processes can use this to create a
- unique identifier for each process.
- """
- self._log_token = token
=== modified file 'setup.py'
--- setup.py 2011-08-15 00:34:15 +0000
+++ setup.py 2011-08-15 06:49:23 +0000
@@ -22,7 +22,7 @@
description = file(os.path.join(os.path.dirname(__file__), 'README'), 'rb').read()
setup(name="oops",
- version="0.0.3",
+ version="0.0.4",
description="OOPS report model and default allocation/[de]serialization.",
long_description=description,
maintainer="Launchpad Developers",
