launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #01145
[Merge] lp:~henninge/launchpad/recife-poimport into lp:~launchpad/launchpad/recife
Henning Eggers has proposed merging lp:~henninge/launchpad/recife-poimport into lp:~launchpad/launchpad/recife.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
== Details ==
This branch was split off the work for bug 611674. There were two goals:
- Make the test use a source package because a lot of code still hardwired to "is_current_upstream". Using a source package instead of a product series avoids collisions between old and new model.
- Update the test to not use (or at least not as much) sample data.
Because the file was quite big, I split off the tests that run the import script into poimport-script.txt. This new file is completely independent of sample data, it even clears out the sample data from the queue before starting.
Since the new model is explicitly referring to Ubuntu in many places, I found it useful to have a "makeUbuntuDistroSeries" factory methods. This avoids having to deal with Celebrities in the test.
== Test ==
bin/test -vvcm lp.translations -t poimport.txt -t poimport-script.txt
--
https://code.launchpad.net/~henninge/launchpad/recife-poimport/+merge/36165
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~henninge/launchpad/recife-poimport into lp:~launchpad/launchpad/recife.
=== modified file 'lib/lp/testing/factory.py'
--- lib/lp/testing/factory.py 2010-09-08 02:19:20 +0000
+++ lib/lp/testing/factory.py 2010-09-21 16:23:42 +0000
@@ -1848,8 +1848,18 @@
series.status = status
return ProxyFactory(series)
+ def makeUbuntuDistroRelease(self, version=None,
+ status=SeriesStatus.DEVELOPMENT,
+ parent_series=None, name=None,
+ displayname=None):
+ """Short cut to use the celebrity 'ubuntu' as the distribution."""
+ ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
+ return self.makeDistroRelease(
+ ubuntu, version, status, parent_series, name, displayname)
+
# Most people think of distro releases as distro series.
makeDistroSeries = makeDistroRelease
+ makeUbuntuDistroSeries = makeUbuntuDistroRelease
def makeDistroSeriesDifference(
self, derived_series=None, source_package_name_str=None,
=== added file 'lib/lp/translations/doc/poimport-script.txt'
--- lib/lp/translations/doc/poimport-script.txt 1970-01-01 00:00:00 +0000
+++ lib/lp/translations/doc/poimport-script.txt 2010-09-21 16:23:42 +0000
@@ -0,0 +1,342 @@
+Import Script
+=============
+
+The imports are performed by a dedicated cron script.
+
+A template and two pofile will be imported.
+
+ >>> potemplate_header = r"""
+ ... msgid ""
+ ... msgstr ""
+ ... "POT-Creation-Date: 2004-07-11 16:16+0900\n"
+ ... "Content-Type: text/plain; charset=CHARSET\n"
+ ... "Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n"
+ ...
+ ... """
+
+ >>> pofile_header = r"""
+ ... msgid ""
+ ... msgstr ""
+ ... "PO-Revision-Date: 2005-06-03 20:41+0100\n"
+ ... "Last-Translator: Foo <no-priv@xxxxxxxxxxxxx>\n"
+ ... "Content-Type: text/plain; charset=UTF-8\n"
+ ... "Plural-Forms: nplurals=2; plural=(n!=1);\n"
+ ...
+ ... """
+
+ >>> po_content = r"""
+ ... #: test.c:13
+ ... msgid "baz"
+ ... msgstr "%s"
+ ...
+ ... #, c-format
+ ... msgid "Foo %%s"
+ ... msgstr "%s"
+ ...
+ ... #, c-format
+ ... msgid "Singular %%d"
+ ... msgid_plural "Plural %%d"
+ ... msgstr[0] "%s"
+ ... msgstr[1] "%s"
+ ...
+ ... msgid "translator-credits"
+ ... msgstr "%s"
+ ... """
+
+ >>> potemplate_content = potemplate_header + po_content % (('',) * 5)
+ >>> pofile_eo_content = pofile_header + po_content % (
+ ... "baz eo", "Foo eo %s", "Singular eo %s", "Plural eo %s",
+ ... "helpful-eo@xxxxxxxxxxx")
+ >>> pofile_nl_content = pofile_header + po_content % (
+ ... "baz nl", "Foo nl %s", "Singular nl %s", "Plural nl %s",
+ ... "helpful-nl@xxxxxxxxxxx")
+
+There is annoying sample data in the queue that needs to be removed.
+
+ >>> from lp.translations.interfaces.translationimportqueue import (
+ ... ITranslationImportQueue, RosettaImportStatus)
+ >>> queue = getUtility(ITranslationImportQueue)
+ >>> for entry in queue:
+ ... queue.remove(entry)
+
+The files have been uploaded to the queue for a source package and have
+already been approved.
+
+ >>> from zope.security.proxy import removeSecurityProxy
+ >>> distroseries = factory.makeUbuntuDistroSeries()
+ >>> naked_distroseries = removeSecurityProxy(distroseries)
+ >>> naked_distroseries.distribution.official_rosetta = True
+ >>> sourcepackagename = factory.makeSourcePackageName()
+ >>> potemplate = factory.makePOTemplate(
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename)
+ >>> pofile_eo = potemplate.newPOFile('eo')
+ >>> pofile_nl = potemplate.newPOFile('nl')
+
+ >>> from canonical.launchpad.interfaces.launchpad import (
+ ... ILaunchpadCelebrities)
+ >>> rosetta_experts = getUtility(ILaunchpadCelebrities).rosetta_experts
+
+ >>> template_entry = queue.addOrUpdateEntry(
+ ... potemplate.path, potemplate_content, True, potemplate.owner,
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate)
+ >>> pofile_eo_entry = queue.addOrUpdateEntry(
+ ... 'eo.po', pofile_eo_content, True, potemplate.owner,
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate, pofile=pofile_eo)
+ >>> pofile_nl_entry = queue.addOrUpdateEntry(
+ ... 'nl.po', pofile_nl_content, True, potemplate.owner,
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate, pofile=pofile_nl)
+ >>> transaction.commit()
+
+ >>> for entry in queue:
+ ... entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
+ >>> transaction.commit()
+
+As it happens, the administrator has blocked imports to the distroseries, e.g.
+because an in-database update of its translations has been scheduled
+and we don't want interference from queued imports while that happens.
+It doesn't really matter whether entries still get auto-approved, but
+we can't accept new translation imports just now.
+
+ >>> distroseries.defer_translation_imports
+ True
+
+ >>> from canonical.launchpad.scripts import FakeLogger
+ >>> from lp.translations.scripts.po_import import TranslationsImport
+ >>> import email
+ >>> from lp.services.mail import stub
+ >>> process = TranslationsImport('poimport', test_args=[])
+ >>> process.logger = FakeLogger()
+ >>> process.main()
+ DEBUG Starting the import process.
+ INFO No requests pending.
+
+When imports are allowed, the import script can do its work.
+
+ >>> naked_distroseries.defer_translation_imports = False
+
+ >>> process = TranslationsImport('poimport', test_args=[])
+ >>> process.logger = FakeLogger()
+ >>> process.main()
+ DEBUG Starting the import process.
+ INFO Importing: Template ...
+ INFO Importing: Esperanto (eo) ... of ...
+ INFO Importing: Dutch (nl) ... of ...
+ INFO Import requests completed.
+ DEBUG Finished the import process.
+
+The import script also generates an email similar to the ones we saw
+composed before, but also sends it.
+
+ >>> len(stub.test_emails)
+ 1
+
+ >>> from_addr, to_addrs, raw_message = stub.test_emails.pop()
+ >>> msg = email.message_from_string(raw_message)
+ >>> print msg["Subject"]
+ Translation template import - ...
+
+ >>> print msg.get_payload(decode=True)
+ Hello ...,
+ <BLANKLINE>
+ On ..., you uploaded a translation
+ template for ... in Launchpad.
+ <BLANKLINE>
+ The template has now been imported successfully.
+ <BLANKLINE>
+ Thank you,
+ <BLANKLINE>
+ The Launchpad team
+
+The entries that remain in the queue as "imported" age over time.
+
+ >>> import datetime
+ >>> for entry in queue:
+ ... removeSecurityProxy(entry).date_status_changed -= (
+ ... datetime.timedelta(days=30))
+
+
+Now the queue gardener runs. This can happen anytime, since it's
+asynchronous to the po-import script. The script tries to approve any
+entries that have not been approved, but look like they could be,
+without human intervention. This involves a bit of guesswork about what
+the imported file is and where it belongs. It similarly blocks entries
+that it thinks should be blocked, and also purges deleted or completed
+entries from the queue. Running at this point, all it does is purge the
+two hand-approved Welsh translations that have just been imported.
+
+ >>> import logging
+ >>> from lp.testing.logger import MockLogger
+ >>> from lp.translations.scripts.import_queue_gardener import (
+ ... ImportQueueGardener)
+ >>> process = ImportQueueGardener('approver', test_args=[])
+ >>> process.logger = MockLogger()
+ >>> process.logger.setLevel(logging.INFO)
+ >>> process.main()
+ log> Removed 3 entries from the queue.
+ >>> transaction.commit()
+
+If users upload two versions of the same file, they are imported in the
+order in which they were uploaded.
+
+ >>> import pytz
+ >>> UTC = pytz.timezone('UTC')
+ >>> first_pofile_content = r'''
+ ... msgid ""
+ ... msgstr ""
+ ... "PO-Revision-Date: 2005-06-04 20:41+0100\n"
+ ... "Last-Translator: Foo <no-priv@xxxxxxxxxxxxx>\n"
+ ... "Content-Type: text/plain; charset=UTF-8\n"
+ ... "X-Rosetta-Export-Date: %s\n"
+ ...
+ ... msgid "Foo %%s"
+ ... msgstr "Bar"
+ ...
+ ... msgid "translator-credits"
+ ... msgstr "The world will never know."
+ ... ''' % datetime.datetime.now(UTC).isoformat()
+
+ >>> second_pofile_content = r'''
+ ... msgid ""
+ ... msgstr ""
+ ... "PO-Revision-Date: 2005-06-04 21:41+0100\n"
+ ... "Last-Translator: Jordi Mallach <jordi@xxxxxxxxxxxxx>\n"
+ ... "Content-Type: text/plain; charset=UTF-8\n"
+ ... "X-Rosetta-Export-Date: %s\n"
+ ...
+ ... msgid "Foo %%s"
+ ... msgstr "Bars"
+ ...
+ ... msgid "translator-credits"
+ ... msgstr "I'd like to thank John, Kathy, my pot plants, and all the..."
+ ... ''' % datetime.datetime.now(UTC).isoformat()
+
+Attach the first version of the file.
+
+ >>> entry = queue.addOrUpdateEntry(
+ ... pofile_eo.path, first_pofile_content, False, rosetta_experts,
+ ... sourcepackagename=sourcepackagename, distroseries=distroseries)
+ >>> transaction.commit()
+
+It's in the queue now.
+
+ >>> queue.countEntries()
+ 1
+
+For the second version, we need a new importer.
+
+ >>> importer_person = factory.makePerson()
+
+Attach the second version of the file.
+
+ >>> entry = queue.addOrUpdateEntry(
+ ... pofile_eo.path, second_pofile_content, False, importer_person,
+ ... sourcepackagename=sourcepackagename, distroseries=distroseries)
+ >>> transaction.commit()
+
+It's in the queue now.
+
+ >>> queue.countEntries()
+ 2
+ >>> print entry.status.name
+ NEEDS_REVIEW
+
+The queue gardener runs again. This time it sees the two submitted
+translations and approves them for import based on some heuristic
+intelligence.
+
+ >>> process = ImportQueueGardener('approver', test_args=[])
+ >>> process.logger = MockLogger()
+ >>> process.logger.setLevel(logging.INFO)
+ >>> process.main()
+ log> The automatic approval system approved some entries.
+ >>> print entry.status.name
+ APPROVED
+ >>> from canonical.launchpad.ftests import syncUpdate
+ >>> syncUpdate(entry)
+
+Now that these submissions have been approved, the next run of the
+import script picks them up and processes them.
+
+ >>> process = TranslationsImport('poimport', test_args=[])
+ >>> process.logger = FakeLogger()
+ >>> process.main()
+ DEBUG Starting the import process.
+ INFO Importing: Esperanto (eo) ... of ...
+ INFO Importing: Esperanto (eo) ... of ...
+ INFO Import requests completed.
+ DEBUG Finished the import process.
+
+ >>> print entry.status.name
+ IMPORTED
+ >>> syncUpdate(entry)
+
+And there are no more entries to import
+
+ >>> queue.getFirstEntryToImport() is None
+ True
+
+We've imported a new translation for "Foo %s."
+
+ >>> from lp.services.worlddata.interfaces.language import ILanguageSet
+ >>> esperanto = getUtility(ILanguageSet).getLanguageByCode('eo')
+ >>> foos = potemplate['Foo %s'].getLocalTranslationMessages(
+ ... potemplate, esperanto)
+ >>> sorted([foo.msgstr0.translation for foo in foos])
+ [u'Bar', u'Bars']
+
+Since this last upload was not the upstream one, however, its credits
+message translations were ignored.
+
+ >>> potmsgset = pofile_eo.potemplate.getPOTMsgSetByMsgIDText(
+ ... u'translator-credits')
+ >>> message = potmsgset.getCurrentTranslationMessage(
+ ... pofile_eo.potemplate, pofile_eo.language)
+ >>> message.msgstr0.translation
+ u'helpful-eo@xxxxxxxxxxx'
+ >>> list(potemplate['translator-credits'].getLocalTranslationMessages(
+ ... potemplate, esperanto))
+ []
+
+
+No Contact Address
+------------------
+
+Not every user has a valid email address. For instance, Kermit the
+Hermit has none at the moment.
+
+ >>> from canonical.launchpad.interfaces.emailaddress import (
+ ... EmailAddressStatus)
+ >>> from canonical.launchpad.helpers import get_contact_email_addresses
+ >>> hermit = factory.makePerson(
+ ... name='hermit', email_address_status=EmailAddressStatus.OLD)
+
+ >>> len(get_contact_email_addresses(hermit))
+ 0
+
+Kermit uploads a translation, which gets approved.
+
+ >>> pofile = factory.makePOFile('lo', potemplate)
+ >>> entry = queue.addOrUpdateEntry(
+ ... 'lo.po', 'Invalid content', True, hermit,
+ ... pofile=pofile, potemplate=potemplate,
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename)
+ >>> entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
+ >>> transaction.commit()
+
+The import fails. The importer would like to send Kermit an email about
+this, but is unable to. This is unfortunate, but does not faze the
+importer. It completes normally.
+
+ >>> process = TranslationsImport('poimport', test_args=[])
+ >>> process.logger = FakeLogger()
+ >>> process.main()
+ DEBUG Starting the import process.
+ INFO Importing: Lao ...
+ INFO Import requests completed.
+ DEBUG Finished the import process.
+
+ >>> print entry.status.name
+ FAILED
=== modified file 'lib/lp/translations/doc/poimport.txt'
--- lib/lp/translations/doc/poimport.txt 2010-08-10 14:39:46 +0000
+++ lib/lp/translations/doc/poimport.txt 2010-09-21 16:23:42 +0000
@@ -1,23 +1,20 @@
-= PO Imports =
+==========
+PO Imports
+==========
The tale of a PO template and a PO file and how they get imported into
Rosetta.
-
-== Test Setup ==
+Test Setup
+==========
Here are some imports we need to get this test running.
- >>> from canonical.launchpad.ftests import syncUpdate
- >>> from canonical.launchpad.interfaces import (
- ... ILanguageSet, ILaunchpadCelebrities, IPersonSet, IProductSet)
+ >>> from canonical.launchpad.interfaces.launchpad import (
+ ... ILaunchpadCelebrities)
+ >>> from lp.registry.interfaces.person import IPersonSet
>>> from lp.translations.interfaces.translationimportqueue import (
... ITranslationImportQueue, RosettaImportStatus)
- >>> from lp.registry.model.sourcepackagename import SourcePackageName
- >>> from lp.translations.model.potemplate import POTemplateSubset
- >>> from lp.translations.scripts.po_import import TranslationsImport
- >>> from lp.translations.scripts.import_queue_gardener import (
- ... ImportQueueGardener)
>>> import datetime
>>> import pytz
>>> UTC = pytz.timezone('UTC')
@@ -36,18 +33,15 @@
>>> login('carlos@xxxxxxxxxxxxx')
-== Importing a Template ==
+Importing a Template
+====================
Normal procedure is to import a template, followed by translations.
A template is created first. After that, imports are done using the
POFile.importFromQueue and POTemplate.importFromQueue methods.
- >>> from lp.registry.model.productrelease import ProductRelease
- >>> release = ProductRelease.get(3)
- >>> release.productseries.product.name
- u'firefox'
- >>> series = release.productseries
- >>> subset = POTemplateSubset(productseries=series)
+ >>> distroseries = factory.makeUbuntuDistroSeries()
+ >>> sourcepackagename = factory.makeSourcePackageName()
Here's the person who'll be doing the import.
@@ -56,10 +50,8 @@
And this is the POTemplate where the import will be done.
- >>> potemplate = subset.new(
- ... name='firefox',
- ... translation_domain='firefox',
- ... path='po/firefox.pot',
+ >>> potemplate = factory.makePOTemplate(
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
... owner=person)
>>> potemplate_id = potemplate.id
@@ -110,7 +102,8 @@
>>> translation_import_queue = getUtility(ITranslationImportQueue)
>>> entry = translation_import_queue.addOrUpdateEntry(
... potemplate.path, potemplate_contents, True, potemplate.owner,
- ... productseries=series, potemplate=potemplate)
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate)
The file data is stored in the Librarian, so we have to commit the
transaction to make sure it's stored properly.
@@ -143,13 +136,13 @@
A successful import is confirmed by email.
- >>> subject
- u'Translation template import - firefox in Mozilla Firefox trunk'
+ >>> print subject
+ Translation template import - ...
>>> print body
Hello Mark Shuttleworth,
<BLANKLINE>
On ..., you uploaded a translation
- template for firefox in Mozilla Firefox trunk in Launchpad.
+ template for ... in Launchpad.
<BLANKLINE>
The template has now been imported successfully.
<BLANKLINE>
@@ -174,7 +167,8 @@
u'test.c:13'
-=== Import Preconditions ===
+Import Preconditions
+====================
The API for POTemplate.importFromQueue demands a translation import
queue entry to import.
@@ -195,12 +189,7 @@
any other file would be an error.
>>> entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
- >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
- >>> other_product = getUtility(IProductSet).getByName('netapplet')
- >>> other_productseries = other_product.getSeries('trunk')
- >>> template_set = getUtility(IPOTemplateSet)
- >>> other_template = template_set.getPOTemplateByPathAndOrigin(
- ... 'po/netapplet.pot', productseries=other_productseries)
+ >>> other_template = factory.makePOTemplate()
>>> other_template.importFromQueue(entry)
Traceback (most recent call last):
...
@@ -208,7 +197,8 @@
to.
-== Importing a Translation ==
+Importing a Translation
+=======================
Now let's get a PO file to import.
@@ -217,8 +207,8 @@
By default, we got a safe path to prevent collisions with other IPOFile.
- >>> pofile.path
- u'po/firefox-cy.po'
+ >>> print pofile.path
+ generic-string...-cy.po
Let's override the default good path with one we know is the right one.
@@ -236,7 +226,8 @@
1
-=== Import With Errors ===
+Import With Errors
+------------------
Here are the contents of the file we'll be importing. It has some
validation errors.
@@ -285,7 +276,8 @@
>>> entry = translation_import_queue.addOrUpdateEntry(
... pofile.path, pofile_with_errors, True, person,
- ... productseries=series, potemplate=potemplate)
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate)
>>> transaction.commit()
The guess IPOFile should be the same we already had.
@@ -381,7 +373,7 @@
Hello Mark Shuttleworth,
<BLANKLINE>
On ..., you uploaded 5
- Welsh (cy) translations for firefox in Mozilla Firefox trunk in Launchpad.
+ Welsh (cy) translations for ... in Launchpad.
<BLANKLINE>
There were problems with 1 of these translations.
<BLANKLINE>
@@ -410,7 +402,8 @@
msgstr "blah %i"
-=== Import With Warnings ===
+Import With Warnings
+--------------------
The import may also succeed but produce syntax warnings. These need not
be tied to particular messages (they could be in the header, for
@@ -435,14 +428,14 @@
... msgid "a"
... msgstr "b"
... ''' % datetime.datetime.now(UTC).isoformat()
- >>> sumerian_pofile = potemplate.newPOFile('sux')
+ >>> eo_pofile = potemplate.newPOFile('eo')
>>> warning_entry = translation_import_queue.addOrUpdateEntry(
- ... 'sux.po', pofile_with_warning, False, potemplate.owner,
- ... productseries=series, potemplate=potemplate,
- ... pofile=sumerian_pofile)
+ ... 'eo.po', pofile_with_warning, False, potemplate.owner,
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate, pofile=eo_pofile)
>>> transaction.commit()
>>> warning_entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
- >>> (subject, message) = sumerian_pofile.importFromQueue(warning_entry)
+ >>> (subject, message) = eo_pofile.importFromQueue(warning_entry)
The warning is noted in the confirmation email. Note that this
particular warning condition is recognized fairly late, so the line
@@ -473,7 +466,8 @@
>>> warning_entry.setStatus(RosettaImportStatus.DELETED, rosetta_experts)
-=== Import Without Errors ===
+Import Without Errors
+---------------------
Now, let's import one without errors.
@@ -495,7 +489,8 @@
... ''' % datetime.datetime.now(UTC).isoformat()
>>> entry = translation_import_queue.addOrUpdateEntry(
... pofile.path, pofile_without_errors, True, rosetta_experts,
- ... productseries=series, potemplate=potemplate)
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate)
>>> transaction.commit()
The new upload clears the entry's error_output.
@@ -580,7 +575,8 @@
u'helpful@xxxxxxxxxxx'
-=== Import Preconditions ===
+Import Preconditions
+====================
The API for POFile.importFromQueue demands a translation import queue
entry to import.
@@ -617,296 +613,28 @@
to.
-== Cron Scripts ==
-
-We tested already that the functionality works. Now it's time to know
-if the cronscript has any problem.
-
-First, we are going to reactivate the entries that were already
-imported or failed. Note that we'll only reactivate the entries we use
-in this test; We don't touch entries that were in the queue previously.
-
- >>> for entry in translation_import_queue:
- ... if (entry.status == RosettaImportStatus.IMPORTED or
- ... entry.status == RosettaImportStatus.FAILED) and (
- ... entry.productseries == series):
- ... entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
- ... syncUpdate(entry)
- >>> transaction.commit()
-
-And run the import script.
-
- >>> import email
- >>> from lp.services.mail import stub
- >>> process = TranslationsImport('poimport', test_args=[])
- >>> process.logger = FakeLogger()
- >>> process.main()
- DEBUG Starting the import process.
- INFO Importing: Template "firefox" in Mozilla Firefox trunk
- INFO Importing: Welsh (cy) ... of firefox in Mozilla Firefox trunk
- INFO Importing: Welsh (cy) ... of firefox in Mozilla Firefox trunk
- INFO Import requests completed.
- DEBUG Finished the import process.
-
-The import script also generates an email similar to the ones we saw
-composed before, but also sends it.
-
- >>> len(stub.test_emails)
- 1
-
- >>> from_addr, to_addrs, raw_message = stub.test_emails.pop()
- >>> msg = email.message_from_string(raw_message)
- >>> msg["Subject"]
- 'Translation template import - firefox in Mozilla Firefox trunk'
-
- >>> print msg.get_payload(decode=True)
- Hello Mark Shuttleworth,
- <BLANKLINE>
- On ..., you uploaded a translation
- template for firefox in Mozilla Firefox trunk in Launchpad.
- <BLANKLINE>
- The template has now been imported successfully.
- <BLANKLINE>
- Thank you,
- <BLANKLINE>
- The Launchpad team
-
-Now the queue gardener runs. This can happen anytime, since it's
-asynchronous to the po-import script. The script tries to approve any
-entries that have not been approved, but look like they could be,
-without human intervention. This involves a bit of guesswork about what
-the imported file is and where it belongs. It similarly blocks entries
-that it thinks should be blocked, and also purges deleted or completed
-entries from the queue. Running at this point, all it does is purge the
-two hand-approved Welsh translations that have just been imported.
-
- >>> import logging
- >>> from lp.testing.logger import MockLogger
- >>> process = ImportQueueGardener('approver', test_args=[])
- >>> process.logger = MockLogger()
- >>> process.logger.setLevel(logging.INFO)
- >>> process.main()
- log> Removed 2 entries from the queue.
- >>> transaction.commit()
-
-If users upload two versions of the same file, they are imported in the
-order in which they were uploaded.
-
- >>> first_pofile_content = r'''
- ... msgid ""
- ... msgstr ""
- ... "PO-Revision-Date: 2005-06-04 20:41+0100\n"
- ... "Last-Translator: Foo <no-priv@xxxxxxxxxxxxx>\n"
- ... "Content-Type: text/plain; charset=UTF-8\n"
- ... "X-Rosetta-Export-Date: %s\n"
- ...
- ... msgid "Foo %%s"
- ... msgstr "Bar"
- ...
- ... msgid "translator-credits"
- ... msgstr "The world will never know."
- ... ''' % datetime.datetime.now(UTC).isoformat()
-
- >>> second_pofile_content = r'''
- ... msgid ""
- ... msgstr ""
- ... "PO-Revision-Date: 2005-06-04 21:41+0100\n"
- ... "Last-Translator: Jordi Mallach <jordi@xxxxxxxxxxxxx>\n"
- ... "Content-Type: text/plain; charset=UTF-8\n"
- ... "X-Rosetta-Export-Date: %s\n"
- ...
- ... msgid "Foo %%s"
- ... msgstr "Bars"
- ...
- ... msgid "translator-credits"
- ... msgstr "I'd like to thank John, Kathy, my pot plants, and all the..."
- ... ''' % datetime.datetime.now(UTC).isoformat()
-
-We flush the entry contents.
-
- >>> for entry in translation_import_queue:
- ... translation_import_queue.remove(entry)
- >>> translation_import_queue.countEntries()
- 0
-
-Attach the first version of the file.
-
- >>> entry = translation_import_queue.addOrUpdateEntry(
- ... pofile.path, first_pofile_content, False, rosetta_experts,
- ... sourcepackagename=pofile.potemplate.sourcepackagename,
- ... distroseries=pofile.potemplate.distroseries,
- ... productseries=pofile.potemplate.productseries)
- >>> transaction.commit()
-
-It's in the queue now.
-
- >>> translation_import_queue.countEntries()
- 1
-
-For the second version, we need a new importer, in this case, Jordi.
-
- >>> jordi = person_set.getByName('jordi')
-
-Attach the second version of the file.
-
- >>> entry = translation_import_queue.addOrUpdateEntry(
- ... pofile.path, second_pofile_content, False, jordi,
- ... sourcepackagename=pofile.potemplate.sourcepackagename,
- ... distroseries=pofile.potemplate.distroseries,
- ... productseries=pofile.potemplate.productseries)
- >>> transaction.commit()
-
-It's in the queue now.
-
- >>> translation_import_queue.countEntries()
- 2
- >>> print entry.status.name
- NEEDS_REVIEW
-
-The queue gardener runs again. This time it sees the two submitted
-translations and approves them for import based on some heuristic
-intelligence.
-
- >>> process = ImportQueueGardener('approver', test_args=[])
- >>> process.logger = MockLogger()
- >>> process.logger.setLevel(logging.INFO)
- >>> process.main()
- log> The automatic approval system approved some entries.
- >>> print entry.status.name
- APPROVED
- >>> syncUpdate(entry)
-
-Now that these submissions have been approved, the next run of the
-import script picks them up and processes them.
-
- >>> process = TranslationsImport('poimport', test_args=[])
- >>> process.logger = FakeLogger()
- >>> process.main()
- DEBUG Starting the import process.
- INFO Importing: Welsh (cy) ... of firefox in Mozilla Firefox trunk
- INFO Importing: Welsh (cy) ... of firefox in Mozilla Firefox trunk
- INFO Import requests completed.
- DEBUG Finished the import process.
-
- >>> print entry.status.name
- IMPORTED
- >>> syncUpdate(entry)
-
-And there are no more entries to import
-
- >>> translation_import_queue.getFirstEntryToImport() is None
- True
-
-We've imported a new translation for "Foo %s."
-
- >>> welsh = getUtility(ILanguageSet).getLanguageByCode('cy')
- >>> foos = potemplate['Foo %s'].getLocalTranslationMessages(
- ... potemplate, welsh)
- >>> sorted([foo.msgstr0.translation for foo in foos])
- [u'Bar', u'Bars', u'blah %i']
-
-Since this last upload was not the upstream one, however, its credits
-message translations were ignored.
-
- >>> message = get_pofile_translation_message(
- ... pofile, u'translator-credits')
- >>> message.msgstr0.translation
- u'helpful@xxxxxxxxxxx'
- >>> list(potemplate['translator-credits'].getLocalTranslationMessages(
- ... potemplate, welsh))
- []
-
-Imports so far have been associated with a product series. We can also
-submit translations for a distroseries.
-
- >>> from lp.registry.interfaces.distribution import (
- ... IDistributionSet)
- >>> ubuntu = getUtility(IDistributionSet).getByName('ubuntu')
- >>> warty = ubuntu.getSeries('warty')
- >>> print warty.name
- warty
- >>> firefox_name = SourcePackageName.byName('mozilla-firefox')
- >>> subset = POTemplateSubset(sourcepackagename=firefox_name,
- ... distroseries=warty)
- >>> potemplate = subset.new(
- ... name='firefox-warty',
- ... translation_domain='firefox-warty',
- ... path='po/firefox.pot',
- ... owner=person)
-
-As it happens, the administrator has blocked imports to warty, e.g.
-because an in-database update of its translations has been scheduled
-and we don't want interference from queued imports while that happens.
-It doesn't really matter whether entries still get auto-approved, but
-we can't accept new translation imports just now.
-
- >>> warty.defer_translation_imports = True
- >>> syncUpdate(warty)
-
-Nevertheless, someone submits an import request for warty, not knowing
-or caring that imports are deferred. The entry still gets approved as
-normal:
-
- >>> entry = translation_import_queue.addOrUpdateEntry(
- ... potemplate.path, potemplate_contents, True, potemplate.owner,
- ... sourcepackagename=firefox_name, distroseries=warty,
- ... potemplate=potemplate)
- >>> entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
- >>> syncUpdate(entry)
- >>> transaction.commit()
-
-Since imports for warty are suspended, and the only entry we happen to
-have waiting right now is for warty, the queue has no importable
-entries for us.
-
- >>> warty.getFirstEntryToImport() is None
- True
-
-So if we try to import now, nothing happens. Our request remains on the
-queue, but doesn't become a candidate for processing until warty
-imports are resumed.
-
- >>> process = TranslationsImport('poimport', test_args=[])
- >>> process.logger = FakeLogger()
- >>> process.main()
- DEBUG Starting the import process.
- INFO No requests pending.
-
- >>> print entry.status.name
- APPROVED
-
-Once imports are allowed again, the import is done after all.
-
- >>> warty.defer_translation_imports = False
- >>> syncUpdate(warty)
- >>> (subject, body) = potemplate.importFromQueue(entry, FakeLogger())
-
- >>> print entry.status.name
- IMPORTED
-
-
-== Plural forms handling ==
+Plural forms handling
+=====================
Apart from the basic plural form handling, which is documented above as
part of the import process, there are some peculiarities with importing
plural forms we want documented as well.
-For a language such as Divehi, which has no plural forms defined, we
+For a language that has no plural forms defined, we
default to two plural forms (the most common value for the number of
plural forms).
- >>> divehi = getUtility(ILanguageSet)['dv']
- >>> print divehi.pluralforms
+ >>> language = factory.makeLanguage()
+ >>> print language.pluralforms
None
- >>> firefox = getUtility(IProductSet).getByName('firefox')
- >>> firefox_trunk = firefox.getSeries('trunk')
- >>> firefox_potemplate = firefox_trunk.getPOTemplate('firefox')
- >>> firefox_dv = firefox_potemplate.newPOFile(divehi.code)
- >>> firefox_dv.plural_forms
+ >>> potemplate = factory.makePOTemplate(
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename)
+ >>> pofile = potemplate.newPOFile(language.code)
+ >>> pofile.plural_forms
2
-We'll import a POFile with 3 plural forms into Divehi POFile:
+We'll import a POFile with 3 plural forms into this POFile:
>>> pofile_with_plurals = r'''
... msgid ""
@@ -925,16 +653,18 @@
... msgstr[2] "Third form %%d"
... ''' % datetime.datetime.now(UTC).isoformat()
-We now import this POFile as Divehi translation of Firefox trunk:
+We now import this POFile as this language's translation for the soure
+package:
>>> entry = translation_import_queue.addOrUpdateEntry(
- ... firefox_dv.path, pofile_with_plurals, True, person,
- ... productseries=firefox_trunk, potemplate=firefox_potemplate)
+ ... pofile.path, pofile_with_plurals, True, person,
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate)
>>> # Allow Librarian to see the change.
>>> transaction.commit()
- >>> entry.pofile = firefox_dv
+ >>> entry.pofile = pofile
>>> entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
- >>> (subject, body) = firefox_dv.importFromQueue(entry, FakeLogger())
+ >>> (subject, body) = pofile.importFromQueue(entry, FakeLogger())
>>> flush_database_updates()
>>> print entry.status.name
IMPORTED
@@ -943,21 +673,22 @@
translations (which is a default when the language has no plural forms
specified):
- >>> potmsgset_plural = firefox_potemplate.getPOTMsgSetByMsgIDText(
+ >>> potmsgset_plural = potemplate.getPOTMsgSetByMsgIDText(
... u'Singular %d', u'Plural %d')
- >>> current_dv = potmsgset_plural.getCurrentTranslationMessage(
- ... firefox_potemplate, divehi)
- >>> current_dv.translations
+ >>> current = potmsgset_plural.getCurrentTranslationMessage(
+ ... potemplate, language)
+ >>> current.translations
[u'First form %d', u'Second form %d']
However, even the third form will be imported into database (this is
useful for when we finally define the number of plural forms for the
language, we should not have to reimport all translations):
- >>> current_dv.msgstr2.translation
+ >>> current.msgstr2.translation
u'Third form %d'
-== Upstream import notifications ==
+Upstream import notifications
+=============================
Add an upstream POFile import (i.e. from a package or bzr branch),
approve and import it.
@@ -974,12 +705,12 @@
... msgid "foo"
... msgstr "blah"
... '''
- >>> pofile = factory.makePOFile('sr')
+ >>> pofile = factory.makePOFile('sr', potemplate=potemplate)
>>> from_upstream = True
>>> entry = translation_import_queue.addOrUpdateEntry(
... pofile.path, pofile_contents, from_upstream, person,
- ... productseries=pofile.potemplate.productseries,
- ... potemplate=pofile.potemplate, pofile=pofile)
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate, pofile=pofile)
>>> transaction.commit()
>>> entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
>>> (subject, message) = pofile.importFromQueue(entry)
@@ -997,8 +728,8 @@
>>> pofile_contents = pofile_contents[:-2]
>>> entry = translation_import_queue.addOrUpdateEntry(
... pofile.path, pofile_contents, from_upstream, person,
- ... productseries=pofile.potemplate.productseries,
- ... potemplate=pofile.potemplate, pofile=pofile)
+ ... distroseries=distroseries, sourcepackagename=sourcepackagename,
+ ... potemplate=potemplate, pofile=pofile)
>>> transaction.commit()
>>> entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
>>> (subject, message) = pofile.importFromQueue(entry)
@@ -1010,44 +741,3 @@
>>> subject
u'Import problem - Serbian (sr) - ...'
-
-No Contact Address
-------------------
-
-Not every user has a valid email address. For instance, Kermit the
-Hermit has none at the moment.
-
- >>> from canonical.launchpad.interfaces.emailaddress import (
- ... EmailAddressStatus)
- >>> from canonical.launchpad.helpers import get_contact_email_addresses
- >>> hermit = factory.makePerson(
- ... name='hermit', email_address_status=EmailAddressStatus.OLD)
-
- >>> len(get_contact_email_addresses(hermit))
- 0
-
-Kermit uploads a translation, which gets approved.
-
- >>> pofile = factory.makePOFile('lo')
-
- >>> entry = translation_import_queue.addOrUpdateEntry(
- ... 'lo.po', 'Invalid content', True, hermit,
- ... pofile=pofile, potemplate=pofile.potemplate,
- ... productseries=pofile.potemplate.productseries)
- >>> entry.setStatus(RosettaImportStatus.APPROVED, rosetta_experts)
- >>> transaction.commit()
-
-The import fails. The importer would like to send Kermit an email about
-this, but is unable to. This is unfortunate, but does not faze the
-importer. It completes normally.
-
- >>> process = TranslationsImport('poimport', test_args=[])
- >>> process.logger = FakeLogger()
- >>> process.main()
- DEBUG Starting the import process.
- INFO Importing: Lao ...
- INFO Import requests completed.
- DEBUG Finished the import process.
-
- >>> print entry.status.name
- FAILED
