launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #00843
lp:~henninge/launchpad/devel-bug-597539-translationmessage-pofile into lp:launchpad/devel
Henning Eggers has proposed merging lp:~henninge/launchpad/devel-bug-597539-translationmessage-pofile into lp:launchpad/devel.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
= Bug 597539 =
With the advent of message sharing, the "pofile" attribute of a TranslationMessage became obsolete. TranslationMessage now has a "language" attribute which can be used to find the right POFile if you also know the POTemplate. Since a TranslationMessage can now be shared by multiple POFiles, a direct link makes no sense now.
A lot of code was still using this link, though, since the column in the database still exists and holds values. Since TranslationMessage was still setting pofile upon creation, it most likely pointed to the first POFile that this message was translated in and that mostly worked.
This branch cleans this mess up by hunting down all these uses of the pofile attribute and tries to find an equivalent replacement. Please excuse the diff size, it contains file deletions an repetitive changes in templates and such.
== Proposed fix ==
For each occurrence, one of these fixes was used:
- Use the new "language" attribute if the "pofile" was only used to get to the language. Luckily there were a lot of those cases.
- In view code it was mostly possible to use "browser_pofile" which caches the POFile that the message was currently being viewed in. Some extra code had to be added to make sure it is always set.
- Sometimes the POFile was actually known and there was no reason to use TranslationMessage.pofile.
- Sometimes getOnePOFile will do which just gets any POFile associated with the translation message.
- A new query to find all TranslationMessages for a POFile was introduced.
- Some scripts did not receive nice treatment, one was crippled, the other one plain deleted. ;)
== Implementation details ==
Notes for some specific files.
=== modified file 'lib/lp/translations/browser/tests/translationmessage-views.txt'
Using POFile statistics to show that a form submission succeeded is overkilll, so I removed them.
=== modified file 'lib/lp/translations/browser/translationmessage.py'
I introduced a new parameter 'local_to_pofile' which is passed in from the top to mark local translations explicitly instead of deriving that from the "pofile" attribute. The last chunk is the core of this change.
=== modified file 'lib/lp/translations/doc/gettext-check-messages.txt'
=== modified file 'lib/lp/translations/scripts/gettext_check_messages.py'
This script operates purely on the TranslationMessage table with no relation to a POFile. Finding an imported message is not possible without that information so it had be ridded of that functionality. Obviously this also affected the test. The change has just been commented out because that script will be revisited later anyway in our current feature work and maybe we come up with a better solution.
=== modified file 'lib/lp/translations/tests/test_doc.py'
=== removed file 'lib/lp/translations/doc/remove-upstream-translations-script.txt'
=== removed file 'scripts/rosetta/remove-upstream-translations.py'
This script is not needed anymore so why bother fixing it? Yeah for code deletion!
=== modified file 'lib/lp/translations/interfaces/pofile.py'
=== modified file 'lib/lp/translations/model/pofile.py'
=== modified file 'lib/lp/translations/tests/test_pofile.py'
As POFile.translation_messages cannot be a simple join anymore, I introduced getTranslationMessages to retrieve the messages. This could also be used (with an extra condition) to replace another direct SQL query on the pofile column. Complete with test.
=== modified file 'lib/lp/translations/interfaces/translationmessage.py'
=== modified file 'lib/lp/translations/model/translationmessage.py'
The method ensureBrowserPOFile is simple but had to be in the model class because otherwise assigment to browser_pofile would not be possible.
=== modified file 'lib/lp/translations/model/potmsgset.py'
This is at the heart of this branch because it stops TranslationMessage.pofile from being assigned a value during its creation.
== Test ==
Just run all translation tests, it takes about half an hour.
bin/test -vvcm lp.translations
== QA ==
When this branch is on staging, the pofile column needs to b NULL'ed.
ALTER TABLE translationmessage DISABLE TRIGGER ALL;
UPDATE translationmessage SET pofile=NULL;
ALTER TABLE translationmessage ENABLE TRIGGER ALL;
Then various +translate pages need to be opened and also some imports would be great and similar. Check that
- nothing oopses
- the pofile column remains NULL'ed.
--
https://code.launchpad.net/~henninge/launchpad/devel-bug-597539-translationmessage-pofile/+merge/34441
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~henninge/launchpad/devel-bug-597539-translationmessage-pofile into lp:launchpad/devel.
=== modified file 'lib/lp/translations/browser/tests/translationmessage-views.txt'
--- lib/lp/translations/browser/tests/translationmessage-views.txt 2010-07-23 09:32:52 +0000
+++ lib/lp/translations/browser/tests/translationmessage-views.txt 2010-09-02 16:31:56 +0000
@@ -37,7 +37,7 @@
Here we can see that it's lacking that information.
- >>> print translationmessage_page_view.context.pofile.language.pluralforms
+ >>> print translationmessage_page_view.context.language.pluralforms
None
And the view class detects it correctly.
@@ -61,7 +61,7 @@
We have the plural form information for this language.
- >>> print translationmessage_page_view.context.pofile.language.pluralforms
+ >>> print translationmessage_page_view.context.language.pluralforms
2
And thus, the view class should know that it doesn't lacks the plural forms
@@ -228,25 +228,7 @@
== Submitting translations ==
-It's time to check the submission of translations and the IPOFile statistics
-update.
-
-But first, let's see current values.
-
- >>> translationmessage = TranslationMessage.get(1)
- >>> pofile = translationmessage.pofile
- >>> pofile.updateStatistics()
- (7, 0, 1, 2)
- >>> pofile.currentCount()
- 7
- >>> pofile.updatesCount()
- 0
- >>> pofile.rosettaCount()
- 1
- >>> pofile.unreviewedCount()
- 2
-
-Now we do the submit.
+Let's submit a new translation through the view.
>>> server_url = '/'.join(
... [canonical_url(translationmessage), '+translate'])
@@ -287,17 +269,7 @@
>>> translationmessage_page_view.initialize()
>>> transaction.commit()
-This time we didn't get any problem with the submission, and we can see that
-statistics were updated accordingly.
-
- >>> pofile.currentCount()
- 6
- >>> pofile.updatesCount()
- 1
- >>> pofile.rosettaCount()
- 2
- >>> pofile.unreviewedCount()
- 2
+This time we didn't get any problem with the submission.
Now, let's see how the system prevents a submission that has a timestamp older
than when last current translation was submitted.
@@ -715,7 +687,6 @@
>>> def submit_translation(
... translationmessage, translation, force_suggestion=False):
... global year_tick
- ... pofile = translationmessage.pofile
... potmsgset = translationmessage.potmsgset
... server_url = '/'.join(
... [canonical_url(translationmessage), '+translate'])
@@ -723,7 +694,8 @@
... datetime.now(UTC).strftime('-%m-%dT%H:%M:%S+00:00'))
... year_tick += 1
... msgset_id = 'msgset_' + str(potmsgset.id)
- ... msgset_id_lang = msgset_id + '_' + pofile.language.code
+ ... language_code = translationmessage.language.code
+ ... msgset_id_lang = msgset_id + '_' + language_code
... form = {
... 'lock_timestamp': now,
... 'alt': None,
=== modified file 'lib/lp/translations/browser/translationmessage.py'
--- lib/lp/translations/browser/translationmessage.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/browser/translationmessage.py 2010-09-02 16:31:56 +0000
@@ -926,7 +926,7 @@
self.force_suggestion = force_suggestion
self.force_diverge = force_diverge
self.user_is_official_translator = (
- current_translation_message.pofile.canEditTranslations(self.user))
+ self.pofile.canEditTranslations(self.user))
self.form_is_writeable = form_is_writeable
if self.context.is_imported:
# The imported translation matches the current one.
@@ -1033,28 +1033,34 @@
# Imported one matches the current one.
imported_submission = None
elif self.imported_translationmessage is not None:
+ self.imported_translationmessage.ensureBrowserPOFile()
+ pofile = self.imported_translationmessage.browser_pofile
imported_submission = (
convert_translationmessage_to_submission(
message=self.imported_translationmessage,
current_message=self.context,
plural_form=index,
- pofile=self.imported_translationmessage.pofile,
+ pofile=pofile,
legal_warning_needed=False,
is_empty=False,
- packaged=True))
+ packaged=True,
+ local_to_pofile=True))
else:
imported_submission = None
if (self.context.potemplate is not None and
self.shared_translationmessage is not None):
+ self.shared_translationmessage.ensureBrowserPOFile()
+ pofile = self.shared_translationmessage.browser_pofile
shared_submission = (
convert_translationmessage_to_submission(
message=self.shared_translationmessage,
current_message=self.context,
plural_form=index,
- pofile=self.shared_translationmessage.pofile,
+ pofile=pofile,
legal_warning_needed=False,
- is_empty=False))
+ is_empty=False,
+ local_to_pofile=True))
else:
shared_submission = None
@@ -1249,7 +1255,7 @@
self.seen_translations.add(imported.translations[index])
local_suggestions = (
self._buildTranslationMessageSuggestions(
- 'Suggestions', local, index))
+ 'Suggestions', local, index, local_to_pofile=True))
externally_used_suggestions = (
self._buildTranslationMessageSuggestions(
'Used in', externally_used, index, legal_warning=True))
@@ -1272,7 +1278,8 @@
len(alternate_language_suggestions.submissions))
def _buildTranslationMessageSuggestions(self, title, suggestions, index,
- legal_warning=False):
+ legal_warning=False,
+ local_to_pofile=False):
"""Build filtered list of submissions to be shown in the view.
`title` is the title for the suggestion type, `suggestions` is
@@ -1282,7 +1289,8 @@
title, self.context,
suggestions[:self.max_entries],
self.user_is_official_translator, self.form_is_writeable,
- index, self.seen_translations, legal_warning=legal_warning)
+ index, self.seen_translations, legal_warning=legal_warning,
+ local_to_pofile=local_to_pofile)
self.seen_translations = iterable_submissions.seen_translations
return iterable_submissions
@@ -1524,10 +1532,11 @@
def __init__(self, title, translation, submissions,
user_is_official_translator, form_is_writeable,
- plural_form, seen_translations=None, legal_warning=False):
+ plural_form, seen_translations=None, legal_warning=False,
+ local_to_pofile=False):
self.title = title
self.potmsgset = translation.potmsgset
- self.pofile = translation.pofile
+ self.pofile = translation.browser_pofile
self.user_is_official_translator = user_is_official_translator
self.form_is_writeable = form_is_writeable
self.submissions = []
@@ -1557,7 +1566,8 @@
plural_form,
self.pofile,
legal_warning,
- is_empty=False))
+ is_empty=False,
+ local_to_pofile=local_to_pofile))
self.seen_translations = seen_translations
@@ -1566,7 +1576,7 @@
def convert_translationmessage_to_submission(
message, current_message, plural_form, pofile, legal_warning_needed,
- is_empty=False, packaged=False):
+ is_empty=False, packaged=False, local_to_pofile=False):
"""Turn a TranslationMessage to an object used for rendering a submission.
:param message: A TranslationMessage.
@@ -1578,10 +1588,10 @@
submission = Submission()
submission.translationmessage = message
- for attribute in ['id', 'language', 'potmsgset', 'pofile',
- 'date_created']:
+ for attribute in ['id', 'language', 'potmsgset', 'date_created']:
setattr(submission, attribute, getattr(message, attribute))
+ submission.pofile = message.browser_pofile
submission.person = message.submitter
submission.is_empty = is_empty
@@ -1589,7 +1599,7 @@
submission.suggestion_text = text_to_html(
message.translations[plural_form],
message.potmsgset.flags)
- submission.is_local_to_pofile = (message.pofile == pofile)
+ submission.is_local_to_pofile = local_to_pofile
submission.legal_warning = legal_warning_needed and (
message.origin == RosettaTranslationOrigin.SCM)
submission.suggestion_html_id = (
=== modified file 'lib/lp/translations/doc/gettext-check-messages.txt'
--- lib/lp/translations/doc/gettext-check-messages.txt 2010-02-19 16:54:42 +0000
+++ lib/lp/translations/doc/gettext-check-messages.txt 2010-09-02 16:31:56 +0000
@@ -118,14 +118,15 @@
DEBUG Checking messages matching: id=...
DEBUG Checking message ...
INFO ... (current): format specifications ... are not the same
- INFO ...: unmasked ...
+ DEBUG Commit point.
+ COMMIT
DEBUG Commit point.
COMMIT
INFO Done.
INFO Messages checked: 1
INFO Validation errors: 1
INFO Messages disabled: 1
- INFO Messages unmasked: 1
+ INFO Messages unmasked: 0
INFO Commit points: ...
The failed message is demoted to a mere suggestion.
@@ -133,15 +134,15 @@
>>> current_message.is_current
False
-In this case, there was a perfectly good imported message that was being
-masked by the invalid current translation. So in addition to disabling
-the bad current message, the script activates the good imported one.
+Unmasking of imported messages is disabled in the script. That's why
+although there was a perfectly good imported message that was being
+masked by the invalid current translation, it will not be activated.
>>> imported_message.is_current
- True
+ False
If the imported message is also bad, this is reported and the imported
-message is not activated.
+message is not activated. But unmasking is disabled anyway.
>>> imported_message.is_current = False
>>> current_message.is_current = True
@@ -157,7 +158,7 @@
COMMIT
INFO Done.
INFO Messages checked: 1
- INFO Validation errors: 2
+ INFO Validation errors: 1
INFO Messages disabled: 1
INFO Messages unmasked: 0
INFO Commit points: ...
@@ -188,7 +189,7 @@
COMMIT
INFO Done.
INFO Messages checked: 2
- INFO Validation errors: 3
+ INFO Validation errors: 2
INFO Messages disabled: 0
INFO Messages unmasked: 0
INFO Commit points: 2
@@ -223,7 +224,7 @@
ABORT
INFO Done.
INFO Messages checked: 1
- INFO Validation errors: 2
+ INFO Validation errors: 1
INFO Messages disabled: 1
INFO Messages unmasked: 0
INFO Commit points: 2
@@ -251,7 +252,7 @@
COMMIT
INFO Done.
INFO Messages checked: 2
- INFO Validation errors: 3
+ INFO Validation errors: 2
INFO Messages disabled: 0
INFO Messages unmasked: 0
INFO Commit points: 3
=== modified file 'lib/lp/translations/doc/pofile.txt'
--- lib/lp/translations/doc/pofile.txt 2010-08-06 07:40:52 +0000
+++ lib/lp/translations/doc/pofile.txt 2010-09-02 16:31:56 +0000
@@ -773,7 +773,7 @@
Found %i invalid file.
>>> print pofile_sr.language.code
sr
- >>> print translation_message.pofile.language.code
+ >>> print translation_message.language.code
sr
This entry is in fact one that is not used anymore, that means, its sequence
=== modified file 'lib/lp/translations/doc/potmsgset.txt'
--- lib/lp/translations/doc/potmsgset.txt 2010-08-06 06:51:37 +0000
+++ lib/lp/translations/doc/potmsgset.txt 2010-09-02 16:31:56 +0000
@@ -472,7 +472,7 @@
A TranslationMessage knows what language it is in.
- >>> print pt_BR_dummy_current.pofile.language.code
+ >>> print pt_BR_dummy_current.language.code
pt_BR
Using another dummy pofile we'll get a POTMsgset that's not a singular
@@ -621,7 +621,8 @@
>>> translationmessage = TranslationMessage.get(2)
>>> potmsgset = translationmessage.potmsgset
- >>> pofile = translationmessage.pofile
+ >>> from lp.translations.model.pofile import POFile
+ >>> pofile = POFile.get(1)
>>> translationmessage.date_reviewed.isoformat()
'2005-04-07T13:19:17.601068+00:00'
>>> potmsgset.isTranslationNewerThan(pofile,
@@ -660,8 +661,9 @@
... usage.append('Upstream')
... if not usage:
... usage.append('None')
+ ... pofile = suggestion.getOnePOFile()
... lines.append('%s: %s (%s)' % (
- ... suggestion.pofile.title,
+ ... pofile.title,
... suggestion.translations[0],
... ' & '.join(usage)))
... for line in sorted(lines):
=== removed file 'lib/lp/translations/doc/remove-upstream-translations-script.txt'
--- lib/lp/translations/doc/remove-upstream-translations-script.txt 2009-07-01 20:45:39 +0000
+++ lib/lp/translations/doc/remove-upstream-translations-script.txt 1970-01-01 00:00:00 +0000
@@ -1,142 +0,0 @@
-This test checks the admin script 'remove-upstream-translations-script.py'.
-That script allows us to remove all translations comming from upstream
-to fix broken data that would pollute our suggestions database.
-
-It was developed to fix bug #32610.
-
-First we do some needed initializations.
-
- >>> from canonical.launchpad.ftests.harness import LaunchpadTestSetup
- >>> LaunchpadTestSetup(dbuser='rosettaadmin').setUp()
- >>> import subprocess, sys
-
-Now, we must be sure that we don't break if the user doesn't give us the
-required arguments.
-
- >>> process = subprocess.Popen([
- ... sys.executable,
- ... 'scripts/rosetta/remove-upstream-translations.py',
- ... '-p', 'evolution', '-vvv'],
- ... stdin=subprocess.PIPE, stdout=subprocess.PIPE,
- ... stderr=subprocess.STDOUT
- ... )
- >>> (output, empty) = process.communicate()
- >>> print output
- WARNING Nothing to do. Exiting...
- <BLANKLINE>
-
-We can remove all translations from a concrete potemplate.
-
- >>> process = subprocess.Popen([
- ... sys.executable,
- ... 'scripts/rosetta/remove-upstream-translations.py',
- ... '-d', 'ubuntu', '-r', 'hoary', '-n', 'evolution', '-vvv'],
- ... stdin=subprocess.PIPE, stdout=subprocess.PIPE,
- ... stderr=subprocess.STDOUT
- ... )
- >>> (output, empty) = process.communicate()
- >>> print output
- DEBUG Processing Spanish (es) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
- DEBUG Removed 8 submissions
- DEBUG Processing Japanese (ja) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
- DEBUG Removed 0 submissions
- DEBUG Processing Xhosa (xh) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
- DEBUG Removed 0 submissions
- DEBUG Processing Spanish (es) translation of man in Ubuntu Hoary package "evolution"...
- DEBUG Removed 0 submissions
- DEBUG Processing Spanish (es) translation of disabled-template in Ubuntu Hoary package "evolution"...
- DEBUG Removed 0 submissions
- DEBUG Removed 8 submissions in total.
- <BLANKLINE>
-
-We need to force a DB reset because the changes are done from an external
-script and the test system is not able to detect the database changes.
-
- >>> LaunchpadTestSetup().force_dirty_database()
- >>> LaunchpadTestSetup().tearDown()
-
-Or with a concrete language.
-
- >>> LaunchpadTestSetup(dbuser='rosettaadmin').setUp()
-
- >>> process = subprocess.Popen([
- ... sys.executable,
- ... 'scripts/rosetta/remove-upstream-translations.py',
- ... '-d', 'ubuntu', '-r', 'hoary', '-n', 'evolution', '-l', 'es',
- ... '-vvv'],
- ... stdin=subprocess.PIPE, stdout=subprocess.PIPE,
- ... stderr=subprocess.STDOUT
- ... )
- >>> (output, empty) = process.communicate()
- >>> print output
- DEBUG Processing Spanish (es) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
- DEBUG Removed 8 submissions
- DEBUG Processing Spanish (es) translation of man in Ubuntu Hoary package "evolution"...
- DEBUG Removed 0 submissions
- DEBUG Processing Spanish (es) translation of disabled-template in Ubuntu Hoary package "evolution"...
- DEBUG Removed 0 submissions
- DEBUG Removed 8 submissions in total.
- <BLANKLINE>
-
-We need to force a DB reset because the changes are done from an external
-script and the test system is not able to detect the database changes.
-
- >>> LaunchpadTestSetup().force_dirty_database()
- >>> LaunchpadTestSetup().tearDown()
-
-Or even with a concrete template name.
-
- >>> LaunchpadTestSetup(dbuser='rosettaadmin').setUp()
-
- >>> process = subprocess.Popen([
- ... sys.executable,
- ... 'scripts/rosetta/remove-upstream-translations.py',
- ... '-d', 'ubuntu', '-r', 'hoary', '-n', 'evolution', '-t', 'evolution-2.2',
- ... '-vvv'],
- ... stdin=subprocess.PIPE, stdout=subprocess.PIPE,
- ... stderr=subprocess.STDOUT
- ... )
- >>> (output, empty) = process.communicate()
- >>> print output
- DEBUG Processing Spanish (es) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
- DEBUG Removed 8 submissions
- DEBUG Processing Japanese (ja) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
- DEBUG Removed 0 submissions
- DEBUG Processing Xhosa (xh) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
- DEBUG Removed 0 submissions
- DEBUG Removed 8 submissions in total.
- <BLANKLINE>
-
-We need to force a DB reset because the changes are done from an external
-script and the test system is not able to detect the database changes.
-
- >>> LaunchpadTestSetup().force_dirty_database()
- >>> LaunchpadTestSetup().tearDown()
-
-And it also works for products!
-
- >>> LaunchpadTestSetup(dbuser='rosettaadmin').setUp()
-
- >>> import subprocess, sys
- >>> process = subprocess.Popen([
- ... sys.executable,
- ... 'scripts/rosetta/remove-upstream-translations.py',
- ... '-p', 'evolution', '-s', 'trunk', '-vvv'],
- ... stdin=subprocess.PIPE, stdout=subprocess.PIPE,
- ... stderr=subprocess.STDOUT
- ... )
- >>> (output, empty) = process.communicate()
- >>> print output
- DEBUG Processing Spanish (es) translation of evolution-2.2 in Evolution trunk...
- DEBUG Removed 9 submissions
- DEBUG Processing Portuguese (Brazil) (pt_BR) translation of evolution-2.2-test in Evolution trunk...
- DEBUG Removed 5 submissions
- DEBUG Removed 14 submissions in total.
- <BLANKLINE>
-
-We need to force a DB reset because the changes are done from an external
-script and the test system is not able to detect the database changes.
-
- >>> LaunchpadTestSetup().force_dirty_database()
- >>> LaunchpadTestSetup().tearDown()
-
=== modified file 'lib/lp/translations/interfaces/pofile.py'
--- lib/lp/translations/interfaces/pofile.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/interfaces/pofile.py 2010-09-02 16:31:56 +0000
@@ -201,6 +201,13 @@
`date_created` with newest first.
"""
+ def getTranslationMessages(condition=None):
+ """Get TranslationMessages in this `IPOFile`.
+
+ If condition is None, return all messages, else narrow the result
+ set down using the condition.
+ """
+
def makeTranslatableMessage(potmsgset):
"""Factory method for an `ITranslatableMessage` object.
=== modified file 'lib/lp/translations/interfaces/translationmessage.py'
--- lib/lp/translations/interfaces/translationmessage.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/interfaces/translationmessage.py 2010-09-02 16:31:56 +0000
@@ -229,6 +229,9 @@
def getOnePOFile():
"""Get any POFile containing this translation."""
+ def ensureBrowserPOFile():
+ """Make sure browser_pofile contains something if possible."""
+
def isHidden(pofile):
"""Whether this is an unused, hidden suggestion in `pofile`.
=== modified file 'lib/lp/translations/model/pofile.py'
--- lib/lp/translations/model/pofile.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/model/pofile.py 2010-09-02 16:31:56 +0000
@@ -21,11 +21,11 @@
BoolCol,
ForeignKey,
IntCol,
- SQLMultipleJoin,
StringCol,
)
from storm.expr import (
And,
+ Coalesce,
Exists,
In,
Join,
@@ -36,7 +36,10 @@
SQL,
)
from storm.info import ClassAlias
-from storm.store import Store
+from storm.store import (
+ EmptyResultSet,
+ Store,
+ )
from zope.component import (
getAdapter,
getUtility,
@@ -56,6 +59,7 @@
)
from canonical.launchpad import helpers
from canonical.launchpad.interfaces.launchpad import ILaunchpadCelebrities
+from canonical.launchpad.interfaces.lpstorm import IStore
from canonical.launchpad.readonly import is_read_only
from canonical.launchpad.webapp.interfaces import (
DEFAULT_FLAVOR,
@@ -525,9 +529,26 @@
from_sourcepackagename = ForeignKey(foreignKey='SourcePackageName',
dbName='from_sourcepackagename', notNull=False, default=None)
- # joins
- translation_messages = SQLMultipleJoin(
- 'TranslationMessage', joinColumn='pofile', orderBy='id')
+ @property
+ def translation_messages(self):
+ """See `IPOFile`."""
+ return self.getTranslationMessages()
+
+ def getTranslationMessages(self, condition=None):
+ """See `IPOFile`."""
+ clauses = [
+ TranslationTemplateItem.potmsgsetID == TranslationMessage.potmsgsetID,
+ TranslationTemplateItem.potemplate == self.potemplate,
+ TranslationMessage.language == self.language,
+ Coalesce(
+ TranslationMessage.potemplateID,
+ self.potemplate.id) == self.potemplate.id,
+ ]
+ if condition is not None:
+ clauses.append(condition)
+
+ return IStore(self).find(
+ TranslationMessage, *clauses).order_by(TranslationMessage.id)
@property
def title(self):
@@ -1086,9 +1107,9 @@
entry_to_import.setErrorOutput(None)
# Prepare the mail notification.
- msgsets_imported = TranslationMessage.select(
- 'was_obsolete_in_last_import IS FALSE AND pofile=%s' %
- (sqlvalues(self.id))).count()
+ msgsets_imported = self.getTranslationMessages(
+ TranslationMessage.was_obsolete_in_last_import == False
+ ).count()
replacements = collect_import_info(entry_to_import, self, warnings)
replacements.update({
@@ -1354,16 +1375,13 @@
"""See `IPOFile`."""
return self.potemplate.translationpermission
- def emptySelectResults(self):
- return POFile.select("1=2")
-
def getTranslationsFilteredBy(self, person):
"""See `IPOFile`."""
return None
def getPOTMsgSetTranslated(self):
"""See `IPOFile`."""
- return self.emptySelectResults()
+ return EmptyResultSet()
def getPOTMsgSetUntranslated(self):
"""See `IPOFile`."""
@@ -1371,15 +1389,19 @@
def getPOTMsgSetWithNewSuggestions(self):
"""See `IPOFile`."""
- return self.emptySelectResults()
+ return EmptyResultSet()
def getPOTMsgSetChangedInLaunchpad(self):
"""See `IPOFile`."""
- return self.emptySelectResults()
+ return EmptyResultSet()
def getPOTMsgSetWithErrors(self):
"""See `IPOFile`."""
- return self.emptySelectResults()
+ return EmptyResultSet()
+
+ def getTranslationMessages(self, condition=None):
+ """See `IPOFile`."""
+ return EmptyResultSet()
def hasMessageID(self, msgid):
"""See `IPOFile`."""
=== modified file 'lib/lp/translations/model/potmsgset.py'
--- lib/lp/translations/model/potmsgset.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/model/potmsgset.py 2010-09-02 16:31:56 +0000
@@ -846,7 +846,7 @@
matching_message = TranslationMessage(
potmsgset=self,
potemplate=pofile.potemplate,
- pofile=pofile,
+ pofile=None,
language=pofile.language,
origin=origin,
submitter=submitter,
=== modified file 'lib/lp/translations/model/translationmessage.py'
--- lib/lp/translations/model/translationmessage.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/model/translationmessage.py 2010-09-02 16:31:56 +0000
@@ -123,7 +123,7 @@
'This translation message already exists in the database.')
self.id = None
- self.pofile = pofile
+ self.pofile = None
self.browser_pofile = pofile
self.potemplate = pofile.potemplate
self.language = pofile.language
@@ -160,6 +160,9 @@
"""See `ITranslationMessage`."""
return None
+ def ensureBrowserPOFile(self):
+ """See `ITranslationMessage`."""
+
@property
def all_msgstrs(self):
"""See `ITranslationMessage`."""
@@ -380,6 +383,11 @@
else:
return None
+ def ensureBrowserPOFile(self):
+ """See `ITranslationMessage`."""
+ if self.browser_pofile is None:
+ self.browser_pofile = self.getOnePOFile()
+
def _getSharedEquivalent(self):
"""Get shared message that otherwise exactly matches this one.
"""
=== modified file 'lib/lp/translations/scripts/gettext_check_messages.py'
--- lib/lp/translations/scripts/gettext_check_messages.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/scripts/gettext_check_messages.py 2010-09-02 16:31:56 +0000
@@ -110,13 +110,18 @@
def _get_imported_alternative(self, translationmessage):
"""Look for a valid, imported alternative for this message."""
- if translationmessage.is_imported:
- return None
+ # Do not search for an imported alternative.
+ return None
+ # This code is disabled because translationmessage.pofile is not
+ # available anymore. Providing the same functionality now would
+ # require extensive changes to the whole script.
+ # if translationmessage.is_imported:
+ # return None
- potmsgset = translationmessage.potmsgset
- pofile = translationmessage.pofile
- return potmsgset.getImportedTranslationMessage(
- pofile.potemplate, pofile.language)
+ # potmsgset = translationmessage.potmsgset
+ # pofile = translationmessage.pofile
+ # return potmsgset.getImportedTranslationMessage(
+ # pofile.potemplate, pofile.language)
def _check_and_fix(self, translationmessage):
"""Check message against gettext, and fix it if necessary."""
=== modified file 'lib/lp/translations/templates/currenttranslationmessage-translate-one.pt'
--- lib/lp/translations/templates/currenttranslationmessage-translate-one.pt 2010-04-23 00:38:35 +0000
+++ lib/lp/translations/templates/currenttranslationmessage-translate-one.pt 2010-09-02 16:31:56 +0000
@@ -5,7 +5,7 @@
omit-tag="">
<tal:language-code
- define="language_code context/pofile/language/code">
+ define="language_code context/language/code">
<tr class="translation">
<td class="icon left right">
<a tal:attributes="
@@ -176,7 +176,7 @@
<tr tal:attributes="class string:secondary translation ${view/html_id}">
<th colspan="3">
<label class="language-code">Current
- <span tal:replace="context/pofile/language/englishname">
+ <span tal:replace="context/language/englishname">
Welsh
</span><span tal:replace="string:[${plural_index}]"
tal:condition="view/is_plural">[0]</span>:
@@ -219,8 +219,8 @@
<div tal:content="structure translation_dictionary/current_translation"
tal:condition="not: view/user_is_official_translator"
tal:attributes="
- lang context/pofile/language/dashedcode;
- dir context/pofile/language/abbreviated_text_dir;
+ lang context/language/dashedcode;
+ dir context/language/abbreviated_text_dir;
id string:${translation_dictionary/html_id_translation}">
current translation
</div>
@@ -229,8 +229,8 @@
tal:content="structure translation_dictionary/current_translation"
tal:condition="view/user_is_official_translator"
tal:attributes="
- lang context/pofile/language/dashedcode;
- dir context/pofile/language/abbreviated_text_dir;
+ lang context/language/dashedcode;
+ dir context/language/abbreviated_text_dir;
id string:${translation_dictionary/html_id_translation};
for string:${translation_dictionary/html_id_translation}_radiobutton">
current translation
@@ -503,8 +503,8 @@
tal:attributes="
name string:${translation_dictionary/html_id_translation}_new;
id string:${translation_dictionary/html_id_translation}_new;
- lang context/pofile/language/dashedcode;
- dir context/pofile/language/abbreviated_text_dir;
+ lang context/language/dashedcode;
+ dir context/language/abbreviated_text_dir;
value translation_dictionary/submitted_translation;
"
class="translate expandable"
@@ -526,8 +526,8 @@
tal:attributes="
id string:${translation_dictionary/html_id_translation}_new;
name string:${translation_dictionary/html_id_translation}_new;
- lang context/pofile/language/dashedcode;
- dir context/pofile/language/abbreviated_text_dir;
+ lang context/language/dashedcode;
+ dir context/language/abbreviated_text_dir;
">
<tal:content replace="translation_dictionary/submitted_translation" /></textarea>
</tal:with-content>
@@ -544,8 +544,8 @@
tal:attributes="
id string:${translation_dictionary/html_id_translation}_new;
name string:${translation_dictionary/html_id_translation}_new;
- lang context/pofile/language/dashedcode;
- dir context/pofile/language/abbreviated_text_dir;
+ lang context/language/dashedcode;
+ dir context/language/abbreviated_text_dir;
"></textarea>
</tal:without-content>
</tal:multi-line>
=== modified file 'lib/lp/translations/templates/translationmessage-translate.pt'
--- lib/lp/translations/templates/translationmessage-translate.pt 2010-05-18 18:04:00 +0000
+++ lib/lp/translations/templates/translationmessage-translate.pt 2010-09-02 16:31:56 +0000
@@ -84,7 +84,7 @@
<!-- Paging doodads. -->
<tal:navigation
replace="structure view/batchnav/@@+navigation-links-lower" />
- <tal:status replace="structure context/pofile/@@+access" />
+ <tal:status replace="structure context/browser_pofile/@@+access" />
</tal:havepluralforms>
<metal:pofile-js-footer
use-macro="context/@@+translations-macros/pofile-js-footer" />
=== modified file 'lib/lp/translations/tests/pofiletranslator.txt'
--- lib/lp/translations/tests/pofiletranslator.txt 2009-08-13 15:12:16 +0000
+++ lib/lp/translations/tests/pofiletranslator.txt 2010-09-02 16:31:56 +0000
@@ -25,15 +25,7 @@
>>> potmsgset_id = 1
-Note that our oldest TranslationMessage references pofile #1
-
- >>> cur.execute("""
- ... SELECT pofile
- ... FROM TranslationMessage
- ... ORDER BY date_created
- ... LIMIT 1""")
- >>> cur.fetchone()[0]
- 1
+Note that our oldest TranslationMessage references pofile #1.
Stub has so far not translated anything in this pofile
=== modified file 'lib/lp/translations/tests/test_doc.py'
--- lib/lp/translations/tests/test_doc.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/tests/test_doc.py 2010-09-02 16:31:56 +0000
@@ -27,12 +27,8 @@
special = {
'pofile-views.txt': LayeredDocFileSuite(
- '../browser/tests/pofile-views.txt',
- setUp=setUp, tearDown=tearDown, layer=LaunchpadFunctionalLayer
- ),
- 'remove-upstream-translations-script.txt': LayeredDocFileSuite(
- '../doc/remove-upstream-translations-script.txt',
- setUp=setGlobs, stdout_logging=False, layer=None
+ '../browser/tests/pofile-views.txt',
+ setUp=setUp, tearDown=tearDown, layer=LaunchpadFunctionalLayer
),
'poexport-queue.txt': LayeredDocFileSuite(
'../doc/poexport-queue.txt',
=== modified file 'lib/lp/translations/tests/test_pofile.py'
--- lib/lp/translations/tests/test_pofile.py 2010-08-20 20:31:18 +0000
+++ lib/lp/translations/tests/test_pofile.py 2010-09-02 16:31:56 +0000
@@ -1729,8 +1729,7 @@
def test_makeTranslatableMessage(self):
# TranslatableMessages can be created from the PO file
- potmsgset = self.factory.makePOTMsgSet(self.potemplate,
- sequence=1)
+ potmsgset = self.factory.makePOTMsgSet(self.potemplate, sequence=1)
message = self.pofile.makeTranslatableMessage(potmsgset)
verifyObject(ITranslatableMessage, message)
@@ -1753,6 +1752,77 @@
"getTranslationRows does not sort obsolete messages "
"(sequence=0) to the end of the file.")
+ def test_getTranslationMessages_baseline(self):
+ # Return all translation messages for this POFile.
+ # Shared, diverged and obsolete.
+ potmsgset = self.factory.makePOTMsgSet(self.potemplate, sequence=1)
+ suggestion = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=self.pofile)
+ current_shared = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=self.pofile, force_shared=True)
+ current_diverged = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=self.pofile, force_diverged=True)
+ obsolete_potmsgset = self.factory.makePOTMsgSet(
+ self.potemplate, sequence=0)
+ obsolete = self.factory.makeTranslationMessage(
+ potmsgset=obsolete_potmsgset, pofile=self.pofile,
+ force_shared=True)
+
+ self.assertContentEqual(
+ [current_shared, current_diverged, suggestion, obsolete],
+ self.pofile.getTranslationMessages())
+
+ def test_getTranslationMessages_condition(self):
+ # Narrow the result set down using a condition.
+ potmsgset = self.factory.makePOTMsgSet(self.potemplate, sequence=1)
+ suggestion = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=self.pofile)
+ current_shared = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=self.pofile, force_shared=True)
+ current_diverged = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=self.pofile, force_diverged=True)
+ obsolete_potmsgset = self.factory.makePOTMsgSet(
+ self.potemplate, sequence=0)
+ obsolete = self.factory.makeTranslationMessage(
+ potmsgset=obsolete_potmsgset, pofile=self.pofile,
+ force_shared=True)
+
+ self.assertContentEqual(
+ [current_shared, suggestion, obsolete],
+ self.pofile.getTranslationMessages(
+ "TranslationMessage.potemplate IS NULL"))
+
+ def test_getTranslationMessages_other_pofile(self):
+ # Messages from other POFiles are not included.
+ potmsgset = self.factory.makePOTMsgSet(self.potemplate, sequence=1)
+ other_pofile = self.factory.makePOFile('de')
+ my_message = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=self.pofile)
+ other_message = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=other_pofile)
+
+ def test_getTranslationMessages_diverged_elsewhere(self):
+ # Diverged messages from sharing POTemplates are not included.
+ potmsgset = self.factory.makePOTMsgSet(self.potemplate, sequence=1)
+
+ # Create a sharing potemplate in another product series and share
+ # potmsgset in both templates.
+ other_series = self.factory.makeProductSeries(
+ product=self.potemplate.productseries.product)
+ other_template = self.factory.makePOTemplate(
+ productseries=other_series, name=self.potemplate.name)
+ other_pofile = other_template.getPOFileByLang(
+ self.pofile.language.code)
+ potmsgset.setSequence(other_template, 1)
+
+ my_message = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=self.pofile)
+ other_message = self.factory.makeTranslationMessage(
+ potmsgset=potmsgset, pofile=other_pofile, force_diverged=True)
+
+ self.assertContentEqual(
+ [my_message], self.pofile.getTranslationMessages())
+
class TestPOFileToTranslationFileDataAdapter(TestCaseWithFactory):
"""Test POFile being adapted to IPOFileToTranslationFileData."""
=== removed file 'scripts/rosetta/remove-upstream-translations.py'
--- scripts/rosetta/remove-upstream-translations.py 2010-08-06 07:02:32 +0000
+++ scripts/rosetta/remove-upstream-translations.py 1970-01-01 00:00:00 +0000
@@ -1,227 +0,0 @@
-#!/usr/bin/python -S
-#
-# Copyright 2009 Canonical Ltd. This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-# pylint: disable-msg=W0403
-"""Remove all translations from upstream.
-
-This script is useful to recover from breakages after importing bad
-.po files like the one reported at #32610.
-"""
-
-import _pythonpath
-
-import sys
-import logging
-from optparse import OptionParser
-
-from zope.component import getUtility
-
-from canonical.config import config
-from canonical.database.constants import UTC_NOW
-from canonical.lp import initZopeless
-from canonical.launchpad.scripts import (
- execute_zcml_for_scripts, logger, logger_options)
-from canonical.launchpad.interfaces import ILaunchpadCelebrities
-from lp.registry.interfaces.product import IProductSet
-from lp.registry.interfaces.distribution import IDistributionSet
-from lp.registry.interfaces.distroseries import IDistroSeriesSet
-from lp.registry.interfaces.sourcepackagename import (
- ISourcePackageNameSet)
-from lp.translations.interfaces.potemplate import IPOTemplateSet
-from lp.translations.interfaces.translationmessage import (
- RosettaTranslationOrigin)
-
-
-
-logger_name = 'remove-upstream-translations'
-
-def parse_options(args):
- """Parse a set of command line options.
-
- Return an optparse.Values object.
- """
- parser = OptionParser()
-
- parser.add_option("-p", "--product", dest="product",
- help="The product where we should look for translations.")
- parser.add_option("-s", "--series", dest="series",
- help="The product series where we should look for translations.")
- parser.add_option("-d", "--distro", dest="distro",
- help="The distribution where we should look for translations.")
- parser.add_option("-r", "--distroseries", dest="distroseries",
- help="The distribution series where we should look for translations."
- )
- parser.add_option("-n", "--sourcepackagename", dest="sourcepackagename",
- help="The distribution where we should look for translations.")
- parser.add_option("-t", "--potemplatename", dest="potemplatename",
- help="The PO Template name where we should look for translations.")
- parser.add_option("-l", "--language-code", dest="languagecode",
- help="The language code where we should look for translations.")
-
- # Add the verbose/quiet options.
- logger_options(parser)
-
- (options, args) = parser.parse_args(args)
-
- return options
-
-def remove_upstream_entries(ztm, potemplates, lang_code=None):
- """Remove all translations that came from upstream.
-
- :arg ztm: Zope transaction manager.
- :arg potemplates: A set of potemplates that we should process.
- :arg lang_code: A string with a language code where we should do the
- removal.
-
- If lang_code is None, we process all available languages.
- """
-
- logger_object = logging.getLogger(logger_name)
-
- items_deleted = 0
- # All changes should be logged as done by Rosetta Expert team.
- rosetta_experts = getUtility(ILaunchpadCelebrities).rosetta_experts
-
- for potemplate in potemplates:
- if lang_code is None:
- pofiles = sorted(
- list(potemplate.pofiles),
- key=lambda p: p.language.code)
- else:
- pofile = potemplate.getPOFileByLang(lang_code)
- if pofile is None:
- pofiles = []
- else:
- pofiles = [pofile]
-
- for pofile in pofiles:
- logger_object.debug('Processing %s...' % pofile.title)
- pofile_items_deleted = 0
- for message in pofile.translation_messages:
- active_changed = False
- if message.origin == RosettaTranslationOrigin.SCM:
- if message.is_current:
- active_changed = True
- message.destroySelf()
- pofile_items_deleted += 1
- if active_changed:
- message.pofile.date_changed = UTC_NOW
- message.pofile.lasttranslator = rosetta_experts
- message.reviewer = rosetta_experts
- message.date_reviewed = UTC_NOW
-
- items_deleted += pofile_items_deleted
- logger_object.debug(
- 'Removed %d submissions' % pofile_items_deleted)
- pofile.updateStatistics()
- ztm.commit()
-
- # We finished the removal process, is time to notify the amount of entries
- # that we removed.
- logger_object.debug(
- 'Removed %d submissions in total.' % items_deleted)
-
-
-def main(argv):
- options = parse_options(argv[1:])
- logger_object = logger(options, logger_name)
-
- execute_zcml_for_scripts()
- ztm = initZopeless(dbuser=config.rosetta.admin_dbuser)
-
- product = None
- series = None
- distro = None
- distroseries = None
- sourcepackagename = None
- potemplatename = None
- language_code = None
- if options.product is not None:
- productset = getUtility(IProductSet)
- product = productset.getByName(options.product)
- if product is None:
- logger_object.error(
- 'The %s product does not exist.' % options.product)
- return 1
-
- if options.series is not None:
- if product is None:
- logger_object.error(
- 'You need to specify a product if you want to select a'
- ' productseries.')
- return 1
-
- series = product.getSeries(options.series)
- if series is None:
- logger_object.error(
- 'The %s series does not exist inside %s product.' % (
- options.series, options.product))
- return 1
-
- if options.distro is not None:
- if product is not None:
- logger_object.error(
- 'You cannot mix distributions and products.')
- return 1
- distroset = getUtility(IDistributionSet)
- distro = distroset.getByName(options.distro)
- if distro is None:
- logger_object.error(
- 'The %s distribution does not exist.' % options.distro)
- return 1
-
- if options.distroseries is not None:
- if distro is None:
- logger_object.error(
- 'You need to specify a distribution if you want to select a'
- ' sourcepackagename.')
- distroseriesset = getUtility(IDistroSeriesSet)
- distroseries = distroseriesset.queryByName(
- distro, options.distroseries)
- if distroseries is None:
- logger_object.error(
- 'The %s distribution does not exist.' % options.distroseries)
- return 1
-
- if options.sourcepackagename is not None:
- if distroseries is None:
- logger_object.error(
- 'You need to specify a distribution series if you want to'
- ' select a sourcepackagename.')
- return 1
- sourcepackagenameset = getUtility(ISourcePackageNameSet)
- sourcepackagename = sourcepackagenameset.queryByName(
- options.sourcepackagename)
- if sourcepackagename is None:
- logger_object.error(
- 'The %s sourcepackagename does not exist.' % (
- options.sourcepackagename))
- return 1
-
- potemplateset = getUtility(IPOTemplateSet)
- if series is None and distroseries is None:
- if options.potemplatename is None:
- logger_object.warning('Nothing to do. Exiting...')
- return 0
- else:
- potemplates = potemplateset.getAllByName(
- options.potemplatename)
- else:
- potemplate_subset = potemplateset.getSubset(
- distroseries=distroseries, sourcepackagename=sourcepackagename,
- productseries=series)
- if options.potemplatename is not None:
- potemplate = potemplate_subset.getPOTemplateByName(
- options.potemplatename)
- potemplates = [potemplate]
- else:
- # Get a list from the subset of potemplates to be able to do
- # transaction commits.
- potemplates = list(potemplate_subset)
-
- remove_upstream_entries(ztm, potemplates, options.languagecode)
-
-if __name__ == '__main__':
- sys.exit(main(sys.argv))
