launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #02797
[Merge] lp:~lifeless/launchpad/bug-711064 into lp:launchpad
Robert Collins has proposed merging lp:~lifeless/launchpad/bug-711064 into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
#711064 POFile:+translate timeouts
https://bugs.launchpad.net/bugs/711064
For more details, see:
https://code.launchpad.net/~lifeless/launchpad/bug-711064/+merge/51491
Drop 1/2 of the very expensive queries in POFile:+translate GET requests by grabbing all the data at once and partitioning in python: we were grabbing all the data anyway, and forcing the DB to figure it out twice. This is more efficient overall and we're dealing with fairly small data sets. I couldn't just migrate the existing properties because they are used in another place singularly, so I would have been making the other call site substantially slower unnecessarily.
Hot queries for this data is fine, but cold ones time out quite a lot; I hope this branch will help - it may save up to 6 seconds on the page for cold loads.
--
https://code.launchpad.net/~lifeless/launchpad/bug-711064/+merge/51491
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~lifeless/launchpad/bug-711064 into lp:launchpad.
=== modified file 'lib/lp/translations/browser/translationmessage.py'
--- lib/lp/translations/browser/translationmessage.py 2010-12-25 21:28:14 +0000
+++ lib/lp/translations/browser/translationmessage.py 2011-02-28 05:03:55 +0000
@@ -1302,8 +1302,10 @@
# Get a list of translations which are _used_ as translations
# for this same message in a different translation template.
+ translations = potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)
externally_used = self._setOnePOFile(sorted(
- potmsgset.getExternallyUsedTranslationMessages(language),
+ translations[1],
key=operator.attrgetter("date_created"),
reverse=True))
@@ -1311,7 +1313,7 @@
# translations for this same message in a different translation
# template, but are not used.
externally_suggested = self._setOnePOFile(sorted(
- potmsgset.getExternallySuggestedTranslationMessages(language),
+ translations[0],
key=operator.attrgetter("date_created"),
reverse=True))
else:
=== modified file 'lib/lp/translations/doc/potmsgset.txt'
--- lib/lp/translations/doc/potmsgset.txt 2011-02-24 14:31:46 +0000
+++ lib/lp/translations/doc/potmsgset.txt 2011-02-28 05:03:55 +0000
@@ -378,6 +378,23 @@
Spanish (es) translation of evolution-2.2 in Evolution trunk:
tarjetas (Launchpad & Upstream)
+We get the same results when using the optimised helper:
+
+ >>> suggestions = (
+ ... evo_product_message.getExternallySuggestedOrUsedTranslationMessages(
+ ... spanish))[1]
+ >>> print_suggestions(suggestions)
+ Spanish (es) translation of evolution-2.2 in Ubuntu Hoary package
+ "evolution": caratas (Launchpad)
+ Spanish (es) translation of evolution-2.2 in Ubuntu Hoary package
+ "evolution": tarjetas (Upstream)
+
+ >>> suggestions = evo_distro_message.getExternallySuggestedOrUsedTranslationMessages(
+ ... spanish)[1]
+ >>> print_suggestions(suggestions)
+ Spanish (es) translation of evolution-2.2 in Evolution trunk:
+ tarjetas (Launchpad & Upstream)
+
We need to be logged in as an admin to do some special attribute
changes:
@@ -394,6 +411,13 @@
>>> len(suggestions)
0
+ >>> suggestions = (
+ ... evo_product_message.getExternallySuggestedOrUsedTranslationMessages(
+ ... spanish))[1]
+ >>> len(suggestions)
+ 0
+
+
The same happens if the distribution is not officially using
translations.
@@ -409,6 +433,11 @@
... evo_product_message.getExternallyUsedTranslationMessages(spanish))
>>> len(suggestions)
0
+ >>> suggestions = (
+ ... evo_product_message.getExternallySuggestedOrUsedTranslationMessages(
+ ... spanish))[1]
+ >>> len(suggestions)
+ 0
And products not using translations officially have the same behaviour.
@@ -419,6 +448,10 @@
... spanish)
>>> len(suggestions)
0
+ >>> suggestions = evo_distro_message.getExternallySuggestedOrUsedTranslationMessages(
+ ... spanish)[1]
+ >>> len(suggestions)
+ 0
Let's restore the flags for next section.
@@ -508,6 +541,17 @@
Spanish (es) translation of man in Ubuntu Hoary package "evolution":
lalalala (None)
+And our optimised helper returns these too:
+
+ >>> wiki_submissions = (
+ ... potmsgset_untranslated.getExternallySuggestedOrUsedTranslationMessages(
+ ... pofile.language))[0]
+ >>> print_suggestions(wiki_submissions)
+ Spanish (es) translation of man in Ubuntu Hoary package "evolution":
+ blah, blah, blah (None)
+ Spanish (es) translation of man in Ubuntu Hoary package "evolution":
+ lalalala (None)
+
However, if the hoary template version is not current and thus hidden,
we get no suggestions.
@@ -516,11 +560,18 @@
>>> transaction.commit()
>>> wiki_submissions = (
- ... potmsgset_untranslated.getExternallyUsedTranslationMessages(
+ ... potmsgset_untranslated.getExternallySuggestedTranslationMessages(
... pofile.language))
>>> len(wiki_submissions)
0
+ >>> wiki_submissions = (
+ ... potmsgset_untranslated.getExternallySuggestedOrUsedTranslationMessages(
+ ... pofile.language))[0]
+ >>> len(wiki_submissions)
+ 0
+
+
Nor do we get any suggestions if the Ubuntu distribution is not using
Launchpad for translations.
@@ -537,6 +588,12 @@
>>> len(wiki_submissions)
0
+ >>> wiki_submissions = (
+ ... potmsgset_untranslated.getExternallySuggestedOrUsedTranslationMessages(
+ ... pofile.language))[0]
+ >>> len(wiki_submissions)
+ 0
+
Suggestions for translator credits
----------------------------------
@@ -576,6 +633,8 @@
[]
>>> new_credits.getExternallySuggestedTranslationMessages(spanish)
[]
+ >>> new_credits.getExternallySuggestedOrUsedTranslationMessages(spanish)[0]
+ []
POTMsgSet.setSequence
---------------------
=== modified file 'lib/lp/translations/interfaces/potmsgset.py'
--- lib/lp/translations/interfaces/potmsgset.py 2011-02-22 16:45:59 +0000
+++ lib/lp/translations/interfaces/potmsgset.py 2011-02-28 05:03:55 +0000
@@ -195,6 +195,16 @@
:param language: language we want translations for.
"""
+ def getExternallySuggestedOrUsedTranslationMessages(language):
+ """Find externally suggested or used translations for the same message.
+
+ This returns a tuple (suggested, used) containing the results of
+ self.getExternallySuggestedTranslationMessages and
+ self.getExternallyUsedTranslationMessages respectively.
+
+ :param language: language we want translations for.
+ """
+
def hasTranslationChangedInLaunchpad(potemplate, language):
"""Whether an imported translation differs from the current one.
=== modified file 'lib/lp/translations/model/potmsgset.py'
--- lib/lp/translations/model/potmsgset.py 2011-02-25 09:25:08 +0000
+++ lib/lp/translations/model/potmsgset.py 2011-02-28 05:03:55 +0000
@@ -361,7 +361,7 @@
return TranslationMessage.select(query)
- def _getExternalTranslationMessages(self, language, used):
+ def _getExternalTranslationMessages(self, language, used=None):
"""Return external suggestions for this message.
External suggestions are all TranslationMessages for the
@@ -372,6 +372,10 @@
Suggestions are read-only, so these objects come from the slave
store.
+
+ :param used: If None, return both used and unused messages.
+ Otherwise, return only used (if bool(used)) or unuesd (if not
+ bool(used)) messages.
"""
if not config.rosetta.global_suggestions_enabled:
return []
@@ -385,7 +389,9 @@
# Also note that there is a NOT(in_use_clause) index.
in_use_clause = (
"(is_current_ubuntu IS TRUE OR is_current_upstream IS TRUE)")
- if used:
+ if used is None:
+ query = []
+ elif used:
query = [in_use_clause]
else:
query = ["(NOT %s)" % in_use_clause]
@@ -440,6 +446,23 @@
"""See `IPOTMsgSet`."""
return self._getExternalTranslationMessages(language, used=False)
+ def getExternallySuggestedOrUsedTranslationMessages(self, language):
+ """See `IPOTMsgSet`."""
+ # This method exists because suggestions + used == all external
+ # messages : its better not to do the work twice. We could use a
+ # temp table and query twice, but as the list length is capped at
+ # 2000, doing a single pass in python should be insignificantly
+ # slower.
+ suggested_messages = []
+ used_messages = []
+ for message in self._getExternalTranslationMessages(language):
+ in_use = message.is_current_ubuntu or message.is_current_upstream
+ if in_use:
+ used_messages.append(message)
+ else:
+ suggested_messages.append(message)
+ return suggested_messages, used_messages
+
@property
def flags(self):
if self.flagscomment is None:
=== modified file 'lib/lp/translations/tests/test_potmsgset.py'
--- lib/lp/translations/tests/test_potmsgset.py 2011-02-25 14:59:44 +0000
+++ lib/lp/translations/tests/test_potmsgset.py 2011-02-28 05:03:55 +0000
@@ -364,6 +364,10 @@
self.assertEquals(
self.potmsgset.getExternallyUsedTranslationMessages(language),
[])
+ self.assertEquals(
+ self.potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)[1],
+ [])
# If there are only suggestions on the external POTMsgSet,
# no externally used suggestions are returned.
@@ -375,6 +379,10 @@
self.assertEquals(
self.potmsgset.getExternallyUsedTranslationMessages(language),
[])
+ self.assertEquals(
+ self.potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)[1],
+ [])
# If there is a translation for the other side on the external
# POTMsgSet, it is returned as an externally used suggestion.
@@ -387,6 +395,10 @@
self.assertEquals(
self.potmsgset.getExternallyUsedTranslationMessages(language),
[other_translation])
+ self.assertEquals(
+ self.potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)[1],
+ [other_translation])
# If there is a current translation on the external POTMsgSet,
# it is returned as the externally used suggestion as well.
@@ -398,6 +410,10 @@
self.assertContentEqual(
self.potmsgset.getExternallyUsedTranslationMessages(language),
[other_translation, current_translation])
+ self.assertEquals(
+ self.potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)[1],
+ [other_translation, current_translation])
def test_getExternallySuggestedTranslationMessages(self):
"""Test retrieval of externally suggested translations."""
@@ -422,6 +438,10 @@
self.potmsgset.getExternallySuggestedTranslationMessages(
language),
[])
+ self.assertEquals(
+ self.potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)[0],
+ [])
# If there is a suggestion on the external POTMsgSet,
# it is returned.
@@ -434,6 +454,10 @@
self.potmsgset.getExternallySuggestedTranslationMessages(
language),
[external_suggestion])
+ self.assertEquals(
+ self.potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)[0],
+ [external_suggestion])
# If there is a translation for the other side on the external
# POTMsgSet, it is not returned as the external suggestion.
@@ -446,6 +470,10 @@
self.potmsgset.getExternallySuggestedTranslationMessages(
language),
[external_suggestion])
+ self.assertEquals(
+ self.potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)[0],
+ [external_suggestion])
# A current translation on the external POTMsgSet is not
# considered an external suggestion.
@@ -458,6 +486,10 @@
self.potmsgset.getExternallySuggestedTranslationMessages(
language),
[external_suggestion])
+ self.assertEquals(
+ self.potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ language)[0],
+ [external_suggestion])
def test_hasTranslationChangedInLaunchpad(self):
"""Check whether a translation is changed in Ubuntu works."""
=== modified file 'lib/lp/translations/tests/test_suggestions.py'
--- lib/lp/translations/tests/test_suggestions.py 2011-02-17 22:08:50 +0000
+++ lib/lp/translations/tests/test_suggestions.py 2011-02-28 05:03:55 +0000
@@ -63,6 +63,9 @@
potmsgset.getExternallyUsedTranslationMessages(self.nl), [])
self.assertEquals(
potmsgset.getExternallySuggestedTranslationMessages(self.nl), [])
+ self.assertEqual(
+ potmsgset.getExternallySuggestedOrUsedTranslationMessages(self.nl),
+ ([], []))
def test_SimpleExternallyUsedSuggestion(self):
# If foo wants to translate "error message 936" and bar happens
@@ -76,13 +79,18 @@
transaction.commit()
+ def check_used_suggested():
+ self.assertEquals(len(used_suggestions), 1)
+ self.assertEquals(used_suggestions[0], translation)
+ self.assertEquals(len(other_suggestions), 0)
used_suggestions = foomsg.getExternallyUsedTranslationMessages(
self.nl)
other_suggestions = foomsg.getExternallySuggestedTranslationMessages(
self.nl)
- self.assertEquals(len(used_suggestions), 1)
- self.assertEquals(used_suggestions[0], translation)
- self.assertEquals(len(other_suggestions), 0)
+ check_used_suggested()
+ other_suggestions, used_suggestions = \
+ foomsg.getExternallySuggestedOrUsedTranslationMessages(self.nl)
+ check_used_suggested()
def test_DisabledExternallyUsedSuggestions(self):
# If foo wants to translate "error message 936" and bar happens
@@ -101,6 +109,9 @@
used_suggestions = foomsg.getExternallyUsedTranslationMessages(
self.nl)
self.assertEquals(len(used_suggestions), 1)
+ used_suggestions = foomsg.getExternallySuggestedOrUsedTranslationMessages(
+ self.nl)[1]
+ self.assertEquals(len(used_suggestions), 1)
# Override the config option to disable global suggestions.
new_config = ("""
@@ -111,6 +122,9 @@
disabled_used_suggestions = (
foomsg.getExternallyUsedTranslationMessages(self.nl))
self.assertEquals(len(disabled_used_suggestions), 0)
+ disabled_used_suggestions = (
+ foomsg.getExternallySuggestedOrUsedTranslationMessages(self.nl))[1]
+ self.assertEquals(len(disabled_used_suggestions), 0)
# Restore the old configuration.
config.pop('disabled_suggestions')
@@ -124,13 +138,18 @@
transaction.commit()
+ def check_used_suggested():
+ self.assertEquals(len(used_suggestions), 0)
+ self.assertEquals(len(other_suggestions), 1)
+ self.assertEquals(other_suggestions[0], suggestion)
used_suggestions = foomsg.getExternallyUsedTranslationMessages(
self.nl)
other_suggestions = foomsg.getExternallySuggestedTranslationMessages(
self.nl)
- self.assertEquals(len(used_suggestions), 0)
- self.assertEquals(len(other_suggestions), 1)
- self.assertEquals(other_suggestions[0], suggestion)
+ check_used_suggested()
+ other_suggestions, used_suggestions = \
+ foomsg.getExternallySuggestedOrUsedTranslationMessages(self.nl)
+ check_used_suggested()
def test_IdenticalSuggestions(self):
# If two suggestions are identical, the most recent one is used.
@@ -164,6 +183,10 @@
self.nl)
self.assertEquals(len(suggestions), 1)
self.assertEquals(suggestions[0], suggestion1)
+ suggestions = oof_potmsgset.getExternallySuggestedOrUsedTranslationMessages(
+ self.nl)[1]
+ self.assertEquals(len(suggestions), 1)
+ self.assertEquals(suggestions[0], suggestion1)
def test_RevertingToUpstream(self):
# When a msgid string is unique and nobody has submitted any
