launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #05254
[Merge] lp:~danilo/launchpad/bug-817398 into lp:launchpad
Данило Шеган has proposed merging lp:~danilo/launchpad/bug-817398 into lp:launchpad.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
Related bugs:
Bug #817398 in Launchpad itself: "Launchpad doesn't generate translation templates"
https://bugs.launchpad.net/launchpad/+bug/817398
For more details, see:
https://code.launchpad.net/~danilo/launchpad/bug-817398/+merge/79571
= Bug 817398: template generation (WIP) =
Template generation has stopped working when buildd default image has switched to Ubuntu Natty (as the development version). That included Python 2.7 by default and generate-translation-templates hard-coded Python 2.6 module path.
This is easy to fix, but this has been broken for quite some time (maybe a full year even) because there is no proper logging for TranslationTemplatesBuild jobs. I decided to introduce that to avoid further breakage going unnoticed for this long.
== Tests ==
== Demo and Q/A ==
* Set up translation import for a project (translations.l.n/<project/+configure-translations and translations.l.n/<project>/trunk/+translations-settings
* Push a branch which should generate translation template (eg. lp:eog) to staging, link it to the trunk series (what was set-up above)
* Run cronscripts/scan-branches.py to scan the branch and generate translation templates build job
* Wait for build-manager to pick it up and build it
* When the job finishes, it should keep the log in the builder history available from the builder page on launchpad.net/builders
= Launchpad lint =
Checking for conflicts and issues in changed files.
Linting changed files:
utilities/make-lp-user
lib/lp/translations/model/translationtemplatesbuildbehavior.py
lib/canonical/config/schema-lazr.conf
lib/lp/translations/model/translationtemplatesbuild.py
lib/canonical/buildd/debian/changelog
lib/lp/translations/configure.zcml
lib/canonical/buildd/generate-translation-templates
lib/lp/code/model/recipebuilder.py
./utilities/make-lp-user
124: Line has trailing whitespace.
./lib/lp/translations/model/translationtemplatesbuildbehavior.py
18: 'transaction' imported but unused
210: W391 blank line at end of file
./lib/canonical/config/schema-lazr.conf
592: Line exceeds 78 characters.
1216: Line exceeds 78 characters.
1223: Line exceeds 78 characters.
./lib/lp/translations/model/translationtemplatesbuild.py
21: 'ProxyFactory' imported but unused
16: 'Store' imported but unused
24: 'ProxiedLibraryFileAlias' imported but unused
./lib/canonical/buildd/debian/changelog
232: Line exceeds 78 characters.
298: Line exceeds 78 characters.
488: Line exceeds 78 characters.
495: Line exceeds 78 characters.
516: Line exceeds 78 characters.
522: Line exceeds 78 characters.
528: Line exceeds 78 characters.
534: Line exceeds 78 characters.
541: Line exceeds 78 characters.
549: Line exceeds 78 characters.
556: Line exceeds 78 characters.
570: Line exceeds 78 characters.
576: Line exceeds 78 characters.
582: Line exceeds 78 characters.
590: Line exceeds 78 characters.
596: Line exceeds 78 characters.
603: Line exceeds 78 characters.
614: Line exceeds 78 characters.
623: Line exceeds 78 characters.
630: Line exceeds 78 characters.
636: Line exceeds 78 characters.
643: Line exceeds 78 characters.
649: Line exceeds 78 characters.
655: Line exceeds 78 characters.
662: Line exceeds 78 characters.
669: Line exceeds 78 characters.
676: Line exceeds 78 characters.
683: Line exceeds 78 characters.
689: Line exceeds 78 characters.
./lib/canonical/buildd/generate-translation-templates
59: Line exceeds 78 characters.
./lib/lp/code/model/recipebuilder.py
131: Line exceeds 78 characters.
132: Line exceeds 78 characters.
132: E501 line too long (81 characters)
--
https://code.launchpad.net/~danilo/launchpad/bug-817398/+merge/79571
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~danilo/launchpad/bug-817398 into lp:launchpad.
=== modified file 'lib/canonical/buildd/debian/changelog'
--- lib/canonical/buildd/debian/changelog 2011-09-26 06:30:07 +0000
+++ lib/canonical/buildd/debian/changelog 2011-10-17 15:21:59 +0000
@@ -1,3 +1,9 @@
+launchpad-buildd (81) hardy-cat; urgency=low
+
+ * generate-translation-templates: switch to Python 2.7.
+
+ -- Danilo Šegan <danilo@xxxxxxxxxxxxx> Mon, 17 Oct 2011 14:46:13 +0200
+
launchpad-buildd (80) hardy-cat; urgency=low
* binfmt-support demonstrated umount ordering issues for us. LP: #851934
=== modified file 'lib/canonical/buildd/generate-translation-templates'
--- lib/canonical/buildd/generate-translation-templates 2010-12-21 17:52:32 +0000
+++ lib/canonical/buildd/generate-translation-templates 2011-10-17 15:21:59 +0000
@@ -41,7 +41,7 @@
BUILDD_PACKAGE=canonical/buildd
POTTERY=$BUILDD_PACKAGE/pottery
# The script should be smarter about detecting the python version.
-PYMODULES=/usr/lib/pymodules/python2.6
+PYMODULES=/usr/lib/pymodules/python2.7
echo -n "Default Python in the chroot is: "
$BUILD_CHROOT/usr/bin/python --version
@@ -62,4 +62,5 @@
# Enter chroot, switch back to unprivileged user, execute the generate script.
$SUDO $CHROOT $BUILD_CHROOT \
- $SU - $USER -c "$GENERATE_SCRIPT $BRANCH_URL $RESULT_NAME"
+ $SU - $USER \
+ -c "PYTHONPATH=$PYMODULES $GENERATE_SCRIPT $BRANCH_URL $RESULT_NAME"
=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf 2011-10-05 18:02:45 +0000
+++ lib/canonical/config/schema-lazr.conf 2011-10-17 15:21:59 +0000
@@ -1661,7 +1661,7 @@
banner: none
# datatype: string
-port: tcp:5022
+port: tcp:5023
# datatype: string
ftp_port: 2121
=== modified file 'lib/lp/code/model/recipebuilder.py'
--- lib/lp/code/model/recipebuilder.py 2010-12-07 00:58:46 +0000
+++ lib/lp/code/model/recipebuilder.py 2011-10-17 15:21:59 +0000
@@ -39,10 +39,6 @@
status = None
@property
- def build(self):
- return self.buildfarmjob.build
-
- @property
def display_name(self):
ret = "%s, %s, %s" % (
self.build.distroseries.displayname, self.build.recipe.name,
=== modified file 'lib/lp/translations/configure.zcml'
--- lib/lp/translations/configure.zcml 2011-08-29 00:13:22 +0000
+++ lib/lp/translations/configure.zcml 2011-10-17 15:21:59 +0000
@@ -659,7 +659,11 @@
<!-- TranslationTemplatesBuild -->
<class
class="lp.translations.model.translationtemplatesbuild.TranslationTemplatesBuild">
- <allow interface="lp.translations.interfaces.translationtemplatesbuild.ITranslationTemplatesBuild"/>
+ <require permission="launchpad.View" interface="lp.translations.interfaces.translationtemplatesbuild.ITranslationTemplatesBuild"/>
+ <!-- This is needed for UploadProcessor to run. The permission isn't
+ important; launchpad.Edit isn't actually held by anybody. -->
+ <require permission="launchpad.Edit" set_attributes="status" />
+
</class>
<securedutility
component="lp.translations.model.translationtemplatesbuild.TranslationTemplatesBuild"
=== modified file 'lib/lp/translations/model/translationtemplatesbuild.py'
--- lib/lp/translations/model/translationtemplatesbuild.py 2011-05-04 04:10:58 +0000
+++ lib/lp/translations/model/translationtemplatesbuild.py 2011-10-17 15:21:59 +0000
@@ -21,6 +21,7 @@
from zope.security.proxy import ProxyFactory
from canonical.launchpad.interfaces.lpstorm import IStore
+from canonical.launchpad.browser.librarian import ProxiedLibraryFileAlias
from lp.buildmaster.model.buildfarmjob import BuildFarmJobDerived
from lp.code.model.branchjob import (
BranchJob,
@@ -49,6 +50,11 @@
branch_id = Int(name='branch', allow_none=False)
branch = Reference(branch_id, 'Branch.id')
+ @property
+ def title(self):
+ return u'%s for %s' % (
+ self.__class__.__name__, self.branch.displayname,)
+
def __init__(self, build_farm_job, branch):
super(TranslationTemplatesBuild, self).__init__()
self.build_farm_job = build_farm_job
@@ -110,3 +116,10 @@
return store.find(
TranslationTemplatesBuild,
TranslationTemplatesBuild.branch == branch)
+
+ @property
+ def log_url(self):
+ """See `IBuildFarmJob`."""
+ if self.log is None:
+ return None
+ return self.log.http_url
=== modified file 'lib/lp/translations/model/translationtemplatesbuildbehavior.py'
--- lib/lp/translations/model/translationtemplatesbuildbehavior.py 2011-09-07 15:10:38 +0000
+++ lib/lp/translations/model/translationtemplatesbuildbehavior.py 2011-10-17 15:21:59 +0000
@@ -11,8 +11,11 @@
'TranslationTemplatesBuildBehavior',
]
+import datetime
import os
+import pytz
import tempfile
+import transaction
from twisted.internet import defer
from zope.component import getUtility
@@ -20,6 +23,7 @@
from zope.security.proxy import removeSecurityProxy
from lp.app.interfaces.launchpad import ILaunchpadCelebrities
+from lp.buildmaster.enums import BuildStatus
from lp.buildmaster.interfaces.buildfarmjobbehavior import (
IBuildFarmJobBehavior,
)
@@ -111,6 +115,38 @@
if len(raw_slave_status) >= 4:
status['filemap'] = raw_slave_status[3]
+ def setBuildStatus(self, status):
+ self.build.status = status
+
+ @staticmethod
+ def getLogFromSlave(templates_build, queue_item):
+ """See `IPackageBuild`."""
+ SLAVE_LOG_FILENAME = 'buildlog'
+ builder = queue_item.builder
+ d = builder.transferSlaveFileToLibrarian(
+ SLAVE_LOG_FILENAME,
+ templates_build.buildfarmjob.getLogFileName(),
+ False)
+ return d
+
+ @staticmethod
+ def storeBuildInfo(build, queue_item, build_status):
+ """See `IPackageBuild`."""
+ def got_log(lfa_id):
+ # log, builder and date_finished are read-only, so we must
+ # currently remove the security proxy to set them.
+ naked_build = removeSecurityProxy(build.build)
+ naked_build.log = lfa_id
+ naked_build.builder = queue_item.builder
+ naked_build.date_started = queue_item.date_started
+ # XXX cprov 20060615 bug=120584: Currently buildduration includes
+ # the scanner latency, it should really be asking the slave for
+ # the duration spent building locally.
+ naked_build.date_finished = datetime.datetime.now(pytz.UTC)
+
+ d = build.getLogFromSlave(build, queue_item)
+ return d.addCallback(got_log)
+
def updateBuild_WAITING(self, queue_item, slave_status, logtail, logger):
"""Deal with a finished ("WAITING" state, perversely) build job.
@@ -121,7 +157,6 @@
retry it.
"""
build_status = self.extractBuildStatus(slave_status)
-
logger.info(
"Templates generation job %s for %s finished with status %s." % (
queue_item.specific_job.getName(),
@@ -152,15 +187,24 @@
queue_item.specific_job.branch, tarball, logger)
logger.debug("Upload complete.")
finally:
+ self.setBuildStatus(BuildStatus.FULLYBUILT)
tarball_file.close()
os.remove(filename)
- if build_status == 'OK':
- logger.debug("Processing successful templates build.")
- filemap = slave_status.get('filemap')
- d = self._readTarball(queue_item, filemap, logger)
- d.addCallback(got_tarball)
- d.addCallback(clean_slave)
- return d
-
- return clean_slave(None)
+ def build_info_stored(ignored):
+ if build_status == 'OK':
+ self.setBuildStatus(BuildStatus.UPLOADING)
+ logger.debug("Processing successful templates build.")
+ filemap = slave_status.get('filemap')
+ d = self._readTarball(queue_item, filemap, logger)
+ d.addCallback(got_tarball)
+ d.addCallback(clean_slave)
+ return d
+
+ self.setBuildStatus(BuildStatus.FAILEDTOBUILD)
+ return clean_slave(None)
+
+ d = self.storeBuildInfo(self, queue_item, build_status)
+ d.addCallback(build_info_stored)
+ return d
+
=== modified file 'utilities/make-lp-user'
--- utilities/make-lp-user 2011-07-11 03:49:12 +0000
+++ utilities/make-lp-user 2011-10-17 15:21:59 +0000
@@ -44,6 +44,7 @@
from canonical.launchpad.interfaces.gpghandler import IGPGHandler
from canonical.launchpad.scripts import execute_zcml_for_scripts
+from canonical.lazr.timeout import set_default_timeout_function
from lp.registry.interfaces.gpg import GPGKeyAlgorithm, IGPGKeySet
from lp.registry.interfaces.person import IPersonSet
from lp.registry.interfaces.ssh import ISSHKeySet
@@ -57,6 +58,7 @@
DEFAULT_PASSWORD = 'test'
factory = LaunchpadObjectFactory()
+set_default_timeout_function(lambda: 100)
def make_person(username, email):
"""Create and return a person with the given username.
Follow ups
