launchpad-dev team mailing list archive

[Barry Warsaw <barry@xxxxxxxxxxxxx>]

Several of us have been sprinting the last two weeks on getting  
Launchpad to run on newer versions of Python.  Let me start by giving  
a huge thank you to gary, salgado, maxb, and simon-o for all their  
amazing work, and to flacoste and sinzui for letting us work on this  
almost uninterrupted for two weeks.  It was a pleasure working with  
everyone on this effort.

Here's the high level summary, followed by the gory details.  You can  
follow along on the wiki page we set up <https://dev.launchpad.net/PythonMigrationStatus 
> which also has details about the branches we created.

1. We were successful in passing the full test suite with Python 2.5 +  
the latest Zope Toolkit, on both Karmic (developers) and Hardy (data  
center).

2. Despite valiant efforts, we were not able to complete the migration  
to Python 2.6.

With these results, we are confident that we can land the work to get  
us on Python 2.5 + ZTK for the Karmic cycle, and we will re-address  
Python 2.6 for the Lucid cycle.  In order to keep things smooth during  
the Karmic release, we won't land any of this work next week, but will  
land it to edge/devel at the beginning of the next Launchpad cycle  
after testing on staging.  Gary will pay the price for any bit-rot  
between now and then.

Even though we did not reach our ultimate goal, the sprint was a huge  
success.  We devs will benefit greatly from being on Python 2.5 and we  
really have paved the way for Python 2.6 support.  I'm very optimistic  
that with another week's work we could be on 2.6.  The right time to  
spend that week is when the data center plans its upgrade to the next  
LTS.

More details:

- We completed the already-begun work of getting Launchpad to build,  
test, and run smoothly on Karmic, and many of these changes have  
already landed on devel.  They primarily had to do with package  
conflicts between site-packages and eggs (e.g. lazr), but also many  
other fixes.

- We got the tests passing on Python 2.5.

- Starting with the 2.5 branch, we upgraded to the newest versions of  
the Zope Toolkit packages <http://docs.zope.org/zopetoolkit/ 
index.html> and got the tests passing.

- We upgraded Launchpad to use the latest stable release of Mailman  
2.1.12, which is Python 2.6 (and of course 2.5) compatible.

- Because we're still running Hardy in the DC, we had to make sure it  
worked there too. It does!  The 2.5+ZTK branch works on Hardy just  
like it does on Karmic, although on Hardy we need to install python2.5- 
dev (probably as a dependency to lp-deps).  Salgado reports that his  
test run dropped from 185m to 172m, so it's maybe 7% faster.  That in  
the rough ballpark of numbers I saw on my Karmic dev machine of about  
a 10% improvement, probably almost all due to the performance work  
that went into Python 2.5.

- There are two known failures in the test suite related to openid and  
shipit, but Salgado has branches that fix both of these issues.

- Python 2.6 was tougher, and we simply ran out of time to address the  
issues.  Actually, most of the known ones were due to  
DeprecationWarnings for the md5, sha, and sets modules used both in  
Launchpad and in dependent packages.  Upgrading to ZTK fixed a lot of  
these, but cscvs caused a lot of problems and Twisted caused a lot of  
problems.  You wouldn't think DeprecationWarnings would be so bad, but  
because we often test for clean stdout/stderr coming from  
subprocesses, we just got clobbered in the test suite.  Liberal use of  
-W and warnings.filterwarnings() got us a long way, as did Michael's  
suggestion to use lp:twisted in our sourcecode.  I believe Michael is  
planning to upgrade our Twisted soon.

We can opportunistically land patches to switch us to using hashlib  
instead of md5/sha for all Launchpad code once the 2.5+ZTK lands,  
since 2.5 comes with hashlib.

Ultimately though Python 2.6 migration hit a roadblock we simply  
didn't have time to fix.  The test suite consistently hangs in the  
codehosting test_acceptance tests.  See the wiki page for details  
(scroll to the bottom).  Fortunately, we really didn't waste much  
effort since we only spent the last day and a half at most on 2.6.

We still need to coordinate with the SAs and LOSAs to get the DC  
upgraded the latest branches, so stay tuned for progress on this.  You  
will have to do some chicanery on your development environment once  
the 2.5 stuff lands (mostly relating to sourcecode), so stay tuned for  
further announcements.

That's it.  Please ask questions, Gary/Salgado add anything I missed,  
and if you're really motivated, try to run Launchpad on 2.5+ZTK now.   
It's not that hard and the wiki instructions should be up-to-date.

Cheers,
-Barry
(on behalf of gary, salgado, maxb, simon-o, barry)

Attachment: PGP.sig
Description: This is a digitally signed message part