launchpad-dev team mailing list archive

[Danilo Šegan <danilo@xxxxxxxxxxxxx>]

У пет, 07. 08 2009. у 11:04 +0700, Stuart Bishop пише:

> There are not 40 other revisions. A theoretical 0.14.1 -> 0.15 is not
> that big a leap.

I must admit to not being completely comfortable with buildout setup for
core pieces like Storm, and I may have misunderstood all the version
names we use there.

2.2.7 branch versions.cfg mentions:

  storm = 0.14salgado-storm-launchpad-288-308

I assumed that was storm 0.14 (rev 283 if I remember correctly) +
revisions 288 and 308.  Storm trunk is already at revision 324.  That's
39 revisions we haven't seen yet in all parts of Launchpad.

If I am mistaken, and 288-308 actually means all revisions between 288
and 308, including them, then it's only 19 revisions of changes to
Storm.  Much better, but still not perfect.

> I thought now was the perfect time to upgrade because we have two
> entire cycles on edge.

Otherwise, it would have been.  But there are two more important things
we are doing right now:

 * we are aiming for a major milestone 3.0, and we do want it well
tested with all the production parameters around them
 * these 'two entire cycles' are supposed to be spent doing mostly UI,
meaning we should not have to devote engineers doing fixes for
incompatible Storm changes

> We are currently running what will be one
> revision away from 0.15 on edge (some compile fixes for Windows need
> to land).

A lot of the integration bits of Launchpad don't run on edge.  For
instance, we still would not have caught bug #408845 with this set-up,
since that happened in only <10% of our poimport script runs.

> I personally would rather run what everyone else will be
> running (Landscape, Online services, external contributers) rather
> than forking our own release containing the bug fixes and features we
> depend on, and trying to support that ourselves as we get burnt by
> bugs that have already been fixed in 0.15. Note that we no longer have
> anyone on the team familiar with the guts of Storm.

I agree.  So, let's fix this right after 3.0 is out the door.

> > What we should be discussing as well is how to avoid having a release of
> > Storm break so spectacularly on us; should we have a more active upgrade
> > policy like we do for bzr, maybe rolling it out on one or two appservers
> > at a time to see what the effect is?
> 
> What exactly broke so spectacularly? We updated one of our
> dependencies (and pulled out a a heap of workarounds). We tested it,
> pushed it out to the testing environments, discovered regressions,
> rolled back, patched, tested again, rolled it out to production.

Well, bug #408845 seems to have been caused by Storm bug #296739.  It
seems it's a bug that can randomly hit our queries and their execution,
and we are talking about hard to reproduce queries at that.  I certainly
hope we won't see many other instances of this particular bug, but what
else has been breaking in peculiar ways?  I'd rather we stabilize on
what we have running on production for 3.0, and considering how close
this you have already gotten us with 0.14 to running Storm trunk, I'd
post-pone the upgrade for 3.0.

> Like all our other dependencies, I'd say stick to best practice of
> testing Launchpad against release candidates, keeping up to date so we
> don't have years worth of changes to deal with at once

Again, I completely agree.  Let's just not do it in the stabilization
and UI cycle for 3.0.

Cheers,
Danilo