launchpad-dev team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

On Fri, Aug 7, 2009 at 5:06 PM, Danilo Šegan<danilo@xxxxxxxxxxxxx> wrote:

> У пет, 07. 08 2009. у 11:04 +0700, Stuart Bishop пише:
>
>> There are not 40 other revisions. A theoretical 0.14.1 -> 0.15 is not
>> that big a leap.
>
> I must admit to not being completely comfortable with buildout setup for
> core pieces like Storm, and I may have misunderstood all the version
> names we use there.
>
> 2.2.7 branch versions.cfg mentions:
>
>  storm = 0.14salgado-storm-launchpad-288-308

On launchpad/devel:

storm = 0.14trunk-321

> I assumed that was storm 0.14 (rev 283 if I remember correctly) +
> revisions 288 and 308.  Storm trunk is already at revision 324.  That's
> 39 revisions we haven't seen yet in all parts of Launchpad.

> If I am mistaken, and 288-308 actually means all revisions between 288
> and 308, including them, then it's only 19 revisions of changes to
> Storm.  Much better, but still not perfect.

I don't know exactly what is in that branch - Salgado landed that one.


>> I thought now was the perfect time to upgrade because we have two
>> entire cycles on edge.
>
> Otherwise, it would have been.  But there are two more important things
> we are doing right now:
>
>  * we are aiming for a major milestone 3.0, and we do want it well
> tested with all the production parameters around them

Sure. So lets test it with 0.15 rather than some fork we have
assembled ourselves. Lots of new UI code means lots of new database
stuff too as lists get sorted differently, new reports are created,
whole new searches implemented. Should all that new code be targeted
at the Storm branch with the most bug fixes, or the branch that
happens to have the bug fixes that have bitten us in the past?

>  * these 'two entire cycles' are supposed to be spent doing mostly UI,
> meaning we should not have to devote engineers doing fixes for
> incompatible Storm changes

You assuming there need to be changes for incompatible Storm changes.
If there are incompatible Storm changes, we could make the decision on
rolling back then. Your also assuming that our 0.14.x fork will
somehow be less buggy are more stable than 0.15. I don't think that is
the case, and we will be on our own if we tickle these bugs.


>> We are currently running what will be one
>> revision away from 0.15 on edge (some compile fixes for Windows need
>> to land).
>
> A lot of the integration bits of Launchpad don't run on edge.  For
> instance, we still would not have caught bug #408845 with this set-up,
> since that happened in only <10% of our poimport script runs.

That bug is an interesting one to cite - Salgado reported it last
year. It is a bug that has been with us since the initial Storm
migration I think - it is unrelated to Storm updates.


>> I personally would rather run what everyone else will be
>> running (Landscape, Online services, external contributers) rather
>> than forking our own release containing the bug fixes and features we
>> depend on, and trying to support that ourselves as we get burnt by
>> bugs that have already been fixed in 0.15. Note that we no longer have
>> anyone on the team familiar with the guts of Storm.
>
> I agree.  So, let's fix this right after 3.0 is out the door.

You agree we should support our own fork and be burnt by bugs that
have been fixed in 0.15? That sounds like an unnecessary hindrance to
getting 3.0 out of the door.


>> > What we should be discussing as well is how to avoid having a release of
>> > Storm break so spectacularly on us; should we have a more active upgrade
>> > policy like we do for bzr, maybe rolling it out on one or two appservers
>> > at a time to see what the effect is?
>>
>> What exactly broke so spectacularly? We updated one of our
>> dependencies (and pulled out a a heap of workarounds). We tested it,
>> pushed it out to the testing environments, discovered regressions,
>> rolled back, patched, tested again, rolled it out to production.

> Well, bug #408845 seems to have been caused by Storm bug #296739.  It
> seems it's a bug that can randomly hit our queries and their execution,
> and we are talking about hard to reproduce queries at that.  I certainly
> hope we won't see many other instances of this particular bug, but what
> else has been breaking in peculiar ways?  I'd rather we stabilize on
> what we have running on production for 3.0, and considering how close
> this you have already gotten us with 0.14 to running Storm trunk, I'd
> post-pone the upgrade for 3.0.

So as above, Storm Bug #296739 has been with us a long time. If
anything, Storm upgrades might have fixed it (like it fixed other
bugs, allowing us to remove workarounds and hacks). So this is an
example of bad habits we are trying to break. Instead of fixing Storm,
we have put a work around in place so the bug is still there waiting
to bite us and others in the future. And it did - it bit Salgado last
year, and it bit you this year. If we had been more proactive about
tracking upstream and fixing things upstream, your pain might have
been avoided.

>> Like all our other dependencies, I'd say stick to best practice of
>> testing Launchpad against release candidates, keeping up to date so we
>> don't have years worth of changes to deal with at once
>
> Again, I completely agree.  Let's just not do it in the stabilization
> and UI cycle for 3.0.

>From my perspective, switching to a genuine Storm release is stabilization.





-- 
Stuart Bishop <stuart@xxxxxxxxxxxxxxxx>
http://www.stuartbishop.net/