fuel-dev team mailing list archive

[Mike Scherbakov <mscherbakov@xxxxxxxxxxxx>]

Yeah 915 runs with 0 failures is really impressive. Especially knowing how
many random issues we had with upgrades.
Commit "Add docker and lxc to ISO" is dated April, 7th in fuel-main repo.
There was a long journey I would say to get such a stability after first
commits, with many engineers participating.

Keep Fuel rock stable!


On Thu, Sep 4, 2014 at 2:23 AM, Roman Alekseenkov <ralekseenkov@xxxxxxxxxxxx
> wrote:

> Evgeniy,
>
> This is great. The patch set looks excellent and thanks for stress testing
> Docker. Very well done!
>
> Thanks,
> Roman
>
>
> On Wed, Sep 3, 2014 at 12:41 PM, Dmitry Borodaenko <
> dborodaenko@xxxxxxxxxxxx> wrote:
>
>> Impressive work Evgeniy! And special kudos for the detailed commit
>> message, an example to be followed!
>>
>> On Wed, Sep 3, 2014 at 10:37 AM, Evgeniy L <eli@xxxxxxxxxxxx> wrote:
>> > Hi,
>> >
>> > I would like to provide status for master node upgrade feature.
>> > We merged a patch [1] which increased stability of upgrade.
>> >
>> > It fixes a lot of problems here is the list of some of them:
>> > * fixed known and unknown raise conditions, e.g. keystone
>> >   db migration interruption
>> > * now we won't have problem with ip duplication, I haven't
>> >   seen the problem, so I cannot say how often it happened,
>> >   but can say that the patch solves the problem in case of upgrade
>> > * the patch twice reduces probability of docker's death during
>> >   the upgrade
>> >
>> > In the last 24 hours I tested the patch on 4 virtual machines,
>> > there were 915 upgrade runs (to reduce the time of upgrade
>> > I enabled only docker engine, which is the most problematic
>> > part of master node upgrade), at the end of the day there
>> > were 0 failed upgrades.
>> >
>> > Thanks,
>> >
>> > [1] https://review.openstack.org/#/c/118387/
>> >
>> >
>> > On Wed, Aug 27, 2014 at 4:05 PM, Matthew Mosesohn <
>> mmosesohn@xxxxxxxxxxxx>
>> > wrote:
>> >>
>> >> Regarding paste.openstack.org, we should look to another pastebin
>> >> provider. It has been giving 500 errors quite nearly consistently for
>> >> me lately and really interferes with my work. We could use pastie.org,
>> >> for example.
>> >>
>> >> On Wed, Aug 27, 2014 at 12:53 PM, Mike Scherbakov
>> >> <mscherbakov@xxxxxxxxxxxx> wrote:
>> >> > OpenStack uses paste.openstack.org all the time, and I've heard
>> issues
>> >> > with
>> >> > how long content is stored there.
>> >> >
>> >> >
>> >> > On Wed, Aug 27, 2014 at 12:45 PM, Aleksandra Fedorova
>> >> > <afedorova@xxxxxxxxxxxx> wrote:
>> >> >>
>> >> >>
>> >> >> I can not find the policy about how long data is stored there, but I
>> >> >> doubt
>> >> >> that pastebin service can be used as a longterm storage. If we don't
>> >> >> want to
>> >> >> lose logs and scripts data, the use of paste.o.o links for bug
>> reports
>> >> >> should be forbidden.
>> >> >>
>> >> >> Launchpad attachments are much more reliable even though less
>> >> >> comfortable
>> >> >> to use.
>> >> >>
>> >> >> On Aug 27, 2014 8:49 AM, "Mike Scherbakov" <
>> mscherbakov@xxxxxxxxxxxx>
>> >> >> wrote:
>> >> >>>
>> >> >>> +1 on updating bug descriptions (not comments) about probability of
>> >> >>> failure, and using paste.openstack.org more often.
>> >> >>>
>> >> >>>
>> >> >>> On Wed, Aug 27, 2014 at 3:41 AM, Dmitry Borodaenko
>> >> >>> <dborodaenko@xxxxxxxxxxxx> wrote:
>> >> >>>>
>> >> >>>> On Tue, Aug 26, 2014 at 12:11 AM, Mike Scherbakov
>> >> >>>> <mscherbakov@xxxxxxxxxxxx> wrote:
>> >> >>>> > "confusing versioning in OpenStack patching" - if we didn't
>> change
>> >> >>>> > puppet
>> >> >>>> > manifests and Fuel/OpenStack reference architecture in next Fuel
>> >> >>>> > versions,
>> >> >>>> > then it would be as simple as patching from 5.0 to 5.1. But it
>> >> >>>> > appeared to
>> >> >>>> > be more complicated system than you would initially think of,
>> so in
>> >> >>>> > general
>> >> >>>> > 5.0.2 may not be equal to 5.1, that's where all things come up.
>> If
>> >> >>>> > we
>> >> >>>> > had
>> >> >>>> > OpenStack upgrades, then we could just say 5.0 -> 6.0 - easy.
>> >> >>>>
>> >> >>>> We may have had technical reasons to make this decision, but it
>> still
>> >> >>>> is confusing and negatively impacts UX. I agree that having an
>> >> >>>> incomplete feature early is better than not having it at all until
>> >> >>>> much later, as long as we don't stop working on it until it's
>> >> >>>> complete
>> >> >>>> and these small but annoying deficiencies are addressed. Our
>> >> >>>> experience with technical debt so far is not very reassuring.
>> >> >>>>
>> >> >>>> > "issues with containers" - we have same issues with everything.
>> >> >>>> > Let's
>> >> >>>> > take
>> >> >>>> > Galera, for example. It's just issues. We can question maturity
>> of
>> >> >>>> > tools we
>> >> >>>> > use, and here I'd agree - we spent too much fixing issues around
>> >> >>>> > Docker. At
>> >> >>>> > the same time, if we were about taking our own journey with
>> LXC, we
>> >> >>>> > would
>> >> >>>> > likely spend even more time inventing our own bicycle.
>> >> >>>>
>> >> >>>> You're assuming that it was just Docker as a piece of software
>> that
>> >> >>>> is
>> >> >>>> the primary cause of all our troubles with Fuel upgrades. Docker
>> is
>> >> >>>> only a small part of the a much large and much more intrusive
>> design
>> >> >>>> decision to use containers for upgrading Fuel (and also the design
>> >> >>>> decision to use a different mechanism based on Puppet for patching
>> >> >>>> OpenStack). I think we should question high-level design decisions
>> >> >>>> like these more often, even after they are implemented.
>> >> >>>>
>> >> >>>> > Also, I'd like to ask everyone to provide
>> >> >>>> > such information in every bug you report if possible (or if get
>> >> >>>> > this
>> >> >>>> > info
>> >> >>>> > later, put comments): in many bug reports it is unclear to
>> >> >>>> > understand
>> >> >>>> > how
>> >> >>>> > severe issue is.
>> >> >>>>
>> >> >>>> I think we should start updating bug description more often, so
>> that
>> >> >>>> you can find a summary of current state of the bug and of all
>> >> >>>> relevant
>> >> >>>> information from the description, without having to scroll through
>> >> >>>> dozens of comments. We should also use paste.openstack.org more
>> >> >>>> heavily and avoid pasting more than 1-2 lines of logs into bug
>> >> >>>> description and comments, also to make it easier to find important
>> >> >>>> bits in bugs history.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Mike Scherbakov
>> >> >>> #mihgen
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Mailing list: https://launchpad.net/~fuel-dev
>> >> >>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> >> >>> Unsubscribe : https://launchpad.net/~fuel-dev
>> >> >>> More help   : https://help.launchpad.net/ListHelp
>> >> >>>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Mike Scherbakov
>> >> > #mihgen
>> >> >
>> >> >
>> >> > --
>> >> > Mailing list: https://launchpad.net/~fuel-dev
>> >> > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> >> > Unsubscribe : https://launchpad.net/~fuel-dev
>> >> > More help   : https://help.launchpad.net/ListHelp
>> >> >
>> >
>> >
>> >
>> > --
>> > Mailing list: https://launchpad.net/~fuel-dev
>> > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> > Unsubscribe : https://launchpad.net/~fuel-dev
>> > More help   : https://help.launchpad.net/ListHelp
>> >
>>
>>
>>
>> --
>> Dmitry Borodaenko
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Mike Scherbakov
#mihgen