fuel-dev team mailing list archive

[Mike Scherbakov <mscherbakov@xxxxxxxxxxxx>]

There is a bug: https://bugs.launchpad.net/fuel/+bug/1358345
and patch on review: https://review.openstack.org/#/c/115954/

Vladimir, please participate there.

Thanks,


On Mon, Aug 25, 2014 at 11:05 PM, Vladimir Kuklin <vkuklin@xxxxxxxxxxxx>
wrote:

> Roman, we figured it out. The problem was that mysql 5.6 was turning
> performance_schema parameter on by default. That lead to the shortage of
> memory as MySQL allocated 1 gigabyte at once. Now we modified the template
> and performance_schema is turned on only if we have 4 gigabytes of RAM on
> the node. All the tests are passing now and we closed all the bugs related
> to intermittent failures because of insufficient resources on the node.
>
>
> On Mon, Aug 25, 2014 at 10:58 PM, Roman Alekseenkov <
> ralekseenkov@xxxxxxxxxxxx> wrote:
>
>> Team,
>>
>> Shall we settle on a conclusion? If we need more memory for a working HA,
>> let's accept this fact and move on.
>>
>> I believe we should define a set of supported scenarios for the
>> VirtualBox, update our scripts, update release notes and also change quick
>> start on Mirantis portal. I filed
>> https://bugs.launchpad.net/fuel/+bug/1361331 to track that.
>>
>> My suggestion is to introduce something along the lines of:
>>
>>    1. config_4GB.sh - non-HA, 1 controller, 1 compute
>>    2. config_8GB.sh - non-HA, 1 controller, 3 compute nodes
>>    3. config_16GB.sh - HA, 3 controllers, 1 compute
>>
>> If we want to go beyond that, there should be an answer in form of
>> deploying OVA/OVF into vCenter and utilizing more compute power from a set
>> of ESX hosts.
>>
>> Thanks,
>> Roman
>>
>>
>> On Fri, Aug 22, 2014 at 1:09 AM, Meg McRoberts <mmcroberts@xxxxxxxxxxxx>
>> wrote:
>>
>>> We should check with our training people to see what they need.  For
>>> example, any
>>> reasonable evaluation of MongoDB (used for Ceilometer) or Zabbix
>>> probably requires
>>> a lot more resources than people are going to have on their personal
>>> machine, but it
>>> would be nice if they could configure some sort of minimally-functional
>>> version of these
>>> on a smaller system just for training purposes.
>>>
>>>
>>> On Fri, Aug 22, 2014 at 12:12 AM, Sergii Golovatiuk <
>>> sgolovatiuk@xxxxxxxxxxxx> wrote:
>>>
>>>> Hi,
>>>>
>>>> I would say we have to update release notes. Even now it's barely
>>>> possible to unleash Fuel power on laptop with 8GB of RAM. With 16Gb you may
>>>> have up to 5 VMs simulating HA environment. If we limit VM RAM to 2.5 user
>>>> may have 5x2.5+1(master node)=13.5 GB which should be enough for demo
>>>> purposes.
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Sergii Golovatiuk,
>>>> Skype #golserge
>>>> IRC #holser
>>>>
>>>>
>>>> On Fri, Aug 22, 2014 at 3:06 AM, Christopher Aedo <caedo@xxxxxxxxxxxx>
>>>> wrote:
>>>>
>>>>> In my opinion it's reasonable to limit expectations on what you can
>>>>> evaluate on virtualbox, and provide appropriate caveats.  The truth is
>>>>> that making a virtual machine do everything it needs to do to stay
>>>>> synced in real-time with two or more other virtual machines requires
>>>>> resources.  Evaluating Fuel as a deployment tool under virtualbox
>>>>> makes sense, but expecting to get a legitimate evaluation of working
>>>>> (and performant) HA under a severely restricted environment doesn't
>>>>> seem reasonable.
>>>>>
>>>>> Would a legitimate customer expect to be able to do a thorough
>>>>> evaluation of MOS without using anything more than a single machine
>>>>> with a moderate amount of ram?
>>>>>
>>>>> -Christopher
>>>>>
>>>>> On Thu, Aug 21, 2014 at 4:59 PM, David Easter <deaster@xxxxxxxxxxxx>
>>>>> wrote:
>>>>> > It has already became nearly impossible to run the default virtualbox
>>>>> > environment (with a cluster size of 3 nodes) on an 8GB system.  It
>>>>> would
>>>>> > become challenging if we begin to tax a 16GB system to do this now…
>>>>> >
>>>>> > Are there alternatives or specific tunings that can be done so that
>>>>> the
>>>>> > sizing for a virtualbox trial is different than what is needed for
>>>>> > production or physical server testing?
>>>>> >
>>>>> > Thanks,
>>>>> >
>>>>> > - David J. Easter
>>>>> >   Director of Product Management,   Mirantis, Inc.
>>>>> >
>>>>> > From: "ralekseenkov@xxxxxxxxxxxx" <ralekseenkov@xxxxxxxxxxxx>
>>>>> > Date: Thursday, August 21, 2014 at 4:29 PM
>>>>> > To: Sergii Golovatiuk <sgolovatiuk@xxxxxxxxxxxx>, David Easter
>>>>> > <deaster@xxxxxxxxxxxx>
>>>>> > Cc: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>, "
>>>>> fuel-dev@xxxxxxxxxxxxxxxxxxx"
>>>>> > <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> > Subject: Re: [Fuel-dev] RAM issues with environments
>>>>> >
>>>>> > David,
>>>>> >
>>>>> > FYI - it will impact VirtualBox trials. 3 controllers x 3GB = 9 GB
>>>>> RAM only
>>>>> > for controllers if you run HA.
>>>>> >
>>>>> > Roman
>>>>> >
>>>>> >
>>>>> > On Thu, Aug 21, 2014 at 5:22 AM, Sergii Golovatiuk
>>>>> > <sgolovatiuk@xxxxxxxxxxxx> wrote:
>>>>> >>
>>>>> >> +1 for not using debug at CI gates (only BVT)
>>>>> >> +1 for adding atop to our builds as it helps to understand what was
>>>>> wrong
>>>>> >> at that particular time
>>>>> >> +1 for increasing RAM (though we'll try to tune rabbit and Galera).
>>>>> 3GB
>>>>> >> should be enough so we'll be able to run up to 2 environments on
>>>>> 32GB RAM
>>>>> >> servers.
>>>>> >>
>>>>> >> --
>>>>> >> Best regards,
>>>>> >> Sergii Golovatiuk,
>>>>> >> Skype #golserge
>>>>> >> IRC #holser
>>>>> >>
>>>>> >>
>>>>> >> On Thu, Aug 21, 2014 at 1:25 PM, Bogdan Dobrelya <
>>>>> bdobrelia@xxxxxxxxxxxx>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote:
>>>>> >>> > Hi,
>>>>> >>> >
>>>>> >>> > Digging the issue with Galera, I found that our environments
>>>>> have very
>>>>> >>> > high RAM utilization which leads to the problem during
>>>>> environment
>>>>> >>> > deployment. For instance "HA deployment + neutron/GRE" requires
>>>>> almost
>>>>> >>> > 2.6-2.7 GB during deployment
>>>>> >>> > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services).
>>>>> I found
>>>>> >>> > high swap in/swap out usage during deployment with very high load
>>>>> >>> > average. This creates many sporadic issues with some services.
>>>>> They
>>>>> >>> > time
>>>>> >>> > out in random place making our debugging very hard. I would like
>>>>> to
>>>>> >>> > review our policy for CI environment and increase RAM (at least
>>>>> for bvt
>>>>> >>> > tests) to 3 GB.
>>>>> >>> >
>>>>> >>> > --
>>>>> >>> > Best regards,
>>>>> >>> > Sergii Golovatiuk,
>>>>> >>> > Skype #golserge
>>>>> >>> > IRC #holser
>>>>> >>> >
>>>>> >>> >
>>>>> >>>
>>>>> >>> I believe we should do at least the following for our CI jobs and
>>>>> bvt
>>>>> >>> tests:
>>>>> >>> 1) Deployment shortcuts: stop deployment abruptly, if any
>>>>> deployment
>>>>> >>> blocker has been met, such as something exceeded given # of
>>>>> retries.
>>>>> >>> That could be done in puppet by overriding 'tries' behavior in exec
>>>>> >>> provider, or at orchestration layer as well.
>>>>> >>> 2) Load management: collect and automatically analyze atop stats
>>>>> (swap
>>>>> >>> rates, load average, io waiters) from jenkins slaves and vm nodes
>>>>> while
>>>>> >>> running the jobs, and stop or freeze some jobs, if some
>>>>> >>> performance-stopper criteria has been met as well.
>>>>> >>> 3) Do not use debug level logging for CI gates, use it only for bvt
>>>>> >>> tests.
>>>>> >>>
>>>>> >>> --
>>>>> >>> Best regards,
>>>>> >>> Bogdan Dobrelya,
>>>>> >>> Skype #bogdando_at_yahoo.com
>>>>> >>> Irc #bogdando
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Mailing list: https://launchpad.net/~fuel-dev
>>>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >> More help   : https://help.launchpad.net/ListHelp
>>>>> >>
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Mailing list: https://launchpad.net/~fuel-dev
>>>>> > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> > Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> > More help   : https://help.launchpad.net/ListHelp
>>>>> >
>>>>>
>>>>
>>>>
>>>> --
>>>> Mailing list: https://launchpad.net/~fuel-dev
>>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>> --
>>> Mailing list: https://launchpad.net/~fuel-dev
>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 45bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin@xxxxxxxxxxxx
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Mike Scherbakov
#mihgen