fuel-dev team mailing list archive

[Vladimir Kuklin <vkuklin@xxxxxxxxxxxx>]

Roman, we figured it out. The problem was that mysql 5.6 was turning
performance_schema parameter on by default. That lead to the shortage of
memory as MySQL allocated 1 gigabyte at once. Now we modified the template
and performance_schema is turned on only if we have 4 gigabytes of RAM on
the node. All the tests are passing now and we closed all the bugs related
to intermittent failures because of insufficient resources on the node.


On Mon, Aug 25, 2014 at 10:58 PM, Roman Alekseenkov <
ralekseenkov@xxxxxxxxxxxx> wrote:

> Team,
>
> Shall we settle on a conclusion? If we need more memory for a working HA,
> let's accept this fact and move on.
>
> I believe we should define a set of supported scenarios for the
> VirtualBox, update our scripts, update release notes and also change quick
> start on Mirantis portal. I filed
> https://bugs.launchpad.net/fuel/+bug/1361331 to track that.
>
> My suggestion is to introduce something along the lines of:
>
>    1. config_4GB.sh - non-HA, 1 controller, 1 compute
>    2. config_8GB.sh - non-HA, 1 controller, 3 compute nodes
>    3. config_16GB.sh - HA, 3 controllers, 1 compute
>
> If we want to go beyond that, there should be an answer in form of
> deploying OVA/OVF into vCenter and utilizing more compute power from a set
> of ESX hosts.
>
> Thanks,
> Roman
>
>
> On Fri, Aug 22, 2014 at 1:09 AM, Meg McRoberts <mmcroberts@xxxxxxxxxxxx>
> wrote:
>
>> We should check with our training people to see what they need.  For
>> example, any
>> reasonable evaluation of MongoDB (used for Ceilometer) or Zabbix probably
>> requires
>> a lot more resources than people are going to have on their personal
>> machine, but it
>> would be nice if they could configure some sort of minimally-functional
>> version of these
>> on a smaller system just for training purposes.
>>
>>
>> On Fri, Aug 22, 2014 at 12:12 AM, Sergii Golovatiuk <
>> sgolovatiuk@xxxxxxxxxxxx> wrote:
>>
>>> Hi,
>>>
>>> I would say we have to update release notes. Even now it's barely
>>> possible to unleash Fuel power on laptop with 8GB of RAM. With 16Gb you may
>>> have up to 5 VMs simulating HA environment. If we limit VM RAM to 2.5 user
>>> may have 5x2.5+1(master node)=13.5 GB which should be enough for demo
>>> purposes.
>>>
>>>
>>> --
>>> Best regards,
>>> Sergii Golovatiuk,
>>> Skype #golserge
>>> IRC #holser
>>>
>>>
>>> On Fri, Aug 22, 2014 at 3:06 AM, Christopher Aedo <caedo@xxxxxxxxxxxx>
>>> wrote:
>>>
>>>> In my opinion it's reasonable to limit expectations on what you can
>>>> evaluate on virtualbox, and provide appropriate caveats.  The truth is
>>>> that making a virtual machine do everything it needs to do to stay
>>>> synced in real-time with two or more other virtual machines requires
>>>> resources.  Evaluating Fuel as a deployment tool under virtualbox
>>>> makes sense, but expecting to get a legitimate evaluation of working
>>>> (and performant) HA under a severely restricted environment doesn't
>>>> seem reasonable.
>>>>
>>>> Would a legitimate customer expect to be able to do a thorough
>>>> evaluation of MOS without using anything more than a single machine
>>>> with a moderate amount of ram?
>>>>
>>>> -Christopher
>>>>
>>>> On Thu, Aug 21, 2014 at 4:59 PM, David Easter <deaster@xxxxxxxxxxxx>
>>>> wrote:
>>>> > It has already became nearly impossible to run the default virtualbox
>>>> > environment (with a cluster size of 3 nodes) on an 8GB system.  It
>>>> would
>>>> > become challenging if we begin to tax a 16GB system to do this now…
>>>> >
>>>> > Are there alternatives or specific tunings that can be done so that
>>>> the
>>>> > sizing for a virtualbox trial is different than what is needed for
>>>> > production or physical server testing?
>>>> >
>>>> > Thanks,
>>>> >
>>>> > - David J. Easter
>>>> >   Director of Product Management,   Mirantis, Inc.
>>>> >
>>>> > From: "ralekseenkov@xxxxxxxxxxxx" <ralekseenkov@xxxxxxxxxxxx>
>>>> > Date: Thursday, August 21, 2014 at 4:29 PM
>>>> > To: Sergii Golovatiuk <sgolovatiuk@xxxxxxxxxxxx>, David Easter
>>>> > <deaster@xxxxxxxxxxxx>
>>>> > Cc: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>, "
>>>> fuel-dev@xxxxxxxxxxxxxxxxxxx"
>>>> > <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>> > Subject: Re: [Fuel-dev] RAM issues with environments
>>>> >
>>>> > David,
>>>> >
>>>> > FYI - it will impact VirtualBox trials. 3 controllers x 3GB = 9 GB
>>>> RAM only
>>>> > for controllers if you run HA.
>>>> >
>>>> > Roman
>>>> >
>>>> >
>>>> > On Thu, Aug 21, 2014 at 5:22 AM, Sergii Golovatiuk
>>>> > <sgolovatiuk@xxxxxxxxxxxx> wrote:
>>>> >>
>>>> >> +1 for not using debug at CI gates (only BVT)
>>>> >> +1 for adding atop to our builds as it helps to understand what was
>>>> wrong
>>>> >> at that particular time
>>>> >> +1 for increasing RAM (though we'll try to tune rabbit and Galera).
>>>> 3GB
>>>> >> should be enough so we'll be able to run up to 2 environments on
>>>> 32GB RAM
>>>> >> servers.
>>>> >>
>>>> >> --
>>>> >> Best regards,
>>>> >> Sergii Golovatiuk,
>>>> >> Skype #golserge
>>>> >> IRC #holser
>>>> >>
>>>> >>
>>>> >> On Thu, Aug 21, 2014 at 1:25 PM, Bogdan Dobrelya <
>>>> bdobrelia@xxxxxxxxxxxx>
>>>> >> wrote:
>>>> >>>
>>>> >>> On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote:
>>>> >>> > Hi,
>>>> >>> >
>>>> >>> > Digging the issue with Galera, I found that our environments have
>>>> very
>>>> >>> > high RAM utilization which leads to the problem during environment
>>>> >>> > deployment. For instance "HA deployment + neutron/GRE" requires
>>>> almost
>>>> >>> > 2.6-2.7 GB during deployment
>>>> >>> > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). I
>>>> found
>>>> >>> > high swap in/swap out usage during deployment with very high load
>>>> >>> > average. This creates many sporadic issues with some services.
>>>> They
>>>> >>> > time
>>>> >>> > out in random place making our debugging very hard. I would like
>>>> to
>>>> >>> > review our policy for CI environment and increase RAM (at least
>>>> for bvt
>>>> >>> > tests) to 3 GB.
>>>> >>> >
>>>> >>> > --
>>>> >>> > Best regards,
>>>> >>> > Sergii Golovatiuk,
>>>> >>> > Skype #golserge
>>>> >>> > IRC #holser
>>>> >>> >
>>>> >>> >
>>>> >>>
>>>> >>> I believe we should do at least the following for our CI jobs and
>>>> bvt
>>>> >>> tests:
>>>> >>> 1) Deployment shortcuts: stop deployment abruptly, if any deployment
>>>> >>> blocker has been met, such as something exceeded given # of retries.
>>>> >>> That could be done in puppet by overriding 'tries' behavior in exec
>>>> >>> provider, or at orchestration layer as well.
>>>> >>> 2) Load management: collect and automatically analyze atop stats
>>>> (swap
>>>> >>> rates, load average, io waiters) from jenkins slaves and vm nodes
>>>> while
>>>> >>> running the jobs, and stop or freeze some jobs, if some
>>>> >>> performance-stopper criteria has been met as well.
>>>> >>> 3) Do not use debug level logging for CI gates, use it only for bvt
>>>> >>> tests.
>>>> >>>
>>>> >>> --
>>>> >>> Best regards,
>>>> >>> Bogdan Dobrelya,
>>>> >>> Skype #bogdando_at_yahoo.com
>>>> >>> Irc #bogdando
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Mailing list: https://launchpad.net/~fuel-dev
>>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >> More help   : https://help.launchpad.net/ListHelp
>>>> >>
>>>> >
>>>> >
>>>> > --
>>>> > Mailing list: https://launchpad.net/~fuel-dev
>>>> > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> > Unsubscribe : https://launchpad.net/~fuel-dev
>>>> > More help   : https://help.launchpad.net/ListHelp
>>>> >
>>>>
>>>
>>>
>>> --
>>> Mailing list: https://launchpad.net/~fuel-dev
>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
45bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com <http://www.mirantis.ru/>
www.mirantis.ru
vkuklin@xxxxxxxxxxxx