← Back to team overview

fuel-dev team mailing list archive

Re: RAM issues with environments

 

We should check with our training people to see what they need.  For
example, any
reasonable evaluation of MongoDB (used for Ceilometer) or Zabbix probably
requires
a lot more resources than people are going to have on their personal
machine, but it
would be nice if they could configure some sort of minimally-functional
version of these
on a smaller system just for training purposes.


On Fri, Aug 22, 2014 at 12:12 AM, Sergii Golovatiuk <
sgolovatiuk@xxxxxxxxxxxx> wrote:

> Hi,
>
> I would say we have to update release notes. Even now it's barely possible
> to unleash Fuel power on laptop with 8GB of RAM. With 16Gb you may have up
> to 5 VMs simulating HA environment. If we limit VM RAM to 2.5 user may have
> 5x2.5+1(master node)=13.5 GB which should be enough for demo purposes.
>
>
> --
> Best regards,
> Sergii Golovatiuk,
> Skype #golserge
> IRC #holser
>
>
> On Fri, Aug 22, 2014 at 3:06 AM, Christopher Aedo <caedo@xxxxxxxxxxxx>
> wrote:
>
>> In my opinion it's reasonable to limit expectations on what you can
>> evaluate on virtualbox, and provide appropriate caveats.  The truth is
>> that making a virtual machine do everything it needs to do to stay
>> synced in real-time with two or more other virtual machines requires
>> resources.  Evaluating Fuel as a deployment tool under virtualbox
>> makes sense, but expecting to get a legitimate evaluation of working
>> (and performant) HA under a severely restricted environment doesn't
>> seem reasonable.
>>
>> Would a legitimate customer expect to be able to do a thorough
>> evaluation of MOS without using anything more than a single machine
>> with a moderate amount of ram?
>>
>> -Christopher
>>
>> On Thu, Aug 21, 2014 at 4:59 PM, David Easter <deaster@xxxxxxxxxxxx>
>> wrote:
>> > It has already became nearly impossible to run the default virtualbox
>> > environment (with a cluster size of 3 nodes) on an 8GB system.  It would
>> > become challenging if we begin to tax a 16GB system to do this now…
>> >
>> > Are there alternatives or specific tunings that can be done so that the
>> > sizing for a virtualbox trial is different than what is needed for
>> > production or physical server testing?
>> >
>> > Thanks,
>> >
>> > - David J. Easter
>> >   Director of Product Management,   Mirantis, Inc.
>> >
>> > From: "ralekseenkov@xxxxxxxxxxxx" <ralekseenkov@xxxxxxxxxxxx>
>> > Date: Thursday, August 21, 2014 at 4:29 PM
>> > To: Sergii Golovatiuk <sgolovatiuk@xxxxxxxxxxxx>, David Easter
>> > <deaster@xxxxxxxxxxxx>
>> > Cc: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>, "
>> fuel-dev@xxxxxxxxxxxxxxxxxxx"
>> > <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>> > Subject: Re: [Fuel-dev] RAM issues with environments
>> >
>> > David,
>> >
>> > FYI - it will impact VirtualBox trials. 3 controllers x 3GB = 9 GB RAM
>> only
>> > for controllers if you run HA.
>> >
>> > Roman
>> >
>> >
>> > On Thu, Aug 21, 2014 at 5:22 AM, Sergii Golovatiuk
>> > <sgolovatiuk@xxxxxxxxxxxx> wrote:
>> >>
>> >> +1 for not using debug at CI gates (only BVT)
>> >> +1 for adding atop to our builds as it helps to understand what was
>> wrong
>> >> at that particular time
>> >> +1 for increasing RAM (though we'll try to tune rabbit and Galera). 3GB
>> >> should be enough so we'll be able to run up to 2 environments on 32GB
>> RAM
>> >> servers.
>> >>
>> >> --
>> >> Best regards,
>> >> Sergii Golovatiuk,
>> >> Skype #golserge
>> >> IRC #holser
>> >>
>> >>
>> >> On Thu, Aug 21, 2014 at 1:25 PM, Bogdan Dobrelya <
>> bdobrelia@xxxxxxxxxxxx>
>> >> wrote:
>> >>>
>> >>> On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote:
>> >>> > Hi,
>> >>> >
>> >>> > Digging the issue with Galera, I found that our environments have
>> very
>> >>> > high RAM utilization which leads to the problem during environment
>> >>> > deployment. For instance "HA deployment + neutron/GRE" requires
>> almost
>> >>> > 2.6-2.7 GB during deployment
>> >>> > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). I
>> found
>> >>> > high swap in/swap out usage during deployment with very high load
>> >>> > average. This creates many sporadic issues with some services. They
>> >>> > time
>> >>> > out in random place making our debugging very hard. I would like to
>> >>> > review our policy for CI environment and increase RAM (at least for
>> bvt
>> >>> > tests) to 3 GB.
>> >>> >
>> >>> > --
>> >>> > Best regards,
>> >>> > Sergii Golovatiuk,
>> >>> > Skype #golserge
>> >>> > IRC #holser
>> >>> >
>> >>> >
>> >>>
>> >>> I believe we should do at least the following for our CI jobs and bvt
>> >>> tests:
>> >>> 1) Deployment shortcuts: stop deployment abruptly, if any deployment
>> >>> blocker has been met, such as something exceeded given # of retries.
>> >>> That could be done in puppet by overriding 'tries' behavior in exec
>> >>> provider, or at orchestration layer as well.
>> >>> 2) Load management: collect and automatically analyze atop stats (swap
>> >>> rates, load average, io waiters) from jenkins slaves and vm nodes
>> while
>> >>> running the jobs, and stop or freeze some jobs, if some
>> >>> performance-stopper criteria has been met as well.
>> >>> 3) Do not use debug level logging for CI gates, use it only for bvt
>> >>> tests.
>> >>>
>> >>> --
>> >>> Best regards,
>> >>> Bogdan Dobrelya,
>> >>> Skype #bogdando_at_yahoo.com
>> >>> Irc #bogdando
>> >>
>> >>
>> >>
>> >> --
>> >> Mailing list: https://launchpad.net/~fuel-dev
>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>> >> More help   : https://help.launchpad.net/ListHelp
>> >>
>> >
>> >
>> > --
>> > Mailing list: https://launchpad.net/~fuel-dev
>> > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> > Unsubscribe : https://launchpad.net/~fuel-dev
>> > More help   : https://help.launchpad.net/ListHelp
>> >
>>
>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>

Follow ups

References