fuel-dev team mailing list archive

[Roman Alekseenkov <ralekseenkov@xxxxxxxxxxxx>]

David,

FYI - it will impact VirtualBox trials. 3 controllers x 3GB = 9 GB RAM only
for controllers if you run HA.

Roman


On Thu, Aug 21, 2014 at 5:22 AM, Sergii Golovatiuk <sgolovatiuk@xxxxxxxxxxxx
> wrote:

> +1 for not using debug at CI gates (only BVT)
> +1 for adding atop to our builds as it helps to understand what was wrong
> at that particular time
> +1 for increasing RAM (though we'll try to tune rabbit and Galera). 3GB
> should be enough so we'll be able to run up to 2 environments on 32GB RAM
> servers.
>
> --
> Best regards,
> Sergii Golovatiuk,
> Skype #golserge
> IRC #holser
>
>
> On Thu, Aug 21, 2014 at 1:25 PM, Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>
> wrote:
>
>> On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote:
>> > Hi,
>> >
>> > Digging the issue with Galera, I found that our environments have very
>> > high RAM utilization which leads to the problem during environment
>> > deployment. For instance "HA deployment + neutron/GRE" requires almost
>> > 2.6-2.7 GB during deployment
>> > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). I found
>> > high swap in/swap out usage during deployment with very high load
>> > average. This creates many sporadic issues with some services. They time
>> > out in random place making our debugging very hard. I would like to
>> > review our policy for CI environment and increase RAM (at least for bvt
>> > tests) to 3 GB.
>> >
>> > --
>> > Best regards,
>> > Sergii Golovatiuk,
>> > Skype #golserge
>> > IRC #holser
>> >
>> >
>>
>> I believe we should do at least the following for our CI jobs and bvt
>> tests:
>> 1) Deployment shortcuts: stop deployment abruptly, if any deployment
>> blocker has been met, such as something exceeded given # of retries.
>> That could be done in puppet by overriding 'tries' behavior in exec
>> provider, or at orchestration layer as well.
>> 2) Load management: collect and automatically analyze atop stats (swap
>> rates, load average, io waiters) from jenkins slaves and vm nodes while
>> running the jobs, and stop or freeze some jobs, if some
>> performance-stopper criteria has been met as well.
>> 3) Do not use debug level logging for CI gates, use it only for bvt tests.
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Skype #bogdando_at_yahoo.com
>> Irc #bogdando
>>
>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>