fuel-dev team mailing list archive

[Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>]

On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote:
> Hi,
> 
> Digging the issue with Galera, I found that our environments have very
> high RAM utilization which leads to the problem during environment
> deployment. For instance "HA deployment + neutron/GRE" requires almost
> 2.6-2.7 GB during deployment
> (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). I found
> high swap in/swap out usage during deployment with very high load
> average. This creates many sporadic issues with some services. They time
> out in random place making our debugging very hard. I would like to
> review our policy for CI environment and increase RAM (at least for bvt
> tests) to 3 GB.
> 
> --
> Best regards,
> Sergii Golovatiuk,
> Skype #golserge
> IRC #holser
> 
> 

I believe we should do at least the following for our CI jobs and bvt tests:
1) Deployment shortcuts: stop deployment abruptly, if any deployment
blocker has been met, such as something exceeded given # of retries.
That could be done in puppet by overriding 'tries' behavior in exec
provider, or at orchestration layer as well.
2) Load management: collect and automatically analyze atop stats (swap
rates, load average, io waiters) from jenkins slaves and vm nodes while
running the jobs, and stop or freeze some jobs, if some
performance-stopper criteria has been met as well.
3) Do not use debug level logging for CI gates, use it only for bvt tests.

-- 
Best regards,
Bogdan Dobrelya,
Skype #bogdando_at_yahoo.com
Irc #bogdando