fuel-dev team mailing list archive

[Sergii Golovatiuk <sgolovatiuk@xxxxxxxxxxxx>]

+1 for not using debug at CI gates (only BVT)
+1 for adding atop to our builds as it helps to understand what was wrong
at that particular time
+1 for increasing RAM (though we'll try to tune rabbit and Galera). 3GB
should be enough so we'll be able to run up to 2 environments on 32GB RAM
servers.

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser


On Thu, Aug 21, 2014 at 1:25 PM, Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>
wrote:

> On 08/21/2014 12:41 PM, Sergii Golovatiuk wrote:
> > Hi,
> >
> > Digging the issue with Galera, I found that our environments have very
> > high RAM utilization which leads to the problem during environment
> > deployment. For instance "HA deployment + neutron/GRE" requires almost
> > 2.6-2.7 GB during deployment
> > (corosync+mysql+puppet++rabbit+neutron+ovs+openstack services). I found
> > high swap in/swap out usage during deployment with very high load
> > average. This creates many sporadic issues with some services. They time
> > out in random place making our debugging very hard. I would like to
> > review our policy for CI environment and increase RAM (at least for bvt
> > tests) to 3 GB.
> >
> > --
> > Best regards,
> > Sergii Golovatiuk,
> > Skype #golserge
> > IRC #holser
> >
> >
>
> I believe we should do at least the following for our CI jobs and bvt
> tests:
> 1) Deployment shortcuts: stop deployment abruptly, if any deployment
> blocker has been met, such as something exceeded given # of retries.
> That could be done in puppet by overriding 'tries' behavior in exec
> provider, or at orchestration layer as well.
> 2) Load management: collect and automatically analyze atop stats (swap
> rates, load average, io waiters) from jenkins slaves and vm nodes while
> running the jobs, and stop or freeze some jobs, if some
> performance-stopper criteria has been met as well.
> 3) Do not use debug level logging for CI gates, use it only for bvt tests.
>
> --
> Best regards,
> Bogdan Dobrelya,
> Skype #bogdando_at_yahoo.com
> Irc #bogdando
>