fuel-dev team mailing list archive

[Andrey Korolev <akorolev@xxxxxxxxxxxx>]

On Wed, Jun 25, 2014 at 12:00 AM, Meg McRoberts <mmcroberts@xxxxxxxxxxxx> wrote:
> How does one know that ones overcommit ratio is too high?  Are there
> monitoring tools
> that would show CPU utilization for all Compute nodes in the environment?
> And, if one
> set the RAM overcommit rate to something other than 1:1, would  it be easy
> to tell whether
> one was CPU or memory-bound?

I am wondered why everyone repeatedly suggesting memory overcommit
ratio to be equal one, not setting to dynamical characteristic which
1. strictly lower than one,
2. dependent on many parameters, including absolute memory size and
existence of co-resident services.

I had said this before two times, as I can remember :)

>
> I suppose if users were complaining about performance, this would be a good
> place to look,
> but this is certainly not the only factor that could affect VM
> performance...
>
>
> On Tue, Jun 24, 2014 at 12:49 PM, Dmitriy Novakovskiy
> <dnovakovskiy@xxxxxxxxxxxx> wrote:
>>
>> David,
>>
>>> I created an etherpad with the additional requirements
>>> (https://etherpad.openstack.org/p/cpu-overcommit-setting) but don’t have
>>> rights to set it as the URL for the specification.  If you or an admin could
>>> do that, it’d be appreciated.
>>
>>
>> Done - added your etherpad as a full spec to BP:
>> https://blueprints.launchpad.net/fuel/+spec/cpu-overcommit-setting
>>
>>> With these expectations, you would need to have the parameter be on the
>>> Settings tab not (just) the Deployment Wizard.
>>
>>
>> Got it, makes sense. It should be on both screens.
>>
>>>  Also, currently the only way to reactivate the Settings Tab today and
>>> “re-apply” is to reset the environment – I.e. delete the existing
>>> environment and re-deploy with the modified settings/parameters.   That may
>>> be a bit heavy handed for just changing the overcommit value, so we may need
>>> a second blueprint that would enable settings to be changed from Fuel
>>> without deleting and redeploying the environment…
>>
>>
>> Got it. Well, IMO it calls for us to think through some key concepts, what
>> Fuel should/should not be able to do and enforce... Ability to adjust CPU
>> overcommit rate on post-deployment phases will be useful, but not pri-1. I
>> just thought that UI+engine already supports it.
>>
>> ---
>> Regards,
>> Dmitriy
>>
>>
>> On Tue, Jun 24, 2014 at 10:36 PM, David Easter <deaster@xxxxxxxxxxxx>
>> wrote:
>>>
>>> Thanks.  With these expectations, you would need to have the parameter be
>>> on the Settings tab not (just) the Deployment Wizard.  Having it only in the
>>> deployment wizard would mean you couldn’t modify it later and that it
>>> wouldn’t be reflected in the UI.   Also, currently the only way to
>>> reactivate the Settings Tab today and “re-apply” is to reset the environment
>>> – I.e. delete the existing environment and re-deploy with the modified
>>> settings/parameters.   That may be a bit heavy handed for just changing the
>>> overcommit value, so we may need a second blueprint that would enable
>>> settings to be changed from Fuel without deleting and redeploying the
>>> environment…
>>>
>>> I created an etherpad with the additional requirements
>>> (https://etherpad.openstack.org/p/cpu-overcommit-setting) but don’t have
>>> rights to set it as the URL for the specification.  If you or an admin could
>>> do that, it’d be appreciated.
>>>
>>> Thanks,
>>>
>>> - David J. Easter
>>>   Director of Product Management, Mirantis
>>>
>>> From: Dmitriy Novakovskiy <dnovakovskiy@xxxxxxxxxxxx>
>>> Date: Tuesday, June 24, 2014 at 11:54 AM
>>> To: David Easter <deaster@xxxxxxxxxxxx>
>>> Cc: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>, fuel-dev
>>> <fuel-dev@xxxxxxxxxxxxxxxxxxx>, Sergii Golovatiuk <sgolovatiuk@xxxxxxxxxxxx>
>>>
>>> Subject: Re: [Fuel-dev] overcommit rates
>>>
>>> David,
>>>
>>> 1. Yes
>>> 2. Yes, same as other Settings which Fuel can "re-apply" today (syslog
>>> server, scheduler type, etc). As Sergey pointed out, there's no danger in
>>> it, and there should not be any additional technical effort to implement.
>>> The only thing is - we need to add a note on UI and/or in documentation to
>>> explain to user that changing overcommit rate to lower will not
>>> automatically offload Compute nodes that are already over-provisioned,
>>> he/she will still need to migrate excessive VMs away:)
>>> 3. No, no monitoring. Fuel master should be the only source of "truth"
>>> for those settings which are exposed to user on Web UI.
>>>
>>>
>>> ---
>>> Regards,
>>> Dmitriy
>>>
>>>
>>> On Tue, Jun 24, 2014 at 8:35 PM, Sergii Golovatiuk
>>> <sgolovatiuk@xxxxxxxxxxxx> wrote:
>>>>
>>>> Hi,
>>>>
>>>> IMHO,
>>>>
>>>> 1. Yes
>>>> 2. In theory, it's possible to change/update values after deployment
>>>> time. It's not a dangerous operation. If node is full, it won't be selected
>>>> for new VM starts.
>>>> 3. If it's changed manually, it will be reset on next puppet run.
>>>>
>>>> ~Sergii
>>>>
>>>>
>>>> On Tue, Jun 24, 2014 at 5:21 PM, David Easter <deaster@xxxxxxxxxxxx>
>>>> wrote:
>>>>>
>>>>> Dmitriy,
>>>>>
>>>>>   Once set through the UI, I have a few questions on expectations
>>>>> post-deployment:
>>>>>
>>>>> Would the value selected need to be shown within the Fuel UI
>>>>> post-deployment to let the administrator know what value was selected per
>>>>> environment?
>>>>> Is there an expectation that the value can be changed in a running
>>>>> environment, or would this only be expected at deployment time?
>>>>> If the value were changed in the environment manually, would be
>>>>> expected that the Fuel Master Node would need to monitor for such changes?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> - David J. Easter
>>>>>   Director of Product Management, Mirantis
>>>>>
>>>>> From: Dmitriy Novakovskiy <dnovakovskiy@xxxxxxxxxxxx>
>>>>> Date: Tuesday, June 24, 2014 at 4:37 AM
>>>>> To: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>
>>>>> Cc: fuel-dev <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> Subject: Re: [Fuel-dev] overcommit rates
>>>>>
>>>>> All,
>>>>>
>>>>> I've submitted a blueprint on this:
>>>>> https://blueprints.launchpad.net/fuel/+spec/cpu-overcommit-setting
>>>>>
>>>>> Bogdan,
>>>>>
>>>>> Nice and creative approach, but IMO the question of "how many instances
>>>>> to run" and "what overcommit to use" belongs more to planning/architecture
>>>>> workshop/BOM calculator phase. Not only amount of VMs to run on given amount
>>>>> of Compute nodes counts here - also the capability of hypervisor to "juggle"
>>>>> oversubscribed cores (poor in KVM, better in ESXi), the kind of environment
>>>>> (Prod, Dev/Test), the level of tolerance to VM failure, etc. Fuel just need
>>>>> to prompt+allow user to apply decisions made at planning phase to deployment
>>>>> phase.
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Regards,
>>>>> Dmitriy
>>>>>
>>>>>
>>>>> On Tue, Jun 24, 2014 at 2:22 PM, Bogdan Dobrelya
>>>>> <bdobrelia@xxxxxxxxxxxx> wrote:
>>>>>>
>>>>>> On 06/24/2014 01:50 PM, Dmitriy Novakovskiy wrote:
>>>>>> > Andrey, Dmitriy
>>>>>> >
>>>>>> > I've just realized that CPU overcommitment setting is defined not
>>>>>> > for
>>>>>> > Controllers but for Compute nodes, and may differ on per-node basis.
>>>>>> > So,
>>>>>> > the 2nd "harm" example that I exposed a few emails ago (with failed
>>>>>> > and
>>>>>> > redeployed controller resulting in inconsistent overcommitment
>>>>>> > behavior)
>>>>>> > is irrelevant.
>>>>>> >
>>>>>> > With this in mind, 3rd example is even worse :)
>>>>>> >
>>>>>> > 3) User goes through architecture workshop and capacity planning
>>>>>> > exercise, decides on CPU overcommit rates he/she will use for
>>>>>> > OpenStack
>>>>>> > cloud. User deploys the cloud with, let's say, 20+ compute nodes.
>>>>>> > Now,
>>>>>> > w/o CPU overcommit setting on GUI, User will:
>>>>>> > - get a cloud that's inconsistent with decisions made during
>>>>>> > planning
>>>>>> > - be not aware of this - GUI did not point user to a need of
>>>>>> > specifying
>>>>>> > this setting
>>>>>> > - be frustrated and put under risk - if the cloud is deployed with
>>>>>> > real
>>>>>> > workloads the compute nodes will soon become over provisioned and
>>>>>> > unreliable due to default Fuel setting (prod KVM clouds often use
>>>>>> > super-soft overcommit setting, or no overcommit at all)
>>>>>> > - finally - *user will have to MANUALLY specify and maintain
>>>>>> > overcommit
>>>>>> > setting across 20+ compute nodes.*
>>>>>> > *
>>>>>> > *
>>>>>> > Doesn't sound like a good UX from cloud lifecycle management app :)
>>>>>> > -
>>>>>> > Fuel will quickly be ditched in favor of config management tool.
>>>>>> >
>>>>>> > So, my proposal on CPU overcommitment setting is following:
>>>>>> >
>>>>>> > - Expose it in Wizard on hypervisor selection screen
>>>>>> > - To simplify things initially - implement the setting as a
>>>>>> > drop-down
>>>>>> > list, with 2 options:
>>>>>> > -- No CPU overcommitment (1:1)
>>>>>> > -- Light CPU overcommitment (3:1)
>>>>>> > -- Heavy CPU overcommitment (6:1) --- default setting in BOM
>>>>>> > calculator
>>>>>>
>>>>>> Good point. We could as well evaluate the suggested ratio based on the
>>>>>> user outputs and roles allocation, e.g.:
>>>>>> 1) Q(wizard):"How many running instances your cloud has to maintain?"
>>>>>> ->
>>>>>> user specifies 10000
>>>>>> 2) and there are 100 compute nodes assigned in UI
>>>>>>
>>>>>> then,
>>>>>> - the CPU overcommitment should be evaluated as 10000/100 : 1
>>>>>> - if the value exceeds the default maximum (16) for OSt, the warning
>>>>>> should be issued in UI notifications (please provide more compute
>>>>>> nodes
>>>>>> in order to maintain the desired capacity for running instances)
>>>>>>
>>>>>> >
>>>>>> >
>>>>>> > ---
>>>>>> > Regards,
>>>>>> > Dmitriy
>>>>>> >
>>>>>> >
>>>>>> > On Tue, Jun 24, 2014 at 1:23 PM, Andrey Danin <adanin@xxxxxxxxxxxx
>>>>>> > <mailto:adanin@xxxxxxxxxxxx>> wrote:
>>>>>> >
>>>>>> >     I think, we can add to openstack.yaml a special tag "advanced"
>>>>>> > to
>>>>>> >     every advanced parameter, and UI can hide all such parameters
>>>>>> > for
>>>>>> >     every group under the button "show more options". Also we can
>>>>>> > add
>>>>>> >     the "expand all hidden options" button at the top of the tab. I
>>>>>> >     don't think it's hard to implement.
>>>>>> >
>>>>>> >     Let's wait an opinion of UI team.
>>>>>> >
>>>>>> >     On Jun 24, 2014 1:58 PM, "Meg McRoberts"
>>>>>> > <mmcroberts@xxxxxxxxxxxx
>>>>>> >     <mailto:mmcroberts@xxxxxxxxxxxx>> wrote:
>>>>>> >
>>>>>> >         Jesse and others make a very good point.  The Settings
>>>>>> > screen
>>>>>> >         has already become a real mess
>>>>>> >         that needs to be split up and organized a bit.  I'm told
>>>>>> > that
>>>>>> >         the serious restructuring should come
>>>>>> >         after we implement the plug-ins scheme (currently planned
>>>>>> > for
>>>>>> >         6.0).  And each section definitely
>>>>>> >         needs an "Advanced" section for configurations that aren't
>>>>>> > routine.
>>>>>> >
>>>>>> >         So do we wait to restructure the "Settings" screen until the
>>>>>> >         plug-ins scheme is implemented?  I
>>>>>> >         think breaking it up into sub-sections would make it more
>>>>>> > usable
>>>>>> >         in the meantime but I don't know
>>>>>> >         if it's worth the effort to do that ahead of the plug-ins.
>>>>>> >
>>>>>> >
>>>>>> >         On Tue, Jun 24, 2014 at 2:23 AM, Jesse Pretorius
>>>>>> >         <jesse.pretorius@xxxxxxxxx
>>>>>> > <mailto:jesse.pretorius@xxxxxxxxx>>
>>>>>> >         wrote:
>>>>>> >
>>>>>> >             On 23 June 2014 23:02, Dmitry Borodaenko
>>>>>> >             <dborodaenko@xxxxxxxxxxxx
>>>>>> > <mailto:dborodaenko@xxxxxxxxxxxx>>
>>>>>> >             wrote:
>>>>>> >
>>>>>> >                 Adding a configuration variable that can easily be
>>>>>> >                 changed after
>>>>>> >                 deployment is completed into Fuel is a slippery
>>>>>> > slope,
>>>>>> >                 you risk
>>>>>> >                 spamming the already overcrowded Fuel settings page
>>>>>> > with
>>>>>> >                 dozens of
>>>>>> >                 trivial (and not so trivial) options from every
>>>>>> >                 OpenStack component.
>>>>>> >
>>>>>> >
>>>>>> >             While I agree, the fact of the matter is that Fuel
>>>>>> >             facilitates the deployment of configuration to all nodes
>>>>>> > in
>>>>>> >             the environment in a consistent manner. This is
>>>>>> > essential to
>>>>>> >             deploy a reliable and supportable environment.
>>>>>> >
>>>>>> >             In order to maintain the simplicity of the UI, but also
>>>>>> >             provide access to OpenStack's customisable
>>>>>> > configuration,
>>>>>> >             why not do something like add an 'advanced' button into
>>>>>> > the
>>>>>> >             UI. Under there, the additional options can be provided,
>>>>>> > and
>>>>>> >             things that are already 'Advanced' (like the scheduler,
>>>>>> > the
>>>>>> >             Ceph replication factor, etc) can move under the
>>>>>> > 'Advanced'
>>>>>> >             button for each section.
>>>>>> >
>>>>>> >             An additional very beneficial feature in the Advanced
>>>>>> >             section would be to allow any arbitrary key-value pairs
>>>>>> > for
>>>>>> >             any settings that a user may wish to deploy into the
>>>>>> >             configuration files for the appropriate system (eg:
>>>>>> >             nova.conf). The user can be warned that these will not
>>>>>> > be
>>>>>> >             validated and that they are used at the user's own risk.
>>>>>> >             This can allow any number of OpenStack underlying
>>>>>> > features
>>>>>> >             to be used without requiring a specific UI for it.
>>>>>> >
>>>>>> >             --
>>>>>> >             Mailing list: https://launchpad.net/~fuel-dev
>>>>>> >             Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>>> >             <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>>> >             Unsubscribe : https://launchpad.net/~fuel-dev
>>>>>> >             More help   : https://help.launchpad.net/ListHelp
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >         --
>>>>>> >         Mailing list: https://launchpad.net/~fuel-dev
>>>>>> >         Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>>> >         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>>> >         Unsubscribe : https://launchpad.net/~fuel-dev
>>>>>> >         More help   : https://help.launchpad.net/ListHelp
>>>>>> >
>>>>>> >
>>>>>> >     --
>>>>>> >     Mailing list: https://launchpad.net/~fuel-dev
>>>>>> >     Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>>> >     <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>>> >     Unsubscribe : https://launchpad.net/~fuel-dev
>>>>>> >     More help   : https://help.launchpad.net/ListHelp
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Bogdan Dobrelya,
>>>>>> Skype #bogdando_at_yahoo.com
>>>>>> Irc #bogdando
>>>>>
>>>>>
>>>>> -- Mailing list: https://launchpad.net/~fuel-dev Post to :
>>>>> fuel-dev@xxxxxxxxxxxxxxxxxxx Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>
>>>>> --
>>>>> Mailing list: https://launchpad.net/~fuel-dev
>>>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>
>>>
>>
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>