fuel-dev team mailing list archive

[Meg McRoberts <mmcroberts@xxxxxxxxxxxx>]

How does one know that ones overcommit ratio is too high?  Are there
monitoring tools
that would show CPU utilization for all Compute nodes in the environment?
And, if one
set the RAM overcommit rate to something other than 1:1, would  it be easy
to tell whether
one was CPU or memory-bound?

I suppose if users were complaining about performance, this would be a good
place to look,
but this is certainly not the only factor that could affect VM
performance...


On Tue, Jun 24, 2014 at 12:49 PM, Dmitriy Novakovskiy <
dnovakovskiy@xxxxxxxxxxxx> wrote:

> David,
>
> *I created an etherpad with the additional requirements
>> (https://etherpad.openstack.org/p/cpu-overcommit-setting
>> <https://etherpad.openstack.org/p/cpu-overcommit-setting>) but don’t have
>> rights to set it as the URL for the specification.  If you or an admin
>> could do that, it’d be appreciated.*
>
>
> Done - added your etherpad as a full spec to BP:
> https://blueprints.launchpad.net/fuel/+spec/cpu-overcommit-setting
>
> *With these expectations, you would need to have the parameter be on the
>> Settings tab not (just) the Deployment Wizard. *
>
>
> Got it, makes sense. It should be on both screens.
>
> * Also, currently the only way to reactivate the Settings Tab today and
>> “re-apply” is to reset the environment – I.e. delete the existing
>> environment and re-deploy with the modified settings/parameters.   That may
>> be a bit heavy handed for just changing the overcommit value, so we may
>> need a second blueprint that would enable settings to be changed from Fuel
>> without deleting and redeploying the environment… *
>>
>
> Got it. Well, IMO it calls for us to think through some key concepts, what
> Fuel should/should not be able to do and enforce... Ability to adjust CPU
> overcommit rate on post-deployment phases will be useful, but not pri-1. I
> just thought that UI+engine already supports it.
>
> ---
> Regards,
> Dmitriy
>
>
> On Tue, Jun 24, 2014 at 10:36 PM, David Easter <deaster@xxxxxxxxxxxx>
> wrote:
>
>> Thanks.  With these expectations, you would need to have the parameter be
>> on the Settings tab not (just) the Deployment Wizard.  Having it only in
>> the deployment wizard would mean you couldn’t modify it later and that it
>> wouldn’t be reflected in the UI.   Also, currently the only way to
>> reactivate the Settings Tab today and “re-apply” is to reset the
>> environment – I.e. delete the existing environment and re-deploy with the
>> modified settings/parameters.   That may be a bit heavy handed for just
>> changing the overcommit value, so we may need a second blueprint that would
>> enable settings to be changed from Fuel without deleting and redeploying
>> the environment…
>>
>> I created an etherpad with the additional requirements (
>> https://etherpad.openstack.org/p/cpu-overcommit-setting) but don’t have
>> rights to set it as the URL for the specification.  If you or an admin
>> could do that, it’d be appreciated.
>>
>> Thanks,
>>
>> - David J. Easter
>>   Director of Product Management, Mirantis
>>
>> From: Dmitriy Novakovskiy <dnovakovskiy@xxxxxxxxxxxx>
>> Date: Tuesday, June 24, 2014 at 11:54 AM
>> To: David Easter <deaster@xxxxxxxxxxxx>
>> Cc: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>, fuel-dev <
>> fuel-dev@xxxxxxxxxxxxxxxxxxx>, Sergii Golovatiuk <
>> sgolovatiuk@xxxxxxxxxxxx>
>>
>> Subject: Re: [Fuel-dev] overcommit rates
>>
>> David,
>>
>> 1. Yes
>> 2. Yes, same as other Settings which Fuel can "re-apply" today (syslog
>> server, scheduler type, etc). As Sergey pointed out, there's no danger in
>> it, and there should not be any additional technical effort to implement.
>> The only thing is - we need to add a note on UI and/or in documentation to
>> explain to user that changing overcommit rate to lower will not
>> automatically offload Compute nodes that are already over-provisioned,
>> he/she will still need to migrate excessive VMs away:)
>> 3. No, no monitoring. Fuel master should be the only source of "truth"
>> for those settings which are exposed to user on Web UI.
>>
>>
>> ---
>> Regards,
>> Dmitriy
>>
>>
>> On Tue, Jun 24, 2014 at 8:35 PM, Sergii Golovatiuk <
>> sgolovatiuk@xxxxxxxxxxxx> wrote:
>>
>>> Hi,
>>>
>>> IMHO,
>>>
>>> 1. Yes
>>> 2. In theory, it's possible to change/update values after deployment
>>> time. It's not a dangerous operation. If node is full, it won't be selected
>>> for new VM starts.
>>> 3. If it's changed manually, it will be reset on next puppet run.
>>>
>>> ~Sergii
>>>
>>>
>>> On Tue, Jun 24, 2014 at 5:21 PM, David Easter <deaster@xxxxxxxxxxxx>
>>> wrote:
>>>
>>>> Dmitriy,
>>>>
>>>>   Once set through the UI, I have a few questions on expectations
>>>> post-deployment:
>>>>
>>>>    1. Would the value selected need to be shown within the Fuel UI
>>>>    post-deployment to let the administrator know what value was selected per
>>>>    environment?
>>>>    2. Is there an expectation that the value can be changed in a
>>>>    running environment, or would this only be expected at deployment time?
>>>>    3. If the value were changed in the environment manually, would be
>>>>    expected that the Fuel Master Node would need to monitor for such changes?
>>>>
>>>> Thanks,
>>>>
>>>> - David J. Easter
>>>>   Director of Product Management, Mirantis
>>>>
>>>> From: Dmitriy Novakovskiy <dnovakovskiy@xxxxxxxxxxxx>
>>>> Date: Tuesday, June 24, 2014 at 4:37 AM
>>>> To: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>
>>>> Cc: fuel-dev <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>> Subject: Re: [Fuel-dev] overcommit rates
>>>>
>>>> All,
>>>>
>>>> I've submitted a blueprint on this:
>>>> https://blueprints.launchpad.net/fuel/+spec/cpu-overcommit-setting
>>>>
>>>> Bogdan,
>>>>
>>>> Nice and creative approach, but IMO the question of "how many instances
>>>> to run" and "what overcommit to use" belongs more to planning/architecture
>>>> workshop/BOM calculator
>>>> <http://www.mirantis.com/openstack-services/bom-calculator/> phase.
>>>> Not only amount of VMs to run on given amount of Compute nodes counts here
>>>> - also the capability of hypervisor to "juggle" oversubscribed cores (poor
>>>> in KVM, better in ESXi), the kind of environment (Prod, Dev/Test), the
>>>> level of tolerance to VM failure, etc. Fuel just need to prompt+allow user
>>>> to apply decisions made at planning phase to deployment phase.
>>>>
>>>>
>>>>
>>>> ---
>>>> Regards,
>>>> Dmitriy
>>>>
>>>>
>>>> On Tue, Jun 24, 2014 at 2:22 PM, Bogdan Dobrelya <
>>>> bdobrelia@xxxxxxxxxxxx> wrote:
>>>>
>>>>> On 06/24/2014 01:50 PM, Dmitriy Novakovskiy wrote:
>>>>> > Andrey, Dmitriy
>>>>> >
>>>>> > I've just realized that CPU overcommitment setting is defined not for
>>>>> > Controllers but for Compute nodes, and may differ on per-node basis.
>>>>> So,
>>>>> > the 2nd "harm" example that I exposed a few emails ago (with failed
>>>>> and
>>>>> > redeployed controller resulting in inconsistent overcommitment
>>>>> behavior)
>>>>> > is irrelevant.
>>>>> >
>>>>> > With this in mind, 3rd example is even worse :)
>>>>> >
>>>>> > 3) User goes through architecture workshop and capacity planning
>>>>> > exercise, decides on CPU overcommit rates he/she will use for
>>>>> OpenStack
>>>>> > cloud. User deploys the cloud with, let's say, 20+ compute nodes.
>>>>> Now,
>>>>> > w/o CPU overcommit setting on GUI, User will:
>>>>> > - get a cloud that's inconsistent with decisions made during planning
>>>>> > - be not aware of this - GUI did not point user to a need of
>>>>> specifying
>>>>> > this setting
>>>>> > - be frustrated and put under risk - if the cloud is deployed with
>>>>> real
>>>>> > workloads the compute nodes will soon become over provisioned and
>>>>> > unreliable due to default Fuel setting (prod KVM clouds often use
>>>>> > super-soft overcommit setting, or no overcommit at all)
>>>>> > - finally - *user will have to MANUALLY specify and maintain
>>>>> overcommit
>>>>> > setting across 20+ compute nodes.*
>>>>> > *
>>>>> > *
>>>>> > Doesn't sound like a good UX from cloud lifecycle management app :) -
>>>>> > Fuel will quickly be ditched in favor of config management tool.
>>>>> >
>>>>> > So, my proposal on CPU overcommitment setting is following:
>>>>> >
>>>>> > - Expose it in Wizard on hypervisor selection screen
>>>>> > - To simplify things initially - implement the setting as a drop-down
>>>>> > list, with 2 options:
>>>>> > -- No CPU overcommitment (1:1)
>>>>> > -- Light CPU overcommitment (3:1)
>>>>> > -- Heavy CPU overcommitment (6:1) --- default setting in BOM
>>>>> calculator
>>>>>
>>>>> Good point. We could as well evaluate the suggested ratio based on the
>>>>> user outputs and roles allocation, e.g.:
>>>>> 1) Q(wizard):"How many running instances your cloud has to maintain?"
>>>>> ->
>>>>> user specifies 10000
>>>>> 2) and there are 100 compute nodes assigned in UI
>>>>>
>>>>> then,
>>>>> - the CPU overcommitment should be evaluated as 10000/100 : 1
>>>>> - if the value exceeds the default maximum (16) for OSt, the warning
>>>>> should be issued in UI notifications (please provide more compute nodes
>>>>> in order to maintain the desired capacity for running instances)
>>>>>
>>>>> >
>>>>> >
>>>>> > ---
>>>>> > Regards,
>>>>> > Dmitriy
>>>>> >
>>>>> >
>>>>> > On Tue, Jun 24, 2014 at 1:23 PM, Andrey Danin <adanin@xxxxxxxxxxxx
>>>>> > <mailto:adanin@xxxxxxxxxxxx>> wrote:
>>>>> >
>>>>> >     I think, we can add to openstack.yaml a special tag "advanced" to
>>>>> >     every advanced parameter, and UI can hide all such parameters for
>>>>> >     every group under the button "show more options". Also we can add
>>>>> >     the "expand all hidden options" button at the top of the tab. I
>>>>> >     don't think it's hard to implement.
>>>>> >
>>>>> >     Let's wait an opinion of UI team.
>>>>> >
>>>>> >     On Jun 24, 2014 1:58 PM, "Meg McRoberts" <
>>>>> mmcroberts@xxxxxxxxxxxx
>>>>> >     <mailto:mmcroberts@xxxxxxxxxxxx>> wrote:
>>>>> >
>>>>> >         Jesse and others make a very good point.  The Settings screen
>>>>> >         has already become a real mess
>>>>> >         that needs to be split up and organized a bit.  I'm told that
>>>>> >         the serious restructuring should come
>>>>> >         after we implement the plug-ins scheme (currently planned for
>>>>> >         6.0).  And each section definitely
>>>>> >         needs an "Advanced" section for configurations that aren't
>>>>> routine.
>>>>> >
>>>>> >         So do we wait to restructure the "Settings" screen until the
>>>>> >         plug-ins scheme is implemented?  I
>>>>> >         think breaking it up into sub-sections would make it more
>>>>> usable
>>>>> >         in the meantime but I don't know
>>>>> >         if it's worth the effort to do that ahead of the plug-ins.
>>>>> >
>>>>> >
>>>>> >         On Tue, Jun 24, 2014 at 2:23 AM, Jesse Pretorius
>>>>> >         <jesse.pretorius@xxxxxxxxx <mailto:jesse.pretorius@xxxxxxxxx
>>>>> >>
>>>>> >         wrote:
>>>>> >
>>>>> >             On 23 June 2014 23:02, Dmitry Borodaenko
>>>>> >             <dborodaenko@xxxxxxxxxxxx <mailto:
>>>>> dborodaenko@xxxxxxxxxxxx>>
>>>>> >             wrote:
>>>>> >
>>>>> >                 Adding a configuration variable that can easily be
>>>>> >                 changed after
>>>>> >                 deployment is completed into Fuel is a slippery
>>>>> slope,
>>>>> >                 you risk
>>>>> >                 spamming the already overcrowded Fuel settings page
>>>>> with
>>>>> >                 dozens of
>>>>> >                 trivial (and not so trivial) options from every
>>>>> >                 OpenStack component.
>>>>> >
>>>>> >
>>>>> >             While I agree, the fact of the matter is that Fuel
>>>>> >             facilitates the deployment of configuration to all nodes
>>>>> in
>>>>> >             the environment in a consistent manner. This is
>>>>> essential to
>>>>> >             deploy a reliable and supportable environment.
>>>>> >
>>>>> >             In order to maintain the simplicity of the UI, but also
>>>>> >             provide access to OpenStack's customisable configuration,
>>>>> >             why not do something like add an 'advanced' button into
>>>>> the
>>>>> >             UI. Under there, the additional options can be provided,
>>>>> and
>>>>> >             things that are already 'Advanced' (like the scheduler,
>>>>> the
>>>>> >             Ceph replication factor, etc) can move under the
>>>>> 'Advanced'
>>>>> >             button for each section.
>>>>> >
>>>>> >             An additional very beneficial feature in the Advanced
>>>>> >             section would be to allow any arbitrary key-value pairs
>>>>> for
>>>>> >             any settings that a user may wish to deploy into the
>>>>> >             configuration files for the appropriate system (eg:
>>>>> >             nova.conf). The user can be warned that these will not be
>>>>> >             validated and that they are used at the user's own risk.
>>>>> >             This can allow any number of OpenStack underlying
>>>>> features
>>>>> >             to be used without requiring a specific UI for it.
>>>>> >
>>>>> >             --
>>>>> >             Mailing list: https://launchpad.net/~fuel-dev
>>>>> >             Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >             <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >             Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >             More help   : https://help.launchpad.net/ListHelp
>>>>> >
>>>>> >
>>>>> >
>>>>> >         --
>>>>> >         Mailing list: https://launchpad.net/~fuel-dev
>>>>> >         Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >         Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >         More help   : https://help.launchpad.net/ListHelp
>>>>> >
>>>>> >
>>>>> >     --
>>>>> >     Mailing list: https://launchpad.net/~fuel-dev
>>>>> >     Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >     Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >     More help   : https://help.launchpad.net/ListHelp
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Bogdan Dobrelya,
>>>>> Skype #bogdando_at_yahoo.com
>>>>> Irc #bogdando
>>>>>
>>>>
>>>> -- Mailing list: https://launchpad.net/~fuel-dev Post to :
>>>> fuel-dev@xxxxxxxxxxxxxxxxxxx Unsubscribe :
>>>> https://launchpad.net/~fuel-dev More help :
>>>> https://help.launchpad.net/ListHelp
>>>>
>>>> --
>>>> Mailing list: https://launchpad.net/~fuel-dev
>>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>