fuel-dev team mailing list archive

[Dmitriy Novakovskiy <dnovakovskiy@xxxxxxxxxxxx>]

Hi Meg,

Let's take it to a separate thread. I will explain the overcommit aspects
to you

---
Regards,
Dmitriy


On Tue, Jun 24, 2014 at 11:00 PM, Meg McRoberts <mmcroberts@xxxxxxxxxxxx>
wrote:

> How does one know that ones overcommit ratio is too high?  Are there
> monitoring tools
> that would show CPU utilization for all Compute nodes in the environment?
> And, if one
> set the RAM overcommit rate to something other than 1:1, would  it be easy
> to tell whether
> one was CPU or memory-bound?
>
> I suppose if users were complaining about performance, this would be a
> good place to look,
> but this is certainly not the only factor that could affect VM
> performance...
>
>
> On Tue, Jun 24, 2014 at 12:49 PM, Dmitriy Novakovskiy <
> dnovakovskiy@xxxxxxxxxxxx> wrote:
>
>> David,
>>
>> *I created an etherpad with the additional requirements
>>> (https://etherpad.openstack.org/p/cpu-overcommit-setting
>>> <https://etherpad.openstack.org/p/cpu-overcommit-setting>) but don’t have
>>> rights to set it as the URL for the specification.  If you or an admin
>>> could do that, it’d be appreciated.*
>>
>>
>> Done - added your etherpad as a full spec to BP:
>> https://blueprints.launchpad.net/fuel/+spec/cpu-overcommit-setting
>>
>> *With these expectations, you would need to have the parameter be on the
>>> Settings tab not (just) the Deployment Wizard. *
>>
>>
>> Got it, makes sense. It should be on both screens.
>>
>> * Also, currently the only way to reactivate the Settings Tab today and
>>> “re-apply” is to reset the environment – I.e. delete the existing
>>> environment and re-deploy with the modified settings/parameters.   That may
>>> be a bit heavy handed for just changing the overcommit value, so we may
>>> need a second blueprint that would enable settings to be changed from Fuel
>>> without deleting and redeploying the environment… *
>>>
>>
>> Got it. Well, IMO it calls for us to think through some key concepts,
>> what Fuel should/should not be able to do and enforce... Ability to adjust
>> CPU overcommit rate on post-deployment phases will be useful, but not
>> pri-1. I just thought that UI+engine already supports it.
>>
>> ---
>> Regards,
>> Dmitriy
>>
>>
>> On Tue, Jun 24, 2014 at 10:36 PM, David Easter <deaster@xxxxxxxxxxxx>
>> wrote:
>>
>>> Thanks.  With these expectations, you would need to have the parameter
>>> be on the Settings tab not (just) the Deployment Wizard.  Having it only in
>>> the deployment wizard would mean you couldn’t modify it later and that it
>>> wouldn’t be reflected in the UI.   Also, currently the only way to
>>> reactivate the Settings Tab today and “re-apply” is to reset the
>>> environment – I.e. delete the existing environment and re-deploy with the
>>> modified settings/parameters.   That may be a bit heavy handed for just
>>> changing the overcommit value, so we may need a second blueprint that would
>>> enable settings to be changed from Fuel without deleting and redeploying
>>> the environment…
>>>
>>> I created an etherpad with the additional requirements (
>>> https://etherpad.openstack.org/p/cpu-overcommit-setting) but don’t have
>>> rights to set it as the URL for the specification.  If you or an admin
>>> could do that, it’d be appreciated.
>>>
>>> Thanks,
>>>
>>> - David J. Easter
>>>   Director of Product Management, Mirantis
>>>
>>> From: Dmitriy Novakovskiy <dnovakovskiy@xxxxxxxxxxxx>
>>> Date: Tuesday, June 24, 2014 at 11:54 AM
>>> To: David Easter <deaster@xxxxxxxxxxxx>
>>> Cc: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>, fuel-dev <
>>> fuel-dev@xxxxxxxxxxxxxxxxxxx>, Sergii Golovatiuk <
>>> sgolovatiuk@xxxxxxxxxxxx>
>>>
>>> Subject: Re: [Fuel-dev] overcommit rates
>>>
>>> David,
>>>
>>> 1. Yes
>>> 2. Yes, same as other Settings which Fuel can "re-apply" today (syslog
>>> server, scheduler type, etc). As Sergey pointed out, there's no danger in
>>> it, and there should not be any additional technical effort to implement.
>>> The only thing is - we need to add a note on UI and/or in documentation to
>>> explain to user that changing overcommit rate to lower will not
>>> automatically offload Compute nodes that are already over-provisioned,
>>> he/she will still need to migrate excessive VMs away:)
>>> 3. No, no monitoring. Fuel master should be the only source of "truth"
>>> for those settings which are exposed to user on Web UI.
>>>
>>>
>>> ---
>>> Regards,
>>> Dmitriy
>>>
>>>
>>> On Tue, Jun 24, 2014 at 8:35 PM, Sergii Golovatiuk <
>>> sgolovatiuk@xxxxxxxxxxxx> wrote:
>>>
>>>> Hi,
>>>>
>>>> IMHO,
>>>>
>>>> 1. Yes
>>>> 2. In theory, it's possible to change/update values after deployment
>>>> time. It's not a dangerous operation. If node is full, it won't be selected
>>>> for new VM starts.
>>>> 3. If it's changed manually, it will be reset on next puppet run.
>>>>
>>>> ~Sergii
>>>>
>>>>
>>>> On Tue, Jun 24, 2014 at 5:21 PM, David Easter <deaster@xxxxxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Dmitriy,
>>>>>
>>>>>   Once set through the UI, I have a few questions on expectations
>>>>> post-deployment:
>>>>>
>>>>>    1. Would the value selected need to be shown within the Fuel UI
>>>>>    post-deployment to let the administrator know what value was selected per
>>>>>    environment?
>>>>>    2. Is there an expectation that the value can be changed in a
>>>>>    running environment, or would this only be expected at deployment time?
>>>>>    3. If the value were changed in the environment manually, would be
>>>>>    expected that the Fuel Master Node would need to monitor for such changes?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> - David J. Easter
>>>>>   Director of Product Management, Mirantis
>>>>>
>>>>> From: Dmitriy Novakovskiy <dnovakovskiy@xxxxxxxxxxxx>
>>>>> Date: Tuesday, June 24, 2014 at 4:37 AM
>>>>> To: Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>
>>>>> Cc: fuel-dev <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> Subject: Re: [Fuel-dev] overcommit rates
>>>>>
>>>>> All,
>>>>>
>>>>> I've submitted a blueprint on this:
>>>>> https://blueprints.launchpad.net/fuel/+spec/cpu-overcommit-setting
>>>>>
>>>>> Bogdan,
>>>>>
>>>>> Nice and creative approach, but IMO the question of "how many
>>>>> instances to run" and "what overcommit to use" belongs more to
>>>>> planning/architecture workshop/BOM calculator
>>>>> <http://www.mirantis.com/openstack-services/bom-calculator/> phase.
>>>>> Not only amount of VMs to run on given amount of Compute nodes counts here
>>>>> - also the capability of hypervisor to "juggle" oversubscribed cores (poor
>>>>> in KVM, better in ESXi), the kind of environment (Prod, Dev/Test), the
>>>>> level of tolerance to VM failure, etc. Fuel just need to prompt+allow user
>>>>> to apply decisions made at planning phase to deployment phase.
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Regards,
>>>>> Dmitriy
>>>>>
>>>>>
>>>>> On Tue, Jun 24, 2014 at 2:22 PM, Bogdan Dobrelya <
>>>>> bdobrelia@xxxxxxxxxxxx> wrote:
>>>>>
>>>>>> On 06/24/2014 01:50 PM, Dmitriy Novakovskiy wrote:
>>>>>> > Andrey, Dmitriy
>>>>>> >
>>>>>> > I've just realized that CPU overcommitment setting is defined not
>>>>>> for
>>>>>> > Controllers but for Compute nodes, and may differ on per-node
>>>>>> basis. So,
>>>>>> > the 2nd "harm" example that I exposed a few emails ago (with failed
>>>>>> and
>>>>>> > redeployed controller resulting in inconsistent overcommitment
>>>>>> behavior)
>>>>>> > is irrelevant.
>>>>>> >
>>>>>> > With this in mind, 3rd example is even worse :)
>>>>>> >
>>>>>> > 3) User goes through architecture workshop and capacity planning
>>>>>> > exercise, decides on CPU overcommit rates he/she will use for
>>>>>> OpenStack
>>>>>> > cloud. User deploys the cloud with, let's say, 20+ compute nodes.
>>>>>> Now,
>>>>>> > w/o CPU overcommit setting on GUI, User will:
>>>>>> > - get a cloud that's inconsistent with decisions made during
>>>>>> planning
>>>>>> > - be not aware of this - GUI did not point user to a need of
>>>>>> specifying
>>>>>> > this setting
>>>>>> > - be frustrated and put under risk - if the cloud is deployed with
>>>>>> real
>>>>>> > workloads the compute nodes will soon become over provisioned and
>>>>>> > unreliable due to default Fuel setting (prod KVM clouds often use
>>>>>> > super-soft overcommit setting, or no overcommit at all)
>>>>>> > - finally - *user will have to MANUALLY specify and maintain
>>>>>> overcommit
>>>>>> > setting across 20+ compute nodes.*
>>>>>> > *
>>>>>> > *
>>>>>> > Doesn't sound like a good UX from cloud lifecycle management app :)
>>>>>> -
>>>>>> > Fuel will quickly be ditched in favor of config management tool.
>>>>>> >
>>>>>> > So, my proposal on CPU overcommitment setting is following:
>>>>>> >
>>>>>> > - Expose it in Wizard on hypervisor selection screen
>>>>>> > - To simplify things initially - implement the setting as a
>>>>>> drop-down
>>>>>> > list, with 2 options:
>>>>>> > -- No CPU overcommitment (1:1)
>>>>>> > -- Light CPU overcommitment (3:1)
>>>>>> > -- Heavy CPU overcommitment (6:1) --- default setting in BOM
>>>>>> calculator
>>>>>>
>>>>>> Good point. We could as well evaluate the suggested ratio based on the
>>>>>> user outputs and roles allocation, e.g.:
>>>>>> 1) Q(wizard):"How many running instances your cloud has to maintain?"
>>>>>> ->
>>>>>> user specifies 10000
>>>>>> 2) and there are 100 compute nodes assigned in UI
>>>>>>
>>>>>> then,
>>>>>> - the CPU overcommitment should be evaluated as 10000/100 : 1
>>>>>> - if the value exceeds the default maximum (16) for OSt, the warning
>>>>>> should be issued in UI notifications (please provide more compute
>>>>>> nodes
>>>>>> in order to maintain the desired capacity for running instances)
>>>>>>
>>>>>> >
>>>>>> >
>>>>>> > ---
>>>>>> > Regards,
>>>>>> > Dmitriy
>>>>>> >
>>>>>> >
>>>>>> > On Tue, Jun 24, 2014 at 1:23 PM, Andrey Danin <adanin@xxxxxxxxxxxx
>>>>>> > <mailto:adanin@xxxxxxxxxxxx>> wrote:
>>>>>> >
>>>>>> >     I think, we can add to openstack.yaml a special tag "advanced"
>>>>>> to
>>>>>> >     every advanced parameter, and UI can hide all such parameters
>>>>>> for
>>>>>> >     every group under the button "show more options". Also we can
>>>>>> add
>>>>>> >     the "expand all hidden options" button at the top of the tab. I
>>>>>> >     don't think it's hard to implement.
>>>>>> >
>>>>>> >     Let's wait an opinion of UI team.
>>>>>> >
>>>>>> >     On Jun 24, 2014 1:58 PM, "Meg McRoberts" <
>>>>>> mmcroberts@xxxxxxxxxxxx
>>>>>> >     <mailto:mmcroberts@xxxxxxxxxxxx>> wrote:
>>>>>> >
>>>>>> >         Jesse and others make a very good point.  The Settings
>>>>>> screen
>>>>>> >         has already become a real mess
>>>>>> >         that needs to be split up and organized a bit.  I'm told
>>>>>> that
>>>>>> >         the serious restructuring should come
>>>>>> >         after we implement the plug-ins scheme (currently planned
>>>>>> for
>>>>>> >         6.0).  And each section definitely
>>>>>> >         needs an "Advanced" section for configurations that aren't
>>>>>> routine.
>>>>>> >
>>>>>> >         So do we wait to restructure the "Settings" screen until the
>>>>>> >         plug-ins scheme is implemented?  I
>>>>>> >         think breaking it up into sub-sections would make it more
>>>>>> usable
>>>>>> >         in the meantime but I don't know
>>>>>> >         if it's worth the effort to do that ahead of the plug-ins.
>>>>>> >
>>>>>> >
>>>>>> >         On Tue, Jun 24, 2014 at 2:23 AM, Jesse Pretorius
>>>>>> >         <jesse.pretorius@xxxxxxxxx <mailto:
>>>>>> jesse.pretorius@xxxxxxxxx>>
>>>>>> >         wrote:
>>>>>> >
>>>>>> >             On 23 June 2014 23:02, Dmitry Borodaenko
>>>>>> >             <dborodaenko@xxxxxxxxxxxx <mailto:
>>>>>> dborodaenko@xxxxxxxxxxxx>>
>>>>>> >             wrote:
>>>>>> >
>>>>>> >                 Adding a configuration variable that can easily be
>>>>>> >                 changed after
>>>>>> >                 deployment is completed into Fuel is a slippery
>>>>>> slope,
>>>>>> >                 you risk
>>>>>> >                 spamming the already overcrowded Fuel settings page
>>>>>> with
>>>>>> >                 dozens of
>>>>>> >                 trivial (and not so trivial) options from every
>>>>>> >                 OpenStack component.
>>>>>> >
>>>>>> >
>>>>>> >             While I agree, the fact of the matter is that Fuel
>>>>>> >             facilitates the deployment of configuration to all
>>>>>> nodes in
>>>>>> >             the environment in a consistent manner. This is
>>>>>> essential to
>>>>>> >             deploy a reliable and supportable environment.
>>>>>> >
>>>>>> >             In order to maintain the simplicity of the UI, but also
>>>>>> >             provide access to OpenStack's customisable
>>>>>> configuration,
>>>>>> >             why not do something like add an 'advanced' button into
>>>>>> the
>>>>>> >             UI. Under there, the additional options can be
>>>>>> provided, and
>>>>>> >             things that are already 'Advanced' (like the scheduler,
>>>>>> the
>>>>>> >             Ceph replication factor, etc) can move under the
>>>>>> 'Advanced'
>>>>>> >             button for each section.
>>>>>> >
>>>>>> >             An additional very beneficial feature in the Advanced
>>>>>> >             section would be to allow any arbitrary key-value pairs
>>>>>> for
>>>>>> >             any settings that a user may wish to deploy into the
>>>>>> >             configuration files for the appropriate system (eg:
>>>>>> >             nova.conf). The user can be warned that these will not
>>>>>> be
>>>>>> >             validated and that they are used at the user's own risk.
>>>>>> >             This can allow any number of OpenStack underlying
>>>>>> features
>>>>>> >             to be used without requiring a specific UI for it.
>>>>>> >
>>>>>> >             --
>>>>>> >             Mailing list: https://launchpad.net/~fuel-dev
>>>>>> >             Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>>> >             <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>>> >             Unsubscribe : https://launchpad.net/~fuel-dev
>>>>>> >             More help   : https://help.launchpad.net/ListHelp
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >         --
>>>>>> >         Mailing list: https://launchpad.net/~fuel-dev
>>>>>> >         Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>>> >         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>>> >         Unsubscribe : https://launchpad.net/~fuel-dev
>>>>>> >         More help   : https://help.launchpad.net/ListHelp
>>>>>> >
>>>>>> >
>>>>>> >     --
>>>>>> >     Mailing list: https://launchpad.net/~fuel-dev
>>>>>> >     Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>>> >     <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>>> >     Unsubscribe : https://launchpad.net/~fuel-dev
>>>>>> >     More help   : https://help.launchpad.net/ListHelp
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Bogdan Dobrelya,
>>>>>> Skype #bogdando_at_yahoo.com
>>>>>> Irc #bogdando
>>>>>>
>>>>>
>>>>> -- Mailing list: https://launchpad.net/~fuel-dev Post to :
>>>>> fuel-dev@xxxxxxxxxxxxxxxxxxx Unsubscribe :
>>>>> https://launchpad.net/~fuel-dev More help :
>>>>> https://help.launchpad.net/ListHelp
>>>>>
>>>>> --
>>>>> Mailing list: https://launchpad.net/~fuel-dev
>>>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>