yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #18057
[Bug 1338822] Re: CPU limit not used in instance resize
I'm going to mark this as not a bug, since you were able to get the
desired behavior by configuration.
As for why CoreFilter isn't in the default filters, these types of items
are part of an ongoing effort in the community to make OpenStack better
for operators. The program facilitates communication between operators
and developers so any issues with running OpenStack can be addressed.
Please see the following links on how you can join this effort, get
answers, and bring about changes that are important to you.
This blog post [1] should give you the information you need to get
started, by signing up for the openstack-operators mailing list and
checking out prior discussions captured in the etherpad (one of them is
"Reasonable Defaults").
I hope this helps.
[1] http://www.openstack.org/blog/2014/04/openstack-summit-now-with-
more-ops/
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1338822
Title:
CPU limit not used in instance resize
Status in OpenStack Compute (Nova):
Invalid
Bug description:
If I resize an instance to a flavor with more cpus than should be
possible, even more cpus than cpu_allocation_ratio would allow, then
nova proceeds with the resize and the instance state goes to error and
it does exist anymore in the hypervisor.
My environment:
Nova 2014.1 (Icehouse)
Libvirt / KVM hypervisor
Ceph RBD for volume storage
1 nova-manage node, 2 nova-compute nodes (all virtualized on my laptop)
Ubuntu 14.04 for all OSes except for the instance
Cirros 0.3.2 for the instance
/etc/nova/nova.conf on all nodes contain:
allow_resize_to_same_host=true (although this didn't make a difference!)
ram_allocation_ratio=0.95
cpu_allocation_ratio=5
CPU and RAM allocation_ratio limits are set but only the memory limit took effect. This is from the log below:
memory limit: 1900.95 MB, free: 1388.95 MB
CPUs limit not specified, defaulting to unlimited
This is the log from nova-compute.log when I resize an instance to a
flavor with 500 vcpus:
AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
AUDIT nova.compute.resource_tracker [-] Free ram (MB): 1489
AUDIT nova.compute.resource_tracker [-] Free disk (GB): 34
AUDIT nova.compute.resource_tracker [-] Free VCPUS: 2
INFO nova.compute.resource_tracker [-] Compute_service record updated for r-ABCDEF1234:r-ABCDEF1234.vagrant.local
AUDIT nova.compute.claims [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Attempting claim: memory 500 MB, disk 1 GB, VCPUs 500
AUDIT nova.compute.claims [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Total memory: 2001 MB, used: 512.00 MB
AUDIT nova.compute.claims [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] memory limit: 1900.95 MB, free: 1388.95 MB
AUDIT nova.compute.claims [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Total disk: 34 GB, used: 0.00 GB
AUDIT nova.compute.claims [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] disk limit not specified, defaulting to unlimited
AUDIT nova.compute.claims [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Total CPUs: 2 VCPUs, used: 0.00 VCPUs
AUDIT nova.compute.claims [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] CPUs limit not specified, defaulting to unlimited
AUDIT nova.compute.claims [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Claim successful
AUDIT nova.compute.resource_tracker [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] Updating from migration 75204b82-70e9-4e13-b56f-d7b78613cac7
AUDIT nova.compute.manager [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Migrating
WARNING nova.virt.libvirt.utils [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] systool is not installed
WARNING nova.virt.libvirt.utils [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] systool is not installed
INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.55.253
INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.168.55.11:5672
INFO nova.virt.libvirt.driver [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Creating image
ERROR glanceclient.common.http [-] Request returned failure status.
WARNING nova.compute.utils [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Can't access image : Image could not be found.
INFO nova.virt.libvirt.firewall [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Called setup_basic_filtering in nwfilter
INFO nova.virt.libvirt.firewall [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Ensuring static filters
ERROR nova.virt.libvirt.driver [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] An error occurred while trying to launch a defined domain with xml: <domain type='qemu'>
<name>instance-00000006</name>
<uuid>75204b82-70e9-4e13-b56f-d7b78613cac7</uuid>
<memory unit='KiB'>512000</memory>
<currentMemory unit='KiB'>512000</currentMemory>
<vcpu placement='static'>500</vcpu>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>2014.1</entry>
<entry name='serial'>cffca416-65a2-48b5-ba0c-40ad122dc9d2</entry>
<entry name='uuid'>75204b82-70e9-4e13-b56f-d7b78613cac7</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='volumes'>
<secret type='ceph' uuid='d9002041-37e4-b53c-1d9e-a5e5ee7e3466'/>
</auth>
<source protocol='rbd' name='volumes/volume-22a7ae26-e55b-4aad-a3a1-bbee6ecd6abb'>
<host name='192.168.55.11' port='6789'/>
</source>
<target dev='vda' bus='virtio'/>
<serial>22a7ae26-e55b-4aad-a3a1-bbee6ecd6abb</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<interface type='bridge'>
<mac address='fa:16:3e:e3:8f:5d'/>
<source bridge='br101'/>
<model type='virtio'/>
<driver name='qemu'/>
<filterref filter='nova-instance-instance-00000006-fa163ee38f5d'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='file'>
<source path='/var/lib/nova/instances/75204b82-70e9-4e13-b56f-d7b78613cac7/console.log'/>
<target port='0'/>
</serial>
<serial type='pty'>
<target port='1'/>
</serial>
<console type='file'>
<source path='/var/lib/nova/instances/75204b82-70e9-4e13-b56f-d7b78613cac7/console.log'/>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='192.168.55.101' keymap='en-us'>
<listen type='address' address='192.168.55.101'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</memballoon>
</devices>
</domain>
ERROR nova.compute.manager [req-960d005b-0ca5-434b-96dc-e1e7575aef0f 49e16f440dcc4dffa485132bf4c1fceb a5a08d831062498995090b1fb6e7fdf2] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Setting instance vm_state to ERROR
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Traceback (most recent call last):
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3559, in finish_resize
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] disk_info, image)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3530, in _finish_resize
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] block_device_info, power_on)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5004, in finish_migration
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] block_device_info, power_on)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3644, in _create_domain_and_network
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] power_on=power_on)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3547, in _create_domain
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] domain.XMLDesc(0))
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] six.reraise(self.type_, self.value, self.tb)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3542, in _create_domain
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] domain.createWithFlags(launch_flags)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] result = proxy_call(self._autowrap, f, *args, **kwargs)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] rv = execute(f,*args,**kwargs)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] rv = meth(*args,**kwargs)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 896, in createWithFlags
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] libvirtError: unsupported configuration: Maximum CPUs greater than specified machine type limit
TRACE nova.compute.manager [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7]
ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: unsupported configuration: Maximum CPUs greater than specified machine type limit
TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
TRACE oslo.messaging.rpc.dispatcher incoming.message))
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
TRACE oslo.messaging.rpc.dispatcher payload)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 280, in decorated_function
TRACE oslo.messaging.rpc.dispatcher pass
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 266, in decorated_function
TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 333, in decorated_function
TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 254, in decorated_function
TRACE oslo.messaging.rpc.dispatcher break
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 235, in decorated_function
TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 309, in decorated_function
TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 296, in decorated_function
TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3571, in finish_resize
TRACE oslo.messaging.rpc.dispatcher self._set_instance_error_state(context, instance['uuid'])
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3559, in finish_resize
TRACE oslo.messaging.rpc.dispatcher disk_info, image)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3530, in _finish_resize
TRACE oslo.messaging.rpc.dispatcher block_device_info, power_on)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5004, in finish_migration
TRACE oslo.messaging.rpc.dispatcher block_device_info, power_on)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3644, in _create_domain_and_network
TRACE oslo.messaging.rpc.dispatcher power_on=power_on)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3547, in _create_domain
TRACE oslo.messaging.rpc.dispatcher domain.XMLDesc(0))
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3542, in _create_domain
TRACE oslo.messaging.rpc.dispatcher domain.createWithFlags(launch_flags)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
TRACE oslo.messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
TRACE oslo.messaging.rpc.dispatcher rv = execute(f,*args,**kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
TRACE oslo.messaging.rpc.dispatcher rv = meth(*args,**kwargs)
TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/libvirt.py", line 896, in createWithFlags
TRACE oslo.messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
TRACE oslo.messaging.rpc.dispatcher libvirtError: unsupported configuration: Maximum CPUs greater than specified machine type limit
AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
WARNING nova.compute.resource_tracker [-] [instance: 75204b82-70e9-4e13-b56f-d7b78613cac7] Instance not resizing, skipping migration.
AUDIT nova.compute.resource_tracker [-] Free ram (MB): 989
AUDIT nova.compute.resource_tracker [-] Free disk (GB): 33
AUDIT nova.compute.resource_tracker [-] Free VCPUS: -498
WARNING nova.compute.manager [-] Found 1 in the database and 0 on the hypervisor.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1338822/+subscriptions
References