← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #12010

[Bug 1295876] [NEW] libvirtError: internal error unable to add domain xxx to cgroup: No space left on device

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

logstash query:   message:"cgroup\: No space left on device" AND
filename:logs*screen-n-cpu.txt

http://logs.openstack.org/12/80412/8/check/check-tempest-dsvm-postgres-
full/f9f6158/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-03-21_17_45_12_490


ERROR nova.compute.manager [req-630b71d6-0fbe-4e9e-99fe-019da7d29a3a FixedIPsNegativeTestJson-475659359 FixedIPsNegativeTestJson-265680949] [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Error: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Traceback (most recent call last):
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1304, in _build_instance
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     set_access_ip=set_access_ip)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 394, in decorated_function
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     return function(self, context, *args, **kwargs)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1716, in _spawn
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     LOG.exception(_('Instance failed to spawn'), instance=instance)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     six.reraise(self.type_, self.value, self.tb)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1713, in _spawn
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     block_device_info)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2241, in spawn
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     block_device_info)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3621, in _create_domain_and_network
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     power_on=power_on)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3531, in _create_domain
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     domain.XMLDesc(0))
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     six.reraise(self.type_, self.value, self.tb)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3526, in _create_domain
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     domain.createWithFlags(launch_flags)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     result = proxy_call(self._autowrap, f, *args, **kwargs)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     rv = execute(f,*args,**kwargs)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     rv = meth(*args,**kwargs)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 581, in createWithFlags
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] libvirtError: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] 
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Traceback (most recent call last):
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1184, in _run_instance
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     instance, image_meta, legacy_bdm_in_spec)
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1354, in _build_instance
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     reason=unicode(exc_info[1]))
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] RescheduledException: Build of instance 3f281136-ed69-4bfb-bf36-a7d4aa1c0640 was re-scheduled: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] 


This started right around the time we started using: https://review.openstack.org/#/c/79816/ (serge's libvirt fix for https://bugs.launchpad.net/nova/+bug/1254872)

** Affects: nova
     Importance: Undecided
         Status: New

** Affects: libvirt (Ubuntu)
     Importance: Undecided
         Status: New

** Also affects: libvirt (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1295876

Title:
   libvirtError: internal error unable to add domain xxx to cgroup: No
  space left on device

Status in OpenStack Compute (Nova):
  New
Status in “libvirt” package in Ubuntu:
  New

Bug description:
  logstash query:   message:"cgroup\: No space left on device" AND
  filename:logs*screen-n-cpu.txt

  http://logs.openstack.org/12/80412/8/check/check-tempest-dsvm-
  postgres-
  full/f9f6158/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-03-21_17_45_12_490

  
  ERROR nova.compute.manager [req-630b71d6-0fbe-4e9e-99fe-019da7d29a3a FixedIPsNegativeTestJson-475659359 FixedIPsNegativeTestJson-265680949] [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Error: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Traceback (most recent call last):
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1304, in _build_instance
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     set_access_ip=set_access_ip)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 394, in decorated_function
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     return function(self, context, *args, **kwargs)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1716, in _spawn
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     six.reraise(self.type_, self.value, self.tb)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1713, in _spawn
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     block_device_info)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2241, in spawn
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     block_device_info)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3621, in _create_domain_and_network
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     power_on=power_on)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3531, in _create_domain
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     domain.XMLDesc(0))
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     six.reraise(self.type_, self.value, self.tb)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3526, in _create_domain
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     domain.createWithFlags(launch_flags)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     rv = execute(f,*args,**kwargs)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     rv = meth(*args,**kwargs)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 581, in createWithFlags
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] libvirtError: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
  26028 TRACE nova.compute.manager [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] 
  26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] Traceback (most recent call last):
  26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1184, in _run_instance
  26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     instance, image_meta, legacy_bdm_in_spec)
  26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1354, in _build_instance
  26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640]     reason=unicode(exc_info[1]))
  26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] RescheduledException: Build of instance 3f281136-ed69-4bfb-bf36-a7d4aa1c0640 was re-scheduled: internal error unable to add domain instance-00000002 task 3057 to cgroup: No space left on device
  26028 TRACE nova.compute.utils [instance: 3f281136-ed69-4bfb-bf36-a7d4aa1c0640] 

  
  This started right around the time we started using: https://review.openstack.org/#/c/79816/ (serge's libvirt fix for https://bugs.launchpad.net/nova/+bug/1254872)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1295876/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next