← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #32770

[Bug 1453666] [NEW] libvirt: guestfs api makes nova-compute hang

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Latest Kilo code.

In inspect_capabilities() of nova/virt/disk/vfs/guestfs.py, guestfs api,
which is C-extension, will hang nova-compute process when it is invoked.
This problem will result in message queue time out error and instance
booting failure.

And example of this problem is:

2015-05-09 17:07:08.393 4449 DEBUG nova.virt.disk.vfs.api [req-1f7c1104-2679-43a5-bbcb-f73114ce9103 - - - - -] Using primary VFSGuestFS instance_for_image /usr/lib/python2.7/site-packages/nova/virt/disk/vfs/api.py:50
2015-05-09 17:08:35.443 4449 DEBUG nova.virt.disk.vfs.guestfs [req-1f7c1104-2679-43a5-bbcb-f73114ce9103 - - - - -] Setting up appliance for /var/lib/nova/instances/0517e2a9-469c-43f4-a129-f489fc1c8356/disk qcow2 setup /usr/lib/python2.7/site-packages/nova/virt/disk/vfs/guestfs.py:169
2015-05-09 17:08:35.457 4449 DEBUG nova.openstack.common.periodic_task [req-bb78b74b-bed7-450f-bd40-19686aab2c3e - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:219
2015-05-09 17:08:35.461 4449 INFO oslo_messaging._drivers.impl_rabbit [req-bb78b74b-bed7-450f-bd40-19686aab2c3e - - - - -] Connecting to AMQP server on 127.0.0.1:5671
2015-05-09 17:08:35.472 4449 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager Traceback (most recent call last):
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1783, in _allocate_network_async
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     system_metadata=sys_meta)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 739, in _instance_update
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     **kwargs)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 308, in instance_update
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     updates, 'conductor')
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 194, in instance_update
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     service=service)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     retry=self.retry)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     timeout=timeout, retry=retry)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     retry=retry)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     result = self._waiter.wait(msg_id, timeout)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     message = self.waiters.get(msg_id, timeout=timeout)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     'to message ID %s' % msg_id)
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager MessagingTimeout: Timed out waiting for a reply to message ID 8ff07520ea8743c997b5017f6638a0df
2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1453666

Title:
  libvirt: guestfs api makes nova-compute hang

Status in OpenStack Compute (Nova):
  New

Bug description:
  Latest Kilo code.

  In inspect_capabilities() of nova/virt/disk/vfs/guestfs.py, guestfs
  api, which is C-extension, will hang nova-compute process when it is
  invoked. This problem will result in message queue time out error and
  instance booting failure.

  And example of this problem is:

  2015-05-09 17:07:08.393 4449 DEBUG nova.virt.disk.vfs.api [req-1f7c1104-2679-43a5-bbcb-f73114ce9103 - - - - -] Using primary VFSGuestFS instance_for_image /usr/lib/python2.7/site-packages/nova/virt/disk/vfs/api.py:50
  2015-05-09 17:08:35.443 4449 DEBUG nova.virt.disk.vfs.guestfs [req-1f7c1104-2679-43a5-bbcb-f73114ce9103 - - - - -] Setting up appliance for /var/lib/nova/instances/0517e2a9-469c-43f4-a129-f489fc1c8356/disk qcow2 setup /usr/lib/python2.7/site-packages/nova/virt/disk/vfs/guestfs.py:169
  2015-05-09 17:08:35.457 4449 DEBUG nova.openstack.common.periodic_task [req-bb78b74b-bed7-450f-bd40-19686aab2c3e - - - - -] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:219
  2015-05-09 17:08:35.461 4449 INFO oslo_messaging._drivers.impl_rabbit [req-bb78b74b-bed7-450f-bd40-19686aab2c3e - - - - -] Connecting to AMQP server on 127.0.0.1:5671
  2015-05-09 17:08:35.472 4449 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager Traceback (most recent call last):
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1783, in _allocate_network_async
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     system_metadata=sys_meta)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 739, in _instance_update
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     **kwargs)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 308, in instance_update
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     updates, 'conductor')
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 194, in instance_update
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     service=service)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     retry=self.retry)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     timeout=timeout, retry=retry)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     retry=retry)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     result = self._waiter.wait(msg_id, timeout)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     message = self.waiters.get(msg_id, timeout=timeout)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager     'to message ID %s' % msg_id)
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager MessagingTimeout: Timed out waiting for a reply to message ID 8ff07520ea8743c997b5017f6638a0df
  2015-05-09 17:08:35.472 4449 TRACE nova.compute.manager

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1453666/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next