yahoo-eng-team team mailing list archive

[ugvddm <zhengguangou@xxxxxxxxx>]

** Changed in: nova
     Assignee: ugvddm (271025598-9) => (unassigned)

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1302889

Title:
  Rescheduling causes PortInUse failure

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  I have a 10 compute nodes OpenStack Icehouse 3 environment. One of
  these compute nodes was incorrectly configured with permission issue
  on /var/lib/nova/instance (as it was mounted on SAN storage device).
  When the instance was schedule to run on this suspect compute node, it
  failed and got re-scheduled to another node. The launch failed on that
  node too, but the failure was due to PortInUse. Scheduler relaunched
  on the 3rd node, and there too it failed with PortInUse error. It
  looks like re-scheduling does not work on all the cases.

  Expected behavior: Anyone of the following:
  1. Don't reschedule an instance in such error condition. 
  2. If scheduled the port being created in the first launch must be de-coupled or detached. 

  
  nova-scheduler.log.1:2014-04-04 04:55:20.647 17861 INFO nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] Attempting to build 1 instance(s) uuids: [u'7b66bc37-cd0a-476a-a0a9-c06b84a9d31a']
  nova-scheduler.log.1:2014-04-04 04:55:20.658 17861 INFO nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] Choosing host WeighedHost [host: i3computebay7, weight: 1.0] for instance 7b66bc37-cd0a-476a-a0a9-c06b84a9d31a
  nova-scheduler.log.1:2014-04-04 04:55:22.899 17861 INFO nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] Attempting to build 1 instance(s) uuids: [u'7b66bc37-cd0a-476a-a0a9-c06b84a9d31a']
  nova-scheduler.log.1:2014-04-04 04:55:22.901 17861 ERROR nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] [instance: 7b66bc37-cd0a-476a-a0a9-c06b84a9d31a] Error from last host: i3computebay7 (node i3computebay7.cdl.local): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1065, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 360, in decorated_function\n    return function(self, context, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1475, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1472, in _spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2230, in spawn\n    admin_pass=admin_password)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2478, in _create_image\n    fileutils.ensure_tree(libvirt_utils.get_instance_path(instance))\n', u'  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 37, in ensure_tree\n    os.makedirs(path)\n', u'  File "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name, mode)\n', u"OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/7b66bc37-cd0a-476a-a0a9-c06b84a9d31a'\n"]
  nova-scheduler.log.1:2014-04-04 04:55:22.913 17861 INFO nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] Choosing host WeighedHost [host: i3computebay15, weight: 1.0] for instance 7b66bc37-cd0a-476a-a0a9-c06b84a9d31a
  nova-scheduler.log.1:2014-04-04 04:55:25.010 17861 INFO nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] Attempting to build 1 instance(s) uuids: [u'7b66bc37-cd0a-476a-a0a9-c06b84a9d31a']
  nova-scheduler.log.1:2014-04-04 04:55:25.012 17861 ERROR nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] [instance: 7b66bc37-cd0a-476a-a0a9-c06b84a9d31a] Error from last host: i3computebay15 (node i3computebay15): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1065, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 360, in decorated_function\n    return function(self, context, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1475, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1472, in _spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2234, in spawn\n    write_to_disk=True)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3386, in to_xml\n    network_info_str = str(network_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 383, in __str__\n    return self._sync_wrapper(fn, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 366, in _sync_wrapper\n    self.wait()\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 398, in wait\n    self[:] = self._gt.wait()\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait\n    return self._exit_event.wait()\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 120, in wait\n    current.throw(*self._exc)\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main\n    result = function(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1253, in _allocate_network_async\n    dhcp_options=dhcp_options)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 231, in allocate_for_instance\n    raise exception.PortInUse(port_id=port_id)\n', u'PortInUse: Port 88b161e4-b18a-40e1-bc52-59ca0ce4fac6 is still in use.\n']
  nova-scheduler.log.1:2014-04-04 04:55:25.027 17861 INFO nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] Choosing host WeighedHost [host: i3computebay10, weight: 1.0] for instance 7b66bc37-cd0a-476a-a0a9-c06b84a9d31a
  nova-scheduler.log.1:2014-04-04 04:55:27.136 17861 INFO nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] Attempting to build 1 instance(s) uuids: [u'7b66bc37-cd0a-476a-a0a9-c06b84a9d31a']
  nova-scheduler.log.1:2014-04-04 04:55:27.136 17861 ERROR nova.scheduler.filter_scheduler [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] [instance: 7b66bc37-cd0a-476a-a0a9-c06b84a9d31a] Error from last host: i3computebay10 (node i3computebay10.cdl.local): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1065, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 360, in decorated_function\n    return function(self, context, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1475, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1472, in _spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2234, in spawn\n    write_to_disk=True)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3386, in to_xml\n    network_info_str = str(network_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 383, in __str__\n    return self._sync_wrapper(fn, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 366, in _sync_wrapper\n    self.wait()\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 398, in wait\n    self[:] = self._gt.wait()\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait\n    return self._exit_event.wait()\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 120, in wait\n    current.throw(*self._exc)\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main\n    result = function(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1253, in _allocate_network_async\n    dhcp_options=dhcp_options)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 231, in allocate_for_instance\n    raise exception.PortInUse(port_id=port_id)\n', u'PortInUse: Port 88b161e4-b18a-40e1-bc52-59ca0ce4fac6 is still in use.\n']
  nova-scheduler.log.1:2014-04-04 04:55:27.137 17861 WARNING nova.scheduler.utils [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] Failed to scheduler_run_instance: No valid host was found. Exceeded max scheduling attempts 3 for instance 7b66bc37-cd0a-476a-a0a9-c06b84a9d31a
  nova-scheduler.log.1:2014-04-04 04:55:27.138 17861 WARNING nova.scheduler.utils [req-5019a4c6-febc-4ea1-84a6-0aaef318d9a9 d8f4d6653e1b4454bf0a5fbcc7d224c6 b65b8645eea0491d8e0147a8bd6c38f6] [instance: 7b66bc37-cd0a-476a-a0a9-c06b84a9d31a] Setting instance to ERROR state.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1302889/+subscriptions