yahoo-eng-team team mailing list archive

[Robert Pothier <rpothier@xxxxxxxxx>]

Public bug reported:

While bringing up 1000 VMs with Heat, some VMs are in error state.
After deleting the VMs, not all ports are removed.
The next time VMs are created, they fail due to no IP addresses left.


Note, initially there are 7 ports, no VMs
804 VMs are created, 811 ports

root@control01:/usr/share/pyshared/heat# nova list | grep ACT | wc -l
358
root@control01:/usr/share/pyshared/heat# neutron port-list | wc -l
811
root@control01:/usr/share/pyshared/heat# nova list | grep ERR | wc -l
270
root@control01:/usr/share/pyshared/heat# nova list | grep stack | wc -l
804

After deleting the VMs, 248 ports remain


root@control01:/usr/share/pyshared/heat# nova list | grep stack | wc -l
0
root@control01:/usr/share/pyshared/heat# neutron port-list | wc -l
248



2013-12-12 20:55:35.320 20945 ERROR nova.scheduler.filter_scheduler [req-844a11    f6-66f1-4fc7-9f3b-8bacbe57a04d e95097acd0b041558f33d07f720c1bd7 354f17bf81924b2    78806c3e3798aa527] [instance: 2a3eadcc-5230-4ed6-bf9f-9a82d43b91c3] Error from     last host: compute134 (node compute134.lab.cisco): [u'Traceback (most recent ca    ll last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.p    y", line 1037, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  Fil    e "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1410, in _sp    awn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n',     u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1407,     in _spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/dist-packag    es/nova/virt/libvirt/driver.py", line 2063, in spawn\n    admin_pass=admin_pass    word)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.p    y", line 2412, in _create_image\n    content=files, extra_md=extra_md, network_    info=network_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/api/meta    data/base.py", line 157, in __init__\n    cfg = netutils.get_injected_network_t    emplate(network_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/    netutils.py", line 74, in get_injected_network_template\n    if not (network_in    fo and template):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/m    odel.py", line 379, in __len__\n    return self._sync_wrapper(fn, *args, **kwar    gs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line     366, in _sync_wrapper\n    self.wait()\n', u'  File "/usr/lib/python2.7/dist-p    ackages/nova/network/model.py", line 398, in wait\n    self[:] = self._gt.wait(    )\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line     168, in wait\n    return self._exit_event.wait()\n', u'  File "/usr/lib/python    2.7/dist-packages/eventlet/event.py", line 120, in wait\n    current.throw(*sel    f._exc)\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py"    , line 194, in main\n    result = function(*args, **kwargs)\n', u'  File "/usr/    lib/python2.7/dist-packages/nova/compute/manager.py", line 1228, in _allocate_n    etwork_async\n    dhcp_options=dhcp_options)\n', u'  File "/usr/lib/python2.7/d    ist-packages/nova/network/api.py", line 49, in wrapper\n    res = f(self, conte    xt, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/networ    k/neutronv2/api.py", line 243, in allocate_for_instance\n    raise exception.Po    rtInUse(port_id=port_id)\n', u'PortInUse: Port 69e55016-b794-4dd9-b3f3-4e78336f    bd11 is still in use.\n']
 93 2013-12-12 20:55:35.321 20945 WARNING nova.scheduler.utils [req-844a11f6-66f1-4    fc7-9f3b-8bacbe57a04d e95097acd0b041558f33d07f720c1bd7 354f17bf81924b278806c3e3    798aa527] Failed to scheduler_run_instance: No valid host was found. Exceeded m    ax scheduling attempts 3 for instance 2a3eadcc-5230-4ed6-bf9f-9a82d43b91c3
 94 2013-12-12 20:55:35.324 20945 WARNING nova.scheduler.utils [req-844a11f6-66f1-4    fc7-9f3b-8bacbe57a04d e95097acd0b041558f33d07f720c1bd7 354f17bf81924b278806c3e3    798aa527] [instance: 2a3eadcc-5230-4ed6-bf9f-9a82d43b91c3] Setting instance to     ERROR state.

** Affects: neutron
     Importance: Undecided
         Status: New

** Attachment added: "server.log"
   https://bugs.launchpad.net/bugs/1260516/+attachment/3928318/+files/server.log

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1260516

Title:
  PortInUse exception leaves orphaned ports

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  While bringing up 1000 VMs with Heat, some VMs are in error state.
  After deleting the VMs, not all ports are removed.
  The next time VMs are created, they fail due to no IP addresses left.

  
  Note, initially there are 7 ports, no VMs
  804 VMs are created, 811 ports

  root@control01:/usr/share/pyshared/heat# nova list | grep ACT | wc -l
  358
  root@control01:/usr/share/pyshared/heat# neutron port-list | wc -l
  811
  root@control01:/usr/share/pyshared/heat# nova list | grep ERR | wc -l
  270
  root@control01:/usr/share/pyshared/heat# nova list | grep stack | wc -l
  804

  After deleting the VMs, 248 ports remain

  
  root@control01:/usr/share/pyshared/heat# nova list | grep stack | wc -l
  0
  root@control01:/usr/share/pyshared/heat# neutron port-list | wc -l
  248





  2013-12-12 20:55:35.320 20945 ERROR nova.scheduler.filter_scheduler [req-844a11    f6-66f1-4fc7-9f3b-8bacbe57a04d e95097acd0b041558f33d07f720c1bd7 354f17bf81924b2    78806c3e3798aa527] [instance: 2a3eadcc-5230-4ed6-bf9f-9a82d43b91c3] Error from     last host: compute134 (node compute134.lab.cisco): [u'Traceback (most recent ca    ll last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.p    y", line 1037, in _build_instance\n    set_access_ip=set_access_ip)\n', u'  Fil    e "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1410, in _sp    awn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n',     u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1407,     in _spawn\n    block_device_info)\n', u'  File "/usr/lib/python2.7/dist-packag    es/nova/virt/libvirt/driver.py", line 2063, in spawn\n    admin_pass=admin_pass    word)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.p    y", line 2412, in _create_image\n    content=files, extra_md=extra_md, network_    info=network_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/api/meta    data/base.py", line 157, in __init__\n    cfg = netutils.get_injected_network_t    emplate(network_info)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/virt/    netutils.py", line 74, in get_injected_network_template\n    if not (network_in    fo and template):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/m    odel.py", line 379, in __len__\n    return self._sync_wrapper(fn, *args, **kwar    gs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line     366, in _sync_wrapper\n    self.wait()\n', u'  File "/usr/lib/python2.7/dist-p    ackages/nova/network/model.py", line 398, in wait\n    self[:] = self._gt.wait(    )\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line     168, in wait\n    return self._exit_event.wait()\n', u'  File "/usr/lib/python    2.7/dist-packages/eventlet/event.py", line 120, in wait\n    current.throw(*sel    f._exc)\n', u'  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py"    , line 194, in main\n    result = function(*args, **kwargs)\n', u'  File "/usr/    lib/python2.7/dist-packages/nova/compute/manager.py", line 1228, in _allocate_n    etwork_async\n    dhcp_options=dhcp_options)\n', u'  File "/usr/lib/python2.7/d    ist-packages/nova/network/api.py", line 49, in wrapper\n    res = f(self, conte    xt, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/networ    k/neutronv2/api.py", line 243, in allocate_for_instance\n    raise exception.Po    rtInUse(port_id=port_id)\n', u'PortInUse: Port 69e55016-b794-4dd9-b3f3-4e78336f    bd11 is still in use.\n']
   93 2013-12-12 20:55:35.321 20945 WARNING nova.scheduler.utils [req-844a11f6-66f1-4    fc7-9f3b-8bacbe57a04d e95097acd0b041558f33d07f720c1bd7 354f17bf81924b278806c3e3    798aa527] Failed to scheduler_run_instance: No valid host was found. Exceeded m    ax scheduling attempts 3 for instance 2a3eadcc-5230-4ed6-bf9f-9a82d43b91c3
   94 2013-12-12 20:55:35.324 20945 WARNING nova.scheduler.utils [req-844a11f6-66f1-4    fc7-9f3b-8bacbe57a04d e95097acd0b041558f33d07f720c1bd7 354f17bf81924b278806c3e3    798aa527] [instance: 2a3eadcc-5230-4ed6-bf9f-9a82d43b91c3] Setting instance to     ERROR state.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1260516/+subscriptions