yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Fixed in 1.5.0 in Pike: https://review.openstack.org/#/c/438272/

** Also affects: os-vif/newton
   Importance: Undecided
       Status: New

** Also affects: os-vif/ocata
   Importance: Undecided
       Status: New

** Changed in: os-vif
       Status: Confirmed => Fix Released

** Changed in: os-vif
     Assignee: (unassigned) => Kevin Benton (kevinbenton)

** Changed in: os-vif/newton
       Status: New => In Progress

** Changed in: os-vif/ocata
       Status: New => In Progress

** Changed in: os-vif/ocata
     Assignee: (unassigned) => Kevin Benton (kevinbenton)

** Changed in: os-vif/newton
     Assignee: (unassigned) => Ihar Hrachyshka (ihar-hrachyshka)

** Changed in: os-vif
   Importance: Undecided => High

** Changed in: os-vif/newton
   Importance: Undecided => High

** Changed in: os-vif/ocata
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1694769

Title:
  Nova fails to plug port because of missing ipset when calling
  iptables-restore

Status in neutron:
  Won't Fix
Status in os-vif:
  Fix Released
Status in os-vif newton series:
  In Progress
Status in os-vif ocata series:
  In Progress

Bug description:
  This is Ocata, linuxbridge.

  http://logs.openstack.org/95/466395/3/gate/gate-tempest-dsvm-neutron-
  linuxbridge-ubuntu-xenial/e5923b4/logs/testr_results.html.gz

    File "tempest/common/compute.py", line 188, in create_test_server
      clients.servers_client, server['id'], wait_until)
    File "tempest/common/waiters.py", line 76, in wait_for_server_status
      server_id=server_id)
  tempest.exceptions.BuildErrorException: Server 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d failed to build and is in ERROR status
  Details: {u'created': u'2017-05-27T03:00:23Z', u'code': 500, u'message': u'No valid host was found. There are not enough hosts available.'}

  The failure in nova-cpu log:
  http://logs.openstack.org/95/466395/3/gate/gate-tempest-dsvm-neutron-
  linuxbridge-ubuntu-
  xenial/e5923b4/logs/screen-n-cpu.txt.gz#_2017-05-27_03_00_21_716

  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [req-06c29149-80d9-4923-b9c4-54591a3f5e7e tempest-ServerActionsTestJSON-1792219232 tempest-ServerActionsTestJSON-1792219232] [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Instance failed to spawn
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Traceback (most recent call last):
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2124, in _build_resources
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     yield resources
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1930, in _build_and_run_instance
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     block_device_info=block_device_info)
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2698, in spawn
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     destroy_disks_on_failure=True)
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5114, in _create_domain_and_network
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     destroy_disks_on_failure)
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     self.force_reraise()
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     six.reraise(self.type_, self.value, self.tb)
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5077, in _create_domain_and_network
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     self.plug_vifs(instance, network_info)
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 749, in plug_vifs
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     self.vif_driver.plug(instance, vif)
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 786, in plug
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     self._plug_os_vif(instance, vif_obj)
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]   File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 766, in _plug_os_vif
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d]     raise exception.InternalError(msg)
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] InternalError: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:16:2c:4d,bridge_name='brq9c933655-e1',has_traffic_filtering=True,id=416d65ee-709e-4b50-a0f1-23d988773b9f,network=Network(9c933655-e176-41b2-9b3a-8e46b13450ca),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap416d65ee-70'). Got error: Unexpected error while running command.
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Command: iptables-restore -c
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Exit code: 2
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Stdout: u''
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] Stderr: u"iptables-restore v1.6.0: Set NIPv47e2555da-67d1-4e12-9317- doesn't exist.\n\nError occurred at line: 202\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"
  2017-05-27 03:00:21.716 1385 ERROR nova.compute.manager [instance: 2a04ac11-2ec6-4a0d-a8f5-c89d129e881d] 

  I see neutron l2 agent logging destroying/creating the ipset in span
  of the job run, including around the time Nova failed to plug:

  2017-05-27 03:00:18.250 30306 DEBUG neutron.agent.linux.utils [req-
  1a4f80b9-372f-4eb4-be1c-f88cadc300ef - -] Running command (rootwrap
  daemon): ['ipset', 'destroy', 'NIPv47e2555da-67d1-4e12-9317-']
  execute_rootwrap_daemon
  /opt/stack/new/neutron/neutron/agent/linux/utils.py:113

  I suspect there is some race between linuxbridge agent and nova
  compute.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1694769/+subscriptions