yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1501558] [NEW] test_associate_already_associated_floating_ip fails in juno with ebtables kernel failure (nova-net)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Thu, 01 Oct 2015 01:12:26 -0000
Reply-to: Bug 1501558 <1501558@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Just started seeing this show up in juno jobs:

http://logs.openstack.org/39/229639/2/check/gate-tempest-dsvm-full-
juno/a0bb0c0/logs/screen-n-net.txt.gz?level=TRACE#_2015-09-30_22_21_27_400

2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/floating_ips.py", line 393, in _associate_floating_ip
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     do_associate()
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 272, in inner
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/floating_ips.py", line 386, in do_associate
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     interface=interface)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/floating_ips.py", line 370, in do_associate
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     interface, fixed['network'])
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/l3.py", line 114, in add_floating_ip
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     l3_interface_id, network)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/linux_net.py", line 784, in ensure_floating_forward
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     ensure_ebtables_rules(*floating_ebtables_rules(fixed_ip, network))
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 272, in inner
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/linux_net.py", line 1649, in ensure_ebtables_rules
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     _execute(*cmd, run_as_root=True)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/linux_net.py", line 1229, in _execute
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return utils.execute(*cmd, **kwargs)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/utils.py", line 187, in execute
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return processutils.execute(*cmd, **kwargs)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/openstack/common/processutils.py", line 222, in execute
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     cmd=sanitized_cmd)
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ebtables -t nat -I PREROUTING --logical-in br100 -p ipv4 --ip-src 10.1.0.7 ! --ip-dst 10.1.0.0/20 -j redirect --redirect-target ACCEPT
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Exit code: 255
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Stdout: u''
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Stderr: u"Unable to update the kernel. Two possible causes:\n1. Multiple ebtables programs were executing simultaneously. The ebtables\n   userspace tool doesn't by default support multiple ebtables programs running\n   concurrently. The ebtables option --concurrent or a tool like flock can be\n   used to support concurrent scripts that update the ebtables kernel tables.\n2. The kernel doesn't support a certain ebtables extension, consider\n   recompiling your kernel or insmod the extension.\n.\n"
2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher

** Affects: nova
     Importance: Low
         Status: Confirmed


** Tags: network

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => Low

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1501558

Title:
  test_associate_already_associated_floating_ip fails in juno with
  ebtables kernel failure (nova-net)

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Just started seeing this show up in juno jobs:

  http://logs.openstack.org/39/229639/2/check/gate-tempest-dsvm-full-
  juno/a0bb0c0/logs/screen-n-net.txt.gz?level=TRACE#_2015-09-30_22_21_27_400

  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/floating_ips.py", line 393, in _associate_floating_ip
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     do_associate()
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 272, in inner
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/floating_ips.py", line 386, in do_associate
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     interface=interface)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/floating_ips.py", line 370, in do_associate
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     interface, fixed['network'])
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/l3.py", line 114, in add_floating_ip
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     l3_interface_id, network)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/linux_net.py", line 784, in ensure_floating_forward
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     ensure_ebtables_rules(*floating_ebtables_rules(fixed_ip, network))
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 272, in inner
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/linux_net.py", line 1649, in ensure_ebtables_rules
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     _execute(*cmd, run_as_root=True)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/network/linux_net.py", line 1229, in _execute
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return utils.execute(*cmd, **kwargs)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/utils.py", line 187, in execute
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     return processutils.execute(*cmd, **kwargs)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/new/nova/nova/openstack/common/processutils.py", line 222, in execute
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher     cmd=sanitized_cmd)
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ebtables -t nat -I PREROUTING --logical-in br100 -p ipv4 --ip-src 10.1.0.7 ! --ip-dst 10.1.0.0/20 -j redirect --redirect-target ACCEPT
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Exit code: 255
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Stdout: u''
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher Stderr: u"Unable to update the kernel. Two possible causes:\n1. Multiple ebtables programs were executing simultaneously. The ebtables\n   userspace tool doesn't by default support multiple ebtables programs running\n   concurrently. The ebtables option --concurrent or a tool like flock can be\n   used to support concurrent scripts that update the ebtables kernel tables.\n2. The kernel doesn't support a certain ebtables extension, consider\n   recompiling your kernel or insmod the extension.\n.\n"
  2015-09-30 22:21:27.400 24730 TRACE oslo.messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1501558/+subscriptions
Follow ups

[Bug 1501558] Re: nova-net: libvirtError: Error while building firewall: Some rules could not be created for interface: Unable to update the kernel
From: Matt Riedemann, 2016-05-13