yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1712185] [NEW] iptables-restore calls fail acquiring 'xlock' with iptables from master

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <1712185@xxxxxxxxxxxxxxxxxx>
Date: Mon, 21 Aug 2017 19:46:51 -0000
Reply-to: Bug 1712185 <1712185@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

This happens when you use iptables that includes
https://git.netfilter.org/iptables/commit/?id=999eaa241212d3952ddff39a99d0d55a74e3639e
(f.e. the one from latest RHEL repos)

neutron.tests.functional.agent.test_firewall.FirewallTestCase.test_established_connection_is_cut(IptablesFirewallDriver,without ipset)
--------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "neutron/tests/functional/agent/test_firewall.py", line 113, in setUp
        self.firewall.prepare_port_filter(self.src_port_desc)
      File "neutron/agent/linux/iptables_firewall.py", line 204, in prepare_port_filter
        return self.iptables.apply()
      File "neutron/agent/linux/iptables_manager.py", line 432, in apply
        return self._apply()
      File "neutron/agent/linux/iptables_manager.py", line 440, in _apply
        first = self._apply_synchronized()
      File "neutron/agent/linux/iptables_manager.py", line 539, in _apply_synchronized
        '
'.join(log_lines))
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "neutron/agent/linux/iptables_manager.py", line 518, in _apply_synchronized
        run_as_root=True)
      File "neutron/agent/linux/utils.py", line 156, in execute
        raise ProcessExecutionError(msg, returncode=returncode)
    neutron.agent.linux.utils.ProcessExecutionError: Exit code: 4; Stdin: # Generated by iptables_manager
    *filter
    :neutron-filter-top - [0:0]
    :run.py-FORWARD - [0:0]
    :run.py-INPUT - [0:0]
    :run.py-OUTPUT - [0:0]
    :run.py-it-veth0bc5 - [0:0]
    :run.py-local - [0:0]
    :run.py-ot-veth0bc5 - [0:0]
    :run.py-sg-chain - [0:0]
    :run.py-sg-fallback - [0:0]
    -I FORWARD 1 -j neutron-filter-top
    -I FORWARD 2 -j run.py-FORWARD
    -I INPUT 1 -j run.py-INPUT
    -I OUTPUT 1 -j neutron-filter-top
    -I OUTPUT 2 -j run.py-OUTPUT
    -I neutron-filter-top 1 -j run.py-local
    -I run.py-FORWARD 1 -m physdev --physdev-out test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j run.py-sg-chain
    -I run.py-FORWARD 2 -m physdev --physdev-in test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j run.py-sg-chain
    -I run.py-INPUT 1 -m physdev --physdev-in test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j run.py-ot-veth0bc5
    -I run.py-it-veth0bc5 1 -p ipv6-icmp -m icmp6 --icmpv6-type 130 -j RETURN
    -I run.py-it-veth0bc5 2 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j RETURN
    -I run.py-it-veth0bc5 3 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j RETURN
    -I run.py-it-veth0bc5 4 -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j RETURN
    -I run.py-it-veth0bc5 5 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
    -I run.py-it-veth0bc5 6 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
    -I run.py-it-veth0bc5 7 -m comment --comment "Send unmatched traffic to the fallback chain." -j run.py-sg-fallback
    -I run.py-ot-veth0bc5 1 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 131 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN
    -I run.py-ot-veth0bc5 2 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN
    -I run.py-ot-veth0bc5 3 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN
    -I run.py-ot-veth0bc5 4 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m comment --comment "Drop IPv6 Router Advts from VM Instance." -j DROP
    -I run.py-ot-veth0bc5 5 -p ipv6-icmp -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN
    -I run.py-ot-veth0bc5 6 -p udp -m udp --sport 546 --dport 547 -m comment --comment "Allow DHCP client traffic." -j RETURN
    -I run.py-ot-veth0bc5 7 -p udp -m udp --sport 547 --dport 546 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP
    -I run.py-ot-veth0bc5 8 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
    -I run.py-ot-veth0bc5 9 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
    -I run.py-ot-veth0bc5 10 -m comment --comment "Send unmatched traffic to the fallback chain." -j run.py-sg-fallback
    -I run.py-sg-chain 1 -m physdev --physdev-out test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j run.py-it-veth0bc5
    -I run.py-sg-chain 2 -m physdev --physdev-in test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j run.py-ot-veth0bc5
    -I run.py-sg-chain 3 -j ACCEPT
    -I run.py-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP
    COMMIT
    # Completed by iptables_manager
    # Generated by iptables_manager
    *raw
    :run.py-OUTPUT - [0:0]
    :run.py-PREROUTING - [0:0]
    -I OUTPUT 1 -j run.py-OUTPUT
    -I PREROUTING 1 -j run.py-PREROUTING
    -I run.py-PREROUTING 1 -m physdev --physdev-in brq7a7f000b-b8 -m comment --comment "Set zone for -veth0bc5b8" -j CT --zone 1
    -I run.py-PREROUTING 2 -i brq7a7f000b-b8 -m comment --comment "Set zone for -veth0bc5b8" -j CT --zone 1
    -I run.py-PREROUTING 3 -m physdev --physdev-in test-veth0bc5b8 -m comment --comment "Set zone for -veth0bc5b8" -j CT --zone 1
    COMMIT
    # Completed by iptables_manager
    ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?

To stay on safe side, we should always call with -w (assuming it's
supported by platform).

** Affects: neutron
     Importance: Medium
         Status: Confirmed


** Tags: functional-tests

** Changed in: neutron
   Importance: Undecided => Medium

** Changed in: neutron
       Status: New => Confirmed

** Tags added: functional-tests

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1712185

Title:
  iptables-restore calls fail acquiring 'xlock' with iptables from
  master

Status in neutron:
  Confirmed

Bug description:
  This happens when you use iptables that includes
  https://git.netfilter.org/iptables/commit/?id=999eaa241212d3952ddff39a99d0d55a74e3639e
  (f.e. the one from latest RHEL repos)

  neutron.tests.functional.agent.test_firewall.FirewallTestCase.test_established_connection_is_cut(IptablesFirewallDriver,without ipset)
  --------------------------------------------------------------------------------------------------------------------------------------

  Captured traceback:
  ~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):
        File "neutron/tests/functional/agent/test_firewall.py", line 113, in setUp
          self.firewall.prepare_port_filter(self.src_port_desc)
        File "neutron/agent/linux/iptables_firewall.py", line 204, in prepare_port_filter
          return self.iptables.apply()
        File "neutron/agent/linux/iptables_manager.py", line 432, in apply
          return self._apply()
        File "neutron/agent/linux/iptables_manager.py", line 440, in _apply
          first = self._apply_synchronized()
        File "neutron/agent/linux/iptables_manager.py", line 539, in _apply_synchronized
          '
  '.join(log_lines))
        File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
          self.force_reraise()
        File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
          six.reraise(self.type_, self.value, self.tb)
        File "neutron/agent/linux/iptables_manager.py", line 518, in _apply_synchronized
          run_as_root=True)
        File "neutron/agent/linux/utils.py", line 156, in execute
          raise ProcessExecutionError(msg, returncode=returncode)
      neutron.agent.linux.utils.ProcessExecutionError: Exit code: 4; Stdin: # Generated by iptables_manager
      *filter
      :neutron-filter-top - [0:0]
      :run.py-FORWARD - [0:0]
      :run.py-INPUT - [0:0]
      :run.py-OUTPUT - [0:0]
      :run.py-it-veth0bc5 - [0:0]
      :run.py-local - [0:0]
      :run.py-ot-veth0bc5 - [0:0]
      :run.py-sg-chain - [0:0]
      :run.py-sg-fallback - [0:0]
      -I FORWARD 1 -j neutron-filter-top
      -I FORWARD 2 -j run.py-FORWARD
      -I INPUT 1 -j run.py-INPUT
      -I OUTPUT 1 -j neutron-filter-top
      -I OUTPUT 2 -j run.py-OUTPUT
      -I neutron-filter-top 1 -j run.py-local
      -I run.py-FORWARD 1 -m physdev --physdev-out test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j run.py-sg-chain
      -I run.py-FORWARD 2 -m physdev --physdev-in test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct traffic from the VM interface to the security group chain." -j run.py-sg-chain
      -I run.py-INPUT 1 -m physdev --physdev-in test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j run.py-ot-veth0bc5
      -I run.py-it-veth0bc5 1 -p ipv6-icmp -m icmp6 --icmpv6-type 130 -j RETURN
      -I run.py-it-veth0bc5 2 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j RETURN
      -I run.py-it-veth0bc5 3 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j RETURN
      -I run.py-it-veth0bc5 4 -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j RETURN
      -I run.py-it-veth0bc5 5 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
      -I run.py-it-veth0bc5 6 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
      -I run.py-it-veth0bc5 7 -m comment --comment "Send unmatched traffic to the fallback chain." -j run.py-sg-fallback
      -I run.py-ot-veth0bc5 1 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 131 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN
      -I run.py-ot-veth0bc5 2 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN
      -I run.py-ot-veth0bc5 3 -s ::/128 -d ff02::/16 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN
      -I run.py-ot-veth0bc5 4 -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m comment --comment "Drop IPv6 Router Advts from VM Instance." -j DROP
      -I run.py-ot-veth0bc5 5 -p ipv6-icmp -m comment --comment "Allow IPv6 ICMP traffic." -j RETURN
      -I run.py-ot-veth0bc5 6 -p udp -m udp --sport 546 --dport 547 -m comment --comment "Allow DHCP client traffic." -j RETURN
      -I run.py-ot-veth0bc5 7 -p udp -m udp --sport 547 --dport 546 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP
      -I run.py-ot-veth0bc5 8 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
      -I run.py-ot-veth0bc5 9 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
      -I run.py-ot-veth0bc5 10 -m comment --comment "Send unmatched traffic to the fallback chain." -j run.py-sg-fallback
      -I run.py-sg-chain 1 -m physdev --physdev-out test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j run.py-it-veth0bc5
      -I run.py-sg-chain 2 -m physdev --physdev-in test-veth0bc5b8 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j run.py-ot-veth0bc5
      -I run.py-sg-chain 3 -j ACCEPT
      -I run.py-sg-fallback 1 -m comment --comment "Default drop rule for unmatched traffic." -j DROP
      COMMIT
      # Completed by iptables_manager
      # Generated by iptables_manager
      *raw
      :run.py-OUTPUT - [0:0]
      :run.py-PREROUTING - [0:0]
      -I OUTPUT 1 -j run.py-OUTPUT
      -I PREROUTING 1 -j run.py-PREROUTING
      -I run.py-PREROUTING 1 -m physdev --physdev-in brq7a7f000b-b8 -m comment --comment "Set zone for -veth0bc5b8" -j CT --zone 1
      -I run.py-PREROUTING 2 -i brq7a7f000b-b8 -m comment --comment "Set zone for -veth0bc5b8" -j CT --zone 1
      -I run.py-PREROUTING 3 -m physdev --physdev-in test-veth0bc5b8 -m comment --comment "Set zone for -veth0bc5b8" -j CT --zone 1
      COMMIT
      # Completed by iptables_manager
      ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?

  To stay on safe side, we should always call with -w (assuming it's
  supported by platform).

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1712185/+subscriptions
Follow ups

[Bug 1712185] Re: iptables-restore calls fail acquiring 'xlock' with iptables from master
From: OpenStack Infra, 2017-09-06