yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1091780] Re: nova-network - "iptables-restore v1.4.12: host/network `None' not found

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Joe Gordon <1091780@xxxxxxxxxxxxxxxxxx>
Date: Wed, 19 Nov 2014 19:03:08 -0000
Reply-to: Bug 1091780 <1091780@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1091780

Title:
  nova-network - "iptables-restore v1.4.12: host/network `None' not
  found

Status in OpenStack Compute (Nova):
  Invalid
Status in “nova” package in Ubuntu:
  Expired

Bug description:
  1- In Precise nova-network crashes because it cannot apply iptables
  rules when trying to apply vpn rules. nova-network tries to set VPN
  iptables rules for openvpn access:

  2012-12-17 07:17:24 TRACE nova Stderr: "iptables-restore v1.4.12:
  host/network `None' not found\nError occurred at line: 23\nTry
  `iptables-restore -h' or 'iptables-restore --help' for more
  information.\n"

  2- How reproducible?

  Not clear. The configuration I used with juju seems to create an
  environment that causes this problem. When this problem is present the
  issue reproduces every time.

  3- How to reproduce:

  When the issue is present just starting up nova-network causes the
  problem to reproduce. Nova-network exits in the end and dies because
  of the error on iptables-restore

  4- I added debugging in nova.conf with --debug=true and added extra
  debugging in

  /usr/lib/python2.7/dist-packages/nova/utils.py

  which showed the full iptables rules that were to be restored by
  iptables-restore:

  2012-12-17 07:17:24 DEBUG nova.utils [req-391688fd-3b99-4b1c-8b46-fb4f64e64246 None None] process input: 
  # Generated by iptables-save v1.4.12 on Mon Dec 17 07:17:21 2012
  *filter
  :INPUT ACCEPT [0:0]
  :FORWARD ACCEPT [0:0]
  :OUTPUT ACCEPT [0:0]
  :nova-api-FORWARD - [0:0]
  :nova-api-INPUT - [0:0]
  :nova-api-OUTPUT - [0:0]
  :nova-api-local - [0:0]
  :nova-network-FORWARD - [0:0]
  :nova-network-INPUT - [0:0]
  :nova-network-local - [0:0]
  :nova-network-OUTPUT - [0:0]
  :nova-filter-top - [0:0]
  -A FORWARD -j nova-filter-top
  -A OUTPUT -j nova-filter-top
  -A nova-filter-top -j nova-network-local
  -A INPUT -j nova-network-INPUT
  -A OUTPUT -j nova-network-OUTPUT
  -A FORWARD -j nova-network-FORWARD
  -A nova-network-FORWARD --in-interface br100 -j ACCEPT
  -A nova-network-FORWARD --out-interface br100 -j ACCEPT
  -A nova-network-FORWARD -d None -p udp --dport 1194 -j ACCEPT
  -A INPUT -j nova-api-INPUT
  -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
  -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
  -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
  -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
  -A FORWARD -j nova-api-FORWARD
  -A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
  -A FORWARD -i virbr0 -o virbr0 -j ACCEPT
  -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
  -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
  -A OUTPUT -j nova-api-OUTPUT
  -A nova-api-INPUT -d 192.168.124.150/32 -p tcp -m tcp --dport 8775 -j ACCEPT
  -A nova-filter-top -j nova-api-local
  COMMIT

  
  4.1- Among the rules above we have:

  -A nova-network-FORWARD -d None -p udp --dport 1194 -j ACCEPT

  which is responsible for the fault in iptables-restore.

  5- These are the error messages:

  2012-12-17 07:17:24 DEBUG nova.utils [req-391688fd-3b99-4b1c-8b46-fb4f64e64246 None None] Result was 2 from (pid=14699) execute /usr/lib/python2.7/dist-packages/nova/utils.py:237
  2012-12-17 07:17:24 CRITICAL nova [-] Unexpected error while running command.
  Command: sudo nova-rootwrap iptables-restore
  Exit code: 2
  Stdout: ''

  Stderr: "iptables-restore v1.4.12: host/network `None' not found\nError occurred at line: 23\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"
  2012-12-17 07:17:24 TRACE nova Traceback (most recent call last):
  2012-12-17 07:17:24 TRACE nova   File "/usr/bin/nova-network", line 49, in <module>
  2012-12-17 07:17:24 TRACE nova     service.wait()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait
  2012-12-17 07:17:24 TRACE nova     _launcher.wait()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait
  2012-12-17 07:17:24 TRACE nova     service.wait()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
  2012-12-17 07:17:24 TRACE nova     return self._exit_event.wait()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
  2012-12-17 07:17:24 TRACE nova     return hubs.get_hub().switch()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
  2012-12-17 07:17:24 TRACE nova     return self.greenlet.switch()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
  2012-12-17 07:17:24 TRACE nova     result = function(*args, **kwargs)
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 101, in run_server
  2012-12-17 07:17:24 TRACE nova     server.start()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in start
  2012-12-17 07:17:24 TRACE nova     self.manager.init_host()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1766, in init_host
  2012-12-17 07:17:24 TRACE nova     NetworkManager.init_host(self)
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 758, in init_host
  2012-12-17 07:17:24 TRACE nova     self._setup_network_on_host(ctxt, network)
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1853, in _setup_network_on_host
  2012-12-17 07:17:24 TRACE nova     network['vpn_private_address'])
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/network/l3.py", line 113, in add_vpn
  2012-12-17 07:17:24 TRACE nova     linux_net.ensure_vpn_forward(public_ip, port, private_ip)
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 499, in ensure_vpn_forward
  2012-12-17 07:17:24 TRACE nova     iptables_manager.apply()
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 945, in inner
  2012-12-17 07:17:24 TRACE nova     retval = f(*args, **kwargs)
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 334, in apply
  2012-12-17 07:17:24 TRACE nova     attempts=5)
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 813, in _execute
  2012-12-17 07:17:24 TRACE nova     return utils.execute(*cmd, **kwargs)
  2012-12-17 07:17:24 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 244, in execute
  2012-12-17 07:17:24 TRACE nova     cmd=' '.join(cmd))
  2012-12-17 07:17:24 TRACE nova ProcessExecutionError: Unexpected error while running command.
  2012-12-17 07:17:24 TRACE nova Command: sudo nova-rootwrap iptables-restore
  2012-12-17 07:17:24 TRACE nova Exit code: 2
  2012-12-17 07:17:24 TRACE nova Stdout: ''
  2012-12-17 07:17:24 TRACE nova Stderr: "iptables-restore v1.4.12: host/network `None' not found\nError occurred at line: 23\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"

  6- The issue happens within:

  network/l3.py:

      def add_vpn(self, public_ip, port, private_ip):
          linux_net.ensure_vpn_forward(public_ip, port, private_ip)

  6.1- which calls, and ensure_vpn_forward, which doesn't do input check
  to see if private_ip or public_ip are actual IP addresses.

  network/linux_net.py:

  def ensure_vpn_forward(public_ip, port, private_ip):
      """Sets up forwarding rules for vlan."""

      iptables_manager.ipv4['filter'].add_rule('FORWARD',
                                               '-d %s -p udp '
                                               '--dport 1194 '
                                               '-j ACCEPT' % private_ip)
      iptables_manager.ipv4['nat'].add_rule('PREROUTING',
                                            '-d %s -p udp '
                                            '--dport %s -j DNAT --to %s:1194' %
                                            (public_ip, port, private_ip))
      iptables_manager.ipv4['nat'].add_rule("OUTPUT",
                                            "-d %s -p udp "
                                            "--dport %s -j DNAT --to %s:1194" %
                                            (public_ip, port, private_ip))
      iptables_manager.apply()

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1091780/+subscriptions