yahoo-eng-team team mailing list archive

[Alan Pevec <1187102@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron/juno
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1187102

Title:
  quantum-ns-metadata-proxy listens on external interfaces too

Status in neutron:
  Fix Released
Status in neutron juno series:
  Fix Released
Status in OpenStack Security Advisory:
  Invalid

Bug description:
  Running Grizzy 2013.1 on Ubuntu 12.04. Three nodes: controller,
  network and compute.

  netnode# ip netns exec qrouter-7a44de32-3ac0-4f3e-92cc-1a37d8211db8 netstat -anp
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  tcp        0      0 0.0.0.0:9697            0.0.0.0:*               LISTEN      18462/python

  So this router is uplinked to an external network:

  netnode# ip netns exec qrouter-7a44de32-3ac0-4f3e-92cc-1a37d8211db8 ip -4 a
  14: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
      inet 127.0.0.1/8 scope host lo
  23: qr-123f9b7f-43: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
      inet 172.17.17.1/24 brd 172.17.17.255 scope global qr-123f9b7f-43
  24: qg-c8a6a6cd-6d: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
      inet 192.168.101.2/24 brd 192.168.101.255 scope global qg-c8a6a6cd-6d

  Now from outside can do:

  $ nmap 192.168.101.2 -p 9697
  Starting Nmap 6.00 ( http://nmap.org ) at 2013-06-03 13:45 IST
  Nmap scan report for 192.168.101.2
  Host is up (0.0018s latency).
  PORT     STATE SERVICE
  9697/tcp open  unknown

  As a test I tried changing namespace_proxy.py so it would not bind to
  0.0.0.0

      proxy.start(handler, self.port, host='127.0.0.1')

  but the metadata stopped working. In iptables this rule is being hit:

    -A quantum-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp
  --dport 80 -j REDIRECT --to-ports 9697

  I'm guessing the intention of that rule is also change the destination
  address to 127.0.0.1  ? as there is this:

    -A quantum-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697
  -j ACCEPT

  but the counters show that this rule is not being hit. Anyway the
  default policy for INPUT is ACCEPT.

  From the iptables man page:
    REDIRECT
         "... It redirects the packet to the machine itself by changing the destination IP to the primary address  of  the  incoming  interface
         (locally-generated packets are mapped to the 127.0.0.1 address).  ..."

  so the primary address of the incoming interface is 172.17.17.1, not
  127.0.0.1.

  So I manually deleted the "-j REDIRECT --to-ports 9697" and added "-j
  DNAT --to-destination 127.0.0.1:9697" but that didn't work - seems
  like it is not possible: http://serverfault.com/questions/351816/dnat-
  to-127-0-0-1-with-iptables-destination-access-control-for-transparent-
  soc

  So I tried changing the ns proxy to listen on 172.17.17.1. I think
  this is the one and only address it should bind to anyway.

      proxy.start(handler, self.port, host='172.17.17.1') # hardwire as
  a test

  Stopped the l3-agent, killed the quantum-ns-metadata-proxy and
  restarted the l3-agent. But the ns proxy gave an error:

  Stderr: 'cat: /proc/10850/cmdline: No such file or directory\n'
  2013-06-03 15:05:18    ERROR [quantum.wsgi] Unable to listen on 172.17.17.1:9697
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 72, in start
      backlog=backlog)
    File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 38, in listen
      sock.bind(addr)
    File "/usr/lib/python2.7/socket.py", line 224, in meth
      return getattr(self._sock,name)(*args)
  error: [Errno 99] Cannot assign requested address

  The l3-agent.log shows the agent deleted the port qr-123f9b7f-43 at
  15:05:10 and did not recreate it until 15:05:19 - ie a second too late
  for the ns proxy. From looking at the code it seems the l3-agent
  spawns the ns proxy just before it plugs its ports. I was able to
  start the ns proxy manually with the command line from the l3-agent
  log, and the metadata worked and was not reachable from outside.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1187102/+subscriptions