← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #15478

[Bug 1328623] [NEW] Certain requests not being routed back to an instance.

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I have a new Icehouse deployment up and running with a controller, 2
network nodes, and 4 compute nodes.  I have found that I am unable to
boot certain VMs (specifically Ubuntu 14.04) because the instance hangs
when making a _specific_ call to the metadata service.

For testing, I brought a CirrOS instance.  From there I can curl any
metadata URI I want, _except_ for this:

curl http://169.254.169.254/openstack/2013-10-17/meta_data.json

Looking closer, I ran a tcpdump on the network node (in the neutron
router's namespace):

cdeng@network1:~$ sudo ip netns exec qrouter-df3f082c-b746-4fa4-9734-f4f976be3645 tcpdump -i any -nn -s0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
13:37:14.678641 IP 192.168.1.30.53737 > 169.254.169.254.80: Flags [S], seq 573754999, win 14600, options [mss 1460,sackOK,TS val 19504399 ecr 0,nop,wscale 3], length 0
13:37:14.678712 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [S.], seq 384294989, ack 573755000, win 28960, options [mss 1460,sackOK,TS val 126506530 ecr 19504399,nop,wscale 7], length 0
13:37:14.679191 IP 192.168.1.30.53737 > 169.254.169.254.80: Flags [.], ack 1, win 1825, options [nop,nop,TS val 19504399 ecr 126506530], length 0
13:37:14.679210 IP 192.168.1.30.53737 > 169.254.169.254.80: Flags [P.], seq 1:178, ack 1, win 1825, options [nop,nop,TS val 19504399 ecr 126506530], length 177
13:37:14.679279 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], ack 178, win 235, options [nop,nop,TS val 126506530 ecr 19504399], length 0
13:37:14.746920 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506547 ecr 19504399], length 1448
13:37:14.746981 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [P.], seq 1449:1609, ack 178, win 235, options [nop,nop,TS val 126506547 ecr 19504399], length 160
13:37:14.747439 IP 192.168.1.30.53737 > 169.254.169.254.80: Flags [.], ack 1, win 1825, options [nop,nop,TS val 19504416 ecr 126506530,nop,nop,sack 1 {1449:1609}], length 0
13:37:14.749387 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506548 ecr 19504416], length 1448
13:37:14.953392 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506599 ecr 19504416], length 1448
13:37:15.361396 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506701 ecr 19504416], length 1448
13:37:16.177396 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506905 ecr 19504416], length 1448
13:37:17.813398 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126507314 ecr 19504416], length 1448
13:37:21.085398 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126508132 ecr 19504416], length 1448
13:37:27.629418 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126509768 ecr 19504416], length 1448
13:37:40.717398 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126513040 ecr 19504416], length 1448
13:38:06.893427 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126519584 ecr 19504416], length 1448

I see the response, from the metadata proxy, being sent back to the
instance (192.168.1.30).  However, the instance never receives it.  The
result is that the requestor (in this case, curl) hangs forever.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1328623

Title:
  Certain requests not being routed back to an instance.

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  I have a new Icehouse deployment up and running with a controller, 2
  network nodes, and 4 compute nodes.  I have found that I am unable to
  boot certain VMs (specifically Ubuntu 14.04) because the instance
  hangs when making a _specific_ call to the metadata service.

  For testing, I brought a CirrOS instance.  From there I can curl any
  metadata URI I want, _except_ for this:

  curl http://169.254.169.254/openstack/2013-10-17/meta_data.json

  Looking closer, I ran a tcpdump on the network node (in the neutron
  router's namespace):

  cdeng@network1:~$ sudo ip netns exec qrouter-df3f082c-b746-4fa4-9734-f4f976be3645 tcpdump -i any -nn -s0 port 80
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
  13:37:14.678641 IP 192.168.1.30.53737 > 169.254.169.254.80: Flags [S], seq 573754999, win 14600, options [mss 1460,sackOK,TS val 19504399 ecr 0,nop,wscale 3], length 0
  13:37:14.678712 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [S.], seq 384294989, ack 573755000, win 28960, options [mss 1460,sackOK,TS val 126506530 ecr 19504399,nop,wscale 7], length 0
  13:37:14.679191 IP 192.168.1.30.53737 > 169.254.169.254.80: Flags [.], ack 1, win 1825, options [nop,nop,TS val 19504399 ecr 126506530], length 0
  13:37:14.679210 IP 192.168.1.30.53737 > 169.254.169.254.80: Flags [P.], seq 1:178, ack 1, win 1825, options [nop,nop,TS val 19504399 ecr 126506530], length 177
  13:37:14.679279 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], ack 178, win 235, options [nop,nop,TS val 126506530 ecr 19504399], length 0
  13:37:14.746920 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506547 ecr 19504399], length 1448
  13:37:14.746981 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [P.], seq 1449:1609, ack 178, win 235, options [nop,nop,TS val 126506547 ecr 19504399], length 160
  13:37:14.747439 IP 192.168.1.30.53737 > 169.254.169.254.80: Flags [.], ack 1, win 1825, options [nop,nop,TS val 19504416 ecr 126506530,nop,nop,sack 1 {1449:1609}], length 0
  13:37:14.749387 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506548 ecr 19504416], length 1448
  13:37:14.953392 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506599 ecr 19504416], length 1448
  13:37:15.361396 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506701 ecr 19504416], length 1448
  13:37:16.177396 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126506905 ecr 19504416], length 1448
  13:37:17.813398 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126507314 ecr 19504416], length 1448
  13:37:21.085398 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126508132 ecr 19504416], length 1448
  13:37:27.629418 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126509768 ecr 19504416], length 1448
  13:37:40.717398 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126513040 ecr 19504416], length 1448
  13:38:06.893427 IP 169.254.169.254.80 > 192.168.1.30.53737: Flags [.], seq 1:1449, ack 178, win 235, options [nop,nop,TS val 126519584 ecr 19504416], length 1448

  I see the response, from the metadata proxy, being sent back to the
  instance (192.168.1.30).  However, the instance never receives it.
  The result is that the requestor (in this case, curl) hangs forever.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1328623/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next