← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1624494] Re: [master] metadata is not working on multi-node setup

 

** Also affects: networking-vsphere
   Importance: Undecided
       Status: New

** Changed in: neutron
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1624494

Title:
  [master] metadata is not working on multi-node setup

Status in devstack:
  New
Status in networking-vsphere:
  New
Status in neutron:
  Incomplete

Bug description:
  Setup:
  1. One controller
  2. KVM and ESX nova compute
  3. Two Network nodes running q-dhcp and q-meta

  Nodes: Ubuntu 14.04.3 amd64 Trusty

  On above setup, tried to run few metadata queries we see 500 internal
  server error.

  $ route -n
  Kernel IP routing table
  Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
  0.0.0.0         1.1.107.1       0.0.0.0         UG    0      0        0 eth0
  1.1.107.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
  169.254.169.254 1.1.107.1       255.255.255.255 UGH   0      0        0 eth0
  $ curl http://169.254.169.254
  500 Internal Server Error

  Remote metadata server experienced an internal server error.

  $

  NOTE: Moved all services to single node(controller) and metadata
  queries seems to work fine.

  vmware@cntr11:~$ neutron agent-list
  +--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
  | id                                   | agent_type     | host      | availability_zone | alive | admin_state_up | binary                 |
  +--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
  | 0168206e-6c13-40df-a0b8-7772220ab9cd | DHCP agent     | network-1 | nova              | :-)   | True           | neutron-dhcp-agent     |
  | 08c8bb6e-0c7f-41e3-b134-81e2d5334aea | Metadata agent | network-1 |                   | :-)   | True           | neutron-metadata-agent |
  | 25e9fa6d-6acc-4b8e-bb0b-f2d3ac20981d | Metadata agent | network-2 |                   | :-)   | True           | neutron-metadata-agent |
  | 534d349c-8830-4648-814b-611a30f59287 | DHCP agent     | network-2 | nova              | :-)   | True           | neutron-dhcp-agent     |
  +--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+
  vmware@cntr11:~$ 
  vmware@cntr11:~$ nova service-list
  +----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
  | Id | Binary           | Host          | Zone     | Status  | State | Updated_at                 | Disabled Reason |
  +----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
  | 7  | nova-conductor   | cntr11        | internal | enabled | up    | 2016-09-16T10:00:22.000000 | -               |
  | 9  | nova-compute     | esx-ubuntu-01 | nova     | enabled | up    | 2016-09-16T10:00:14.000000 | -               |
  | 10 | nova-compute     | kvm-3         | nova     | enabled | up    | 2016-09-16T10:00:23.000000 | -               |
  | 11 | nova-compute     | kvm-2         | nova     | enabled | up    | 2016-09-16T10:00:19.000000 | -               |
  | 12 | nova-compute     | kvm-1         | nova     | enabled | up    | 2016-09-16T10:00:19.000000 | -               |
  | 13 | nova-scheduler   | cntr11        | internal | enabled | up    | 2016-09-16T10:00:15.000000 | -               |
  | 14 | nova-consoleauth | cntr11        | internal | enabled | up    | 2016-09-16T10:00:20.000000 | -               |
  +----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
  vmware@cntr11:~$ 

  Logs:
  2016-09-13 13:31:50.713 14309 DEBUG eventlet.wsgi.server [-] (14309) accepted '' server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
  2016-09-13 13:31:50.715 14309 DEBUG neutron.agent.metadata.agent [-] Request: GET / HTTP/1.0^M
  Accept-Encoding: gzip, deflate^M
  Content-Length: 0^M
  Content-Type: text/plain^M
  Host: 169.254.169.254^M
  User-Agent: Python-httplib2/0.9.2 (gzip)^M
  X-Forwarded-For: 1.1.107.3^M
  X-Neutron-Router-Id: bbe453a5-db77-4cd9-af02-31232a222f16 __call__ /opt/stack/neutron/neutron/agent/metadata/agent.py:86
  2016-09-13 13:31:50.716 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 50b283bdeab945cb93872300abd5b47c exchange 'neutron' topic 'q-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
  2016-09-13 13:31:50.807 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 50b283bdeab945cb93872300abd5b47c __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
  2016-09-13 13:31:50.808 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 52646ead6313405fb688b80cbb4bfc73 exchange 'neutron' topic 'q-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
  2016-09-13 13:31:50.846 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 52646ead6313405fb688b80cbb4bfc73 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent [-] Unexpected error.
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 90, in __call__
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     return self._proxy_request(instance_id, tenant_id, req)
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 193, in _proxy_request
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     body=req.body)
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     response = conn.getresponse()
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/httplib.py", line 1129, in getresponse
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     response = self.response_class(*args, **kwds)
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/httplib.py", line 390, in __init__
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     self.fp = sock.makefile('rb', 0)
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 298, in makefile
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     dupped = self.dup()
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 289, in dup
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent     newsock = type(self)(sock, set_nonblocking=False)
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent TypeError: __init__() got an unexpected keyword argument 'set_nonblocking'
  2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent 
  2016-09-13 13:31:50.849 14309 INFO eventlet.wsgi.server [-] 1.1.107.3,<local> - - [13/Sep/2016 13:31:50] "GET / HTTP/1.1" 500 229 0.134582
  2016-09-13 13:31:51.854 13766 DEBUG oslo_messaging._drivers.amqpdriver [-] CAST unique_id: 757d1802bce344a18d2ae1df8221058f exchange 'neutron' topic 'q-reports-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448

To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1624494/+subscriptions