yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1590816] [NEW] metadata agent make invalid token requests

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bjoern Teipel <bjoern.teipel@xxxxxxxxxxxxx>
Date: Thu, 09 Jun 2016 14:44:06 -0000
Reply-to: Bug 1590816 <1590816@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Sporadically the neutron metadata agent seems to return 401 wrapped up in a 404.
For still unknown reasons, the metadata agents creates sporadically invalid v3 token requests 

2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent
Unauthorized: {"error": {"message": "The resource could not be found.",
"code": 404, "title": "Not Found"}}

POST /tokens HTTP/1.1
Host: 1.2.3.4:35357
Content-Length: 91
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient

and the response is

HTTP/1.1 404 Not Found
Date: Tue, 01 Mar 2016 22:14:58 GMT
Server: Apache
Vary: X-Auth-Token
Content-Length: 93
Content-Type: application/json

and the agent will stop responding with a full stack. At first we thought this issue would be related to a improper auth_url configuration (see https://bugs.launchpad.net/openstack-ansible/liberty/+bug/1552394) but the issue came back.
Interestingly the agent start working once we restart it but the problem slowly appears once you start putting more workload on it (spinning up instances)


2016-02-26 13:34:46.478 33371 INFO eventlet.wsgi.server [-] (33371) accepted ''
2016-02-26 13:34:46.486 33371 ERROR neutron.agent.metadata.agent [-] Unexpected error.
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 109, in __call__
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 204, in _get_instance_and_tenant_id
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 197, in _get_ports
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 101, in __call__
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_from_cache(target_self, *args, **kwargs)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 79, in _get_from_cache
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent item = self.func(target_self, *args, **kwargs)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 166, in _get_ports_for_remote_address
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ip_address=remote_address)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 135, in _get_ports_from_server
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_ports_using_client(filters)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 177, in _get_ports_using_client
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ports = client.list_ports(**filters)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ret = self.function(instance, *args, **kwargs)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 534, in list_ports
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent **_params)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 307, in list
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent for r in self._pagination(collection, path, **params):
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 320, in _pagination
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent res = self.get(path, params=params)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in get
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent headers=headers, params=params)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent headers=headers, params=params)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 200, in do_request
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent content_type=self.content_type())
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in do_request
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self.authenticate_and_fetch_endpoint_url()
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in authenticate_and_fetch_endpoint_url
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self.authenticate()
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in authenticate
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self._authenticate_keystone()
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in _authenticate_keystone
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent raise exceptions.Unauthorized(message=resp_body)
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent Unauthorized: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1590816

Title:
  metadata agent make invalid token requests

Status in neutron:
  New

Bug description:
  Sporadically the neutron metadata agent seems to return 401 wrapped up in a 404.
  For still unknown reasons, the metadata agents creates sporadically invalid v3 token requests 

  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent
  Unauthorized: {"error": {"message": "The resource could not be
  found.", "code": 404, "title": "Not Found"}}

  POST /tokens HTTP/1.1
  Host: 1.2.3.4:35357
  Content-Length: 91
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-neutronclient

  and the response is

  HTTP/1.1 404 Not Found
  Date: Tue, 01 Mar 2016 22:14:58 GMT
  Server: Apache
  Vary: X-Auth-Token
  Content-Length: 93
  Content-Type: application/json

  and the agent will stop responding with a full stack. At first we thought this issue would be related to a improper auth_url configuration (see https://bugs.launchpad.net/openstack-ansible/liberty/+bug/1552394) but the issue came back.
  Interestingly the agent start working once we restart it but the problem slowly appears once you start putting more workload on it (spinning up instances)


  2016-02-26 13:34:46.478 33371 INFO eventlet.wsgi.server [-] (33371) accepted ''
  2016-02-26 13:34:46.486 33371 ERROR neutron.agent.metadata.agent [-] Unexpected error.
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 109, in __call__
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 204, in _get_instance_and_tenant_id
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 197, in _get_ports
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_ports_for_remote_address(remote_address, networks)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 101, in __call__
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_from_cache(target_self, *args, **kwargs)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 79, in _get_from_cache
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent item = self.func(target_self, *args, **kwargs)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 166, in _get_ports_for_remote_address
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ip_address=remote_address)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 135, in _get_ports_from_server
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent return self._get_ports_using_client(filters)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 177, in _get_ports_using_client
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ports = client.list_ports(**filters)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent ret = self.function(instance, *args, **kwargs)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 534, in list_ports
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent **_params)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 307, in list
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent for r in self._pagination(collection, path, **params):
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 320, in _pagination
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent res = self.get(path, params=params)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in get
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent headers=headers, params=params)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270, in retry_request
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent headers=headers, params=params)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 200, in do_request
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent content_type=self.content_type())
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in do_request
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self.authenticate_and_fetch_endpoint_url()
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in authenticate_and_fetch_endpoint_url
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self.authenticate()
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in authenticate
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent self._authenticate_keystone()
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in _authenticate_keystone
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent raise exceptions.Unauthorized(message=resp_body)
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent Unauthorized: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
  2016-02-26 13:34:46.486 33371 TRACE neutron.agent.metadata.agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1590816/+subscriptions
Follow ups

[Bug 1590816] Re: metadata agent make invalid token requests
From: Ian Cordasco, 2016-06-10