yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1279595] Re: [heat] stack cannot download metadata from nova-api-metadata.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Steve Baker <sbaker@xxxxxxxxxx>
Date: Thu, 13 Feb 2014 01:13:51 -0000
Reply-to: Bug 1279595 <1279595@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This looks like a nova issue, and its likely you could reproduce it by
booting a server directly with nova.

** Also affects: nova
   Importance: Undecided
       Status: New

** No longer affects: heat

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1279595

Title:
  [heat] stack cannot download metadata from nova-api-metadata.

Status in OpenStack Compute (Nova):
  New

Bug description:
  My OpenStack version is HAVANA.
  When I create stack, the log is this.
  ---------------------------------------------------------------------------------------------------------------------------------------------
  2014-02-12 04:42:59,941 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: bad status code [500]
  2014-02-12 04:43:01,441 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: bad status code [500]
  2014-02-12 04:43:02,792 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [4/120s]: bad status code [500]
  2014-02-12 04:43:04,357 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [5/120s]: bad status code [500]
  2014-02-12 04:43:05,909 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [7/120s]: bad status code [500]
  2014-02-12 04:43:07,480 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [8/120s]: bad status code [500]
  2014-02-12 04:43:10,064 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [11/120s]: bad status code [500]
  2014-02-12 04:43:12,286 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [13/120s]: bad status code [500]
  2014-02-12 04:43:14,866 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [16/120s]: bad status code [500]
  2014-02-12 04:43:17,419 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [18/120s]: bad status code [500]
  2014-02-12 04:43:19,868 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [21/120s]: bad status code [500]
  2014-02-12 04:43:23,135 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [24/120s]: bad status code [500]
  2014-02-12 04:43:26,681 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [27/120s]: bad status code [500]
  2014-02-12 04:43:30,225 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [31/120s]: bad status code [500]
  2014-02-12 04:43:33,607 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [34/120s]: bad status code [500]
  2014-02-12 04:43:36,811 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [38/120s]: bad status code [500]
  2014-02-12 04:43:41,352 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [42/120s]: bad status code [500]
  2014-02-12 04:43:46,108 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [47/120s]: bad status code [500]
  2014-02-12 04:43:50,556 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [51/120s]: bad status code [500]
  2014-02-12 04:43:54,773 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [56/120s]: bad status code [500]
  2014-02-12 04:43:58,984 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [60/120s]: bad status code [500]
  2014-02-12 04:44:04,454 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [65/120s]: bad status code [500]
  2014-02-12 04:44:09,809 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [71/120s]: bad status code [500]
  2014-02-12 04:44:15,262 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [76/120s]: bad status code [500]
  2014-02-12 04:44:20,793 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [82/120s]: bad status code [500]
  2014-02-12 04:44:26,282 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [87/120s]: bad status code [500]
  2014-02-12 04:44:32,774 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [94/120s]: bad status code [500]
  2014-02-12 04:44:39,197 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [100/120s]: bad status code [500]
  2014-02-12 04:44:45,637 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [106/120s]: bad status code [500]
  2014-02-12 04:44:52,098 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [113/120s]: bad status code [500]
  2014-02-12 04:44:58,173 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by <class 'socket.error'>: [Errno 115] Operation now in progress)]
  ---------------------------------------------------------------------------------------------------------------------------------------------

  1. this is my configuration.
  [nova.conf]
  service_neutron_metadata_proxy=true

  2. error 
  [stack : curl command, download metadata.]
  # curl http://169.254.169.254/2009-04-04/metadata
  <html>
   <head>
    <title>500 Internal Server Error</title>
   </head>
   <body>
    <h1>500 Internal Server Error</h1>
    Remote metadata server experienced an internal server error.<br /><br />


   </body>
  </html>
  --------------------------------------------------------------------------------------------------------------------------------------------
  [nova-api-metadata]
  2014-02-13 09:28:42.164 20307 INFO nova.api.ec2 [-] 0.3440s 27.122.227.181 GET /2009-04-04/metadata None:None 400 [Python-httplib2/0.7.2 (gzip)] text/plain text/plain
  2014-02-13 09:28:42.165 20307 INFO nova.metadata.wsgi.server [-] 10.0.1.2,27.122.227.181 "GET /2009-04-04/metadata HTTP/1.1" status: 400 len: 191 time: 0.0051610
  --------------------------------------------------------------------------------------------------------------------------------------------
  [neutron-metadata-agent]
  2014-02-13 09:28:42.170 31407 ERROR neutron.agent.metadata.agent [-] Unexpected error.
  2014-02-13 09:28:42.170 31407 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
  2014-02-13 09:28:42.170 31407 TRACE neutron.agent.metadata.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 97, in __call__
  2014-02-13 09:28:42.170 31407 TRACE neutron.agent.metadata.agent     return self._proxy_request(instance_id, req)
  2014-02-13 09:28:42.170 31407 TRACE neutron.agent.metadata.agent   File "/usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 179, in _proxy_request
  2014-02-13 09:28:42.170 31407 TRACE neutron.agent.metadata.agent     raise Exception(_('Unexpected response code: %s') % resp.status)
  2014-02-13 09:28:42.170 31407 TRACE neutron.agent.metadata.agent Exception: Unexpected response code: 400
  2014-02-13 09:28:42.170 31407 TRACE neutron.agent.metadata.agent
  ---------------------------------------------------------------------------------------------------------------------------------------------
  [neutron-ns-metadata-proxy] (router)
  2014-02-13 09:28:42.175 31063 DEBUG neutron.agent.metadata.namespace_proxy [-] Remote metadata server experienced an internal server error. _proxy_request /usr/local/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py:119
  ---------------------------------------------------------------------------------------------------------------------------------------------

  3. source 
  [nova/api/metadata/handler.py] : 
  149 line:
   def _handle_instance_id_request(self, req):
          instance_id = req.headers.get('X-Instance-ID')
          tenant_id = req.headers.get('X-Tenant-ID')
          signature = req.headers.get('X-Instance-ID-Signature')
          remote_address = req.headers.get('X-Forwarded-For')

  -> The problem is tenant_id/
  Tenant_id does not exist in req.headers. 
  so error occur.

  
  thx.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1279595/+subscriptions