yahoo-eng-team team mailing list archive

[Sam Morrison <1362766@xxxxxxxxxxxxxxxxxx>]

I'm getting this in nova-api too, we don't use neutron.

I get it when I do a nova list or nova show. Restarting nova-api fixed
it for a while but then it comes back again.

==> /var/log/nova/nova-api.log <==
2015-03-27 13:56:06.649 10962 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-03-27 13:56:07.153 10962 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-03-27 13:56:08.156 10962 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-03-27 13:56:10.159 10962 ERROR keystonemiddleware.auth_token [-] HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure'
2015-03-27 13:56:10.161 10962 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2015-03-27 13:56:10.162 10962 INFO nova.osapi_compute.wsgi.server [-] 128.250.116.173,172.26.9.144 "GET /v1.1/0bdf024c921848c4b74d9e69af9edf08/servers/detail HTTP/1.1" status: 401 len: 282 time: 3.5197592


** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1362766

Title:
  ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool'
  object has no attribute 'insecure'

Status in OpenStack Compute (Nova):
  New
Status in Python client library for Glance:
  Fix Released
Status in Python client library for Neutron:
  Incomplete

Bug description:
  While compute manager was trying to authenticate with neutronclient,
  we see the following:

  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager Traceback (most recent call last):
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager   File "/usr/lib/python2.7/site-packages/powervc_nova/compute/manager.py", line 672, in _populate_admin_context
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager     nclient.authenticate()
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 231, in authenticate
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager     self._authenticate_keystone()
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 209, in _authenticate_keystone
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager     allow_redirects=True)
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 113, in _cs_request
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager     raise exceptions.ConnectionFailed(reason=e)
  2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure'

  Setting a pdb breakpoint and stepping into the code, I see that the
  requests library is getting a connection object from a pool.  The
  interesting thing is that the connection object is actually from
  glanceclient.common.https.HTTPSConnectionPool.  It seems odd to me
  that neutronclient is using a connection object from glanceclient
  pool, but I do not know this requests code.  Here is the stack just
  before failure:

    /usr/lib/python2.7/site-packages/neutronclient/client.py(234)authenticate()
  -> self._authenticate_keystone()
    /usr/lib/python2.7/site-packages/neutronclient/client.py(212)_authenticate_keystone()
  -> allow_redirects=True)
    /usr/lib/python2.7/site-packages/neutronclient/client.py(106)_cs_request()
  -> resp, body = self.request(*args, **kargs)
    /usr/lib/python2.7/site-packages/neutronclient/client.py(151)request()
  -> **kwargs)
    /usr/lib/python2.7/site-packages/requests/api.py(44)request()
  -> return session.request(method=method, url=url, **kwargs)
    /usr/lib/python2.7/site-packages/requests/sessions.py(335)request()
  -> resp = self.send(prep, **send_kwargs)
    /usr/lib/python2.7/site-packages/requests/sessions.py(438)send()
  -> r = adapter.send(request, **kwargs)
    /usr/lib/python2.7/site-packages/requests/adapters.py(292)send()
  -> timeout=timeout
    /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(454)urlopen()
  -> conn = self._get_conn(timeout=pool_timeout)
    /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(272)_get_conn()
  -> return conn or self._new_conn()
  > /usr/lib/python2.7/site-packages/glanceclient/common/https.py(100)_new_conn()
  -> return VerifiedHTTPSConnection(host=self.host,

  The code about to run there is this:

  class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool):
      """
      HTTPSConnectionPool will be instantiated when a new
      connection is requested to the HTTPSAdapter.This
      implementation overwrites the _new_conn method and
      returns an instances of glanceclient's VerifiedHTTPSConnection
      which handles no compression.

      ssl_compression is hard-coded to False because this will
      be used just when the user sets --no-ssl-compression.
      """

      scheme = 'https'

      def _new_conn(self):
          self.num_connections += 1
          return VerifiedHTTPSConnection(host=self.host,
                                         port=self.port,
                                         key_file=self.key_file,
                                         cert_file=self.cert_file,
                                         cacert=self.ca_certs,
                                         insecure=self.insecure,
                                         ssl_compression=False)

  Note the self.insecure, which does not exist here.  I see the
  following fairly recent change in the glanceclient.common.https code
  that added this:

  https://github.com/openstack/python-
  glanceclient/commit/dbb242b776908ca50ed8557ebfe7cfcd879366c8#diff-
  524c1a0b226fa8e5fb4df6ff5574e153R104

  I do not understand this whole flow and if self.insecure was supposed
  to have been initialized somewhere else, but it's obviously not.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1362766/+subscriptions