yahoo-eng-team team mailing list archive

[Michael Johnson <johnsomor@xxxxxxxxx>]

** Project changed: neutron => octavia

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1611509

Title:
  lbaasv2 doesn't support "https" keystone endpoint

Status in octavia:
  Confirmed

Bug description:
  I am trying to enable lbaasv2 using octavia driver in one of our mitaka deployment. And we got the error
  {code}
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin [req-87d34869-7fec-4269-894b-81a4f1771736 6928cf223a0948699fab55612678cfdc 10d7de26713241a2b623f2028c77e8eb - - -] There was an error in the driver
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin Traceback (most recent call last):
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/services/loadbalancer/plugin.py", line 489, in _call_driver_operation
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     driver_method(context, db_entity)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 118, in func_wrapper
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     args[0].failed_completion(args[1], args[2])
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     self.force_reraise()
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     six.reraise(self.type_, self.value, self.tb)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 108, in func_wrapper
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     r = func(*args, **kwargs)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 220, in create
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     self.driver.req.post(self._url(lb), args)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 150, in post
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.request('POST', url, args)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/octavia/driver.py", line 131, in request
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     token = self.auth_session.get_token()
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 618, in get_token
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     return (self.get_auth_headers(auth) or {}).get('X-Auth-Token')
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 597, in get_auth_headers
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     return auth.get_headers(self, **kwargs)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 84, in get_headers
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     token = self.get_token(session)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 89, in get_token
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.get_access(session).auth_token
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 135, in get_access
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     self.auth_ref = self.get_auth_ref(session)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 166, in get_auth_ref
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     authenticated=False, log=False, **rkwargs)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 545, in post
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     return self.request(url, 'POST', **kwargs)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/_utils.py", line 180, in inner
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     return func(*args, **kwargs)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 425, in request
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     resp = send(**kwargs)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 463, in _send_request
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin     raise exceptions.SSLError(msg)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin SSLError: SSL exception connecting to https://10.240.118.24:35357/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
  neutron-server.log:2016-08-09 20:15:25.462 74450 ERROR neutron_lbaas.services.loadbalancer.plugin

  {code}

  The problem is neutron-lbaas doesn't support ssl keystone endpoint. Looking at the following code in neutron_lbaas/common/keystone.py
  {code}
          try:
              kc = client.Password(**kwargs)
              _SESSION = session.Session(auth=kc)
  {code}
   when it try to create the session it use the default value for cert and verify which is "cert=None, verify=True". This means it only support http keystone endpoint. Since a lot of deployment use https keystone endpoints, we need to fix this problem.

  The step to reproduce this problem should be pretty straight forward just configure keystone https endpoint in your devstack and enable lbaasv2 and octavia then run the following command
  ```
  neutron lbaas-loadbalancer-create --name lb1 test-subnet
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/octavia/+bug/1611509/+subscriptions