yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1507723] [NEW] Octavia Barbican cert manager broken

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: German Eichberger <german.eichberger@xxxxxx>
Date: Mon, 19 Oct 2015 18:30:27 -0000
Reply-to: Bug 1507723 <1507723@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Followed instruction on
https://wiki.openstack.org/wiki/Network/LBaaS/docs/how-to-create-tls-
loadbalancer#Create_TLS_enabled_load_balancer: to create a TLS listener.

After setting the undocumented cert_manager option to barbican I get the
following error:

015-10-14 05:18:12.138 24174 DEBUG keystoneclient.session [-] RESP: [300] Conte
nt-Length: 351 Content-Type: application/json; charset=UTF-8 Connection: close
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2015-04-28T
00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vn
d.openstack.key-manager-v1+json"}], "id": "v1", "links": [{"href": "http://172.1
6.90.132:9311/v1/", "rel": "self"}, {"href": "http://docs.openstack.org/";, "type
": "text/html", "rel": "describedby"}]}]}}
 _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/sessio
n.py:216
2015-10-14 05:18:12.139 24174 DEBUG keystoneclient.session [-] REQ: curl -g -i -
X POST http://172.16.90.132:9311/v1/containers/29c5530d-400f-4c15-ae0e-ed16b4dd1
dc3/consumers/ -H "User-Agent: python-keystoneclient" -H "Content-Type: applicat
ion/json" -H "X-Auth-Token: {SHA1}8653790acd55f3ec43967c55b93c524be8b3521a" -d '
{"URL": null, "name": "Octavia"}' _http_log_request /usr/local/lib/python2.7/dis
t-packages/keystoneclient/session.py:198
2015-10-14 05:18:12.243 24174 DEBUG keystoneclient.session [-] RESP: [400] Conne
ction: close Content-Type: application/json; charset=UTF-8 Content-Length: 159 x
-openstack-request-id: req-528d0877-f2c9-4853-8040-aad3e6fa62ca
RESP BODY: {"code": 400, "description": "Provided object does not match schema '
Consumer': None is not of type 'string'. Invalid property: 'URL'", "title": "Bad
 Request"}
 _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/sessio
n.py:216
2015-10-14 05:18:12.244 24174 DEBUG barbicanclient.client [-] Response status 40
0 _check_status_code /opt/stack/python-barbicanclient/barbicanclient/client.py:8
9
2015-10-14 05:18:12.244 24174 ERROR barbicanclient.client [-] 4xx Client error:
Bad Request
2015-10-14 05:18:12.244 24174 ERROR octavia.certificates.manager.barbican [-] Er
ror getting http://172.16.90.132:9311/v1/containers/29c5530d-400f-4c15-ae0e-ed16
b4dd1dc3: Bad Request
2015-10-14 05:18:12.248 24174 WARNING octavia.controller.worker.controller_worke
r [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenerUpdate'
 (f36941f6-a553-48b8-b30b-6e375619ca9a) transitioned into state 'FAILURE' from s
tate 'RUNNING'
1 predecessors (most recent atoms first):
  octavia-create-listener_flow
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker
Traceback (most recent call last):
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker
  File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/ex
ecutor.py", line 82, in _execute_task
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     result = task.execute(**arguments)
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 53, in execute
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     self.amphora_driver.update(listener, vip)
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 64, in update
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     certs = self._process_tls_certificates(listener)
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 142, in _process_tls_certificates
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     self.cert_manager.get_cert(listener.tls_certificate_id))
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/certificates/manager/barbican.py", line 150, in get_cert
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     ).format(cert_ref, str(e)))
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     six.reraise(self.type_, self.value, self.tb)
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/certificates/manager/barbican.py", line 143, in get_cert
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     url=resource_ref
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/python-barbicanclient/barbicanclient/containers.py", line 752, in register_consumer
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     response = self._api.post(href, json=consumer_dict)
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/python-barbicanclient/barbicanclient/client.py", line 77, in post
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     return super(_HTTPClient, self).post(path, *args, **kwargs).json()
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 176, in post
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     return self.request(url, 'POST', **kwargs)
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/python-barbicanclient/barbicanclient/client.py", line 65, in request
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     self._check_status_code(resp)
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/python-barbicanclient/barbicanclient/client.py", line 109, in _check_status_code
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     status
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker HTTPClientError: Bad Request
2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker
2015-10-14 05:18:12.252 24174 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenerUpdate' (f36941f6-a553-48b8-b30b-6e375619ca9a) transitioned into state 'REVERTING' from state 'FAILURE' _task_receiver /usr/local/lib/python2.7/dist-packages/taskflow/listeners/logging.py:189
2015-10-14 05:18:12.252 24174 WARNING octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting listener update.
2015-10-14 05:18:12.260 24174 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenerUpdate' (f36941f6-a553-48b8-b30b-6e375619ca9a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
2015-10-14 05:18:12.260 24174 WARNING octavia.controller.worker.controller_worker [-] Flow 'octavia-create-listener_flow' (d564710c-0fe8-47b4-b29e-f4857197a0aa) transitioned into state 'REVERTED' from state 'RUNNING'
2015-10-14 05:18:12.262 24174 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: Bad Request

It seems that there is a problem in the Barbican client we are using.

** Affects: neutron
     Importance: Critical
         Status: Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1507723

Title:
  Octavia Barbican cert manager broken

Status in neutron:
  Confirmed

Bug description:
  Followed instruction on
  https://wiki.openstack.org/wiki/Network/LBaaS/docs/how-to-create-tls-
  loadbalancer#Create_TLS_enabled_load_balancer: to create a TLS
  listener.

  After setting the undocumented cert_manager option to barbican I get
  the following error:

  015-10-14 05:18:12.138 24174 DEBUG keystoneclient.session [-] RESP: [300] Conte
  nt-Length: 351 Content-Type: application/json; charset=UTF-8 Connection: close
  RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2015-04-28T
  00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vn
  d.openstack.key-manager-v1+json"}], "id": "v1", "links": [{"href": "http://172.1
  6.90.132:9311/v1/", "rel": "self"}, {"href": "http://docs.openstack.org/";, "type
  ": "text/html", "rel": "describedby"}]}]}}
   _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/sessio
  n.py:216
  2015-10-14 05:18:12.139 24174 DEBUG keystoneclient.session [-] REQ: curl -g -i -
  X POST http://172.16.90.132:9311/v1/containers/29c5530d-400f-4c15-ae0e-ed16b4dd1
  dc3/consumers/ -H "User-Agent: python-keystoneclient" -H "Content-Type: applicat
  ion/json" -H "X-Auth-Token: {SHA1}8653790acd55f3ec43967c55b93c524be8b3521a" -d '
  {"URL": null, "name": "Octavia"}' _http_log_request /usr/local/lib/python2.7/dis
  t-packages/keystoneclient/session.py:198
  2015-10-14 05:18:12.243 24174 DEBUG keystoneclient.session [-] RESP: [400] Conne
  ction: close Content-Type: application/json; charset=UTF-8 Content-Length: 159 x
  -openstack-request-id: req-528d0877-f2c9-4853-8040-aad3e6fa62ca
  RESP BODY: {"code": 400, "description": "Provided object does not match schema '
  Consumer': None is not of type 'string'. Invalid property: 'URL'", "title": "Bad
   Request"}
   _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/sessio
  n.py:216
  2015-10-14 05:18:12.244 24174 DEBUG barbicanclient.client [-] Response status 40
  0 _check_status_code /opt/stack/python-barbicanclient/barbicanclient/client.py:8
  9
  2015-10-14 05:18:12.244 24174 ERROR barbicanclient.client [-] 4xx Client error:
  Bad Request
  2015-10-14 05:18:12.244 24174 ERROR octavia.certificates.manager.barbican [-] Er
  ror getting http://172.16.90.132:9311/v1/containers/29c5530d-400f-4c15-ae0e-ed16
  b4dd1dc3: Bad Request
  2015-10-14 05:18:12.248 24174 WARNING octavia.controller.worker.controller_worke
  r [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenerUpdate'
   (f36941f6-a553-48b8-b30b-6e375619ca9a) transitioned into state 'FAILURE' from s
  tate 'RUNNING'
  1 predecessors (most recent atoms first):
    octavia-create-listener_flow
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker
  Traceback (most recent call last):
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker
    File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/ex
  ecutor.py", line 82, in _execute_task
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     result = task.execute(**arguments)
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 53, in execute
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     self.amphora_driver.update(listener, vip)
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 64, in update
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     certs = self._process_tls_certificates(listener)
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 142, in _process_tls_certificates
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     self.cert_manager.get_cert(listener.tls_certificate_id))
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/certificates/manager/barbican.py", line 150, in get_cert
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     ).format(cert_ref, str(e)))
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     six.reraise(self.type_, self.value, self.tb)
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/octavia/octavia/certificates/manager/barbican.py", line 143, in get_cert
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     url=resource_ref
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/python-barbicanclient/barbicanclient/containers.py", line 752, in register_consumer
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     response = self._api.post(href, json=consumer_dict)
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/python-barbicanclient/barbicanclient/client.py", line 77, in post
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     return super(_HTTPClient, self).post(path, *args, **kwargs).json()
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 176, in post
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     return self.request(url, 'POST', **kwargs)
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/python-barbicanclient/barbicanclient/client.py", line 65, in request
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     self._check_status_code(resp)
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker   File "/opt/stack/python-barbicanclient/barbicanclient/client.py", line 109, in _check_status_code
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker     status
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker HTTPClientError: Bad Request
  2015-10-14 05:18:12.248 24174 ERROR octavia.controller.worker.controller_worker
  2015-10-14 05:18:12.252 24174 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenerUpdate' (f36941f6-a553-48b8-b30b-6e375619ca9a) transitioned into state 'REVERTING' from state 'FAILURE' _task_receiver /usr/local/lib/python2.7/dist-packages/taskflow/listeners/logging.py:189
  2015-10-14 05:18:12.252 24174 WARNING octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting listener update.
  2015-10-14 05:18:12.260 24174 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenerUpdate' (f36941f6-a553-48b8-b30b-6e375619ca9a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
  2015-10-14 05:18:12.260 24174 WARNING octavia.controller.worker.controller_worker [-] Flow 'octavia-create-listener_flow' (d564710c-0fe8-47b4-b29e-f4857197a0aa) transitioned into state 'REVERTED' from state 'RUNNING'
  2015-10-14 05:18:12.262 24174 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: Bad Request

  It seems that there is a problem in the Barbican client we are using.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1507723/+subscriptions
Follow ups

[Bug 1507723] Re: Octavia Barbican cert manager broken
From: German Eichberger, 2015-10-20