yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #49684
[Bug 1572062] [NEW] nova-consoleauth doesn't play well with memcached
Public bug reported:
When running with the Ubuntu Xenial Mitaka packages, I'm seeing the
following behaviour:
Following the release notes at
http://docs.openstack.org/releasenotes/nova/mitaka.html I remove the old
option "memcached_servers" and set up a cache section with
[cache]
enabled = true
memcache_servers = host1:11211,host2:11211,host3:11211
The result is that there are errors logged when creating a token:
2016-04-19 10:17:33.501 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Token: dd0e7bbc-c593-421f-a56b-52578aaec20e failed to save into memcached.
2016-04-19 10:17:33.503 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Instance: d3267abb-258d-4b5d-a22e-8e3b0a39905c failed to save into memcached
Only after a lot of debugging it turns out that the default backend for
oslo_cache is dogpile.cache.null, implying that no values get cached and
token validation always fails. Only after adding
[cache]
backend = oslo_cache.memcache_pool
does console authentication start working. Strangely though, the above
warning messages are still being logged in the working setup, which made
debugging this even more difficult.
So I suggest the following fixes:
1. Change the text of the warnings from "failed to save into memcached" to "failed to save into cache", as with the change to using oslo_cache, there may be other backends in use instead of memcached.
2. Either override the default of using the null backend or refuse to run with it or at the very least give a big fat warning that the configuration can not work.
3. Stop generating the warning messages when the data got in fact saved into cache properly.
Package versions for reference:
# dpkg -l | grep nova
ii nova-api-metadata 2:13.0.0-0ubuntu2 all OpenStack Compute - metadata API frontend
ii nova-api-os-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - OpenStack Compute API frontend
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii nova-spiceproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - spice html5 proxy
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
# dpkg -l | grep oslo
ii python-oslo.cache 1.6.0-2 all cache storage for Openstack projects - Python 2.7
ii python-oslo.concurrency 3.7.0-2 all concurrency and locks for OpenStack projects - Python 2.x
ii python-oslo.config 1:3.9.0-3 all Common code for Openstack Projects (configuration API) - Python 2.x
ii python-oslo.context 2.2.0-2 all WSGI context helpers for OpenStack - Python 2.x
ii python-oslo.db 4.7.0-2ubuntu1 all database connectivity to the different backends and helper utils - Python 2.x
ii python-oslo.i18n 3.5.0-2 all Oslo Internationalization Utilities - Python 2.x
ii python-oslo.log 3.2.0-2 all OpenStack logging configuration library - Python 2.x
ii python-oslo.messaging 4.6.1-2ubuntu1 all oslo messaging library - Python 2.x
ii python-oslo.middleware 3.8.0-2 all various WSGI middleware components for OpenStack - Python 2.x
ii python-oslo.policy 1.6.0-2 all RBAC policy enforcement library for OpenStack - Python 2.x
ii python-oslo.reports 1.7.0-2 all reports serialized in various data types - Python 2.7
ii python-oslo.rootwrap 4.1.0-2 all allows fine filtering of shell commands to run as root - Python 2.x
ii python-oslo.serialization 2.4.0-2 all utilities for serialization , especially JSON - Python 2.x
ii python-oslo.service 1.8.0-1ubuntu1 all library for running OpenStack services - Python 2.x
ii python-oslo.utils 3.8.0-2 all set of utility functions for OpenStack - Python 2.x
ii python-oslo.versionedobjects 1.8.0-1 all deals with DB schema versions and code expectations - Python 2.x
ii python-oslo.vmware 2.5.0-2 all VMware library for OpenStack projects - Python 2.7
** Affects: nova
Importance: Undecided
Assignee: Dr. Jens Rosenboom (j-rosenboom-j)
Status: In Progress
** Changed in: nova
Assignee: (unassigned) => Dr. Jens Rosenboom (j-rosenboom-j)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1572062
Title:
nova-consoleauth doesn't play well with memcached
Status in OpenStack Compute (nova):
In Progress
Bug description:
When running with the Ubuntu Xenial Mitaka packages, I'm seeing the
following behaviour:
Following the release notes at
http://docs.openstack.org/releasenotes/nova/mitaka.html I remove the
old option "memcached_servers" and set up a cache section with
[cache]
enabled = true
memcache_servers = host1:11211,host2:11211,host3:11211
The result is that there are errors logged when creating a token:
2016-04-19 10:17:33.501 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Token: dd0e7bbc-c593-421f-a56b-52578aaec20e failed to save into memcached.
2016-04-19 10:17:33.503 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Instance: d3267abb-258d-4b5d-a22e-8e3b0a39905c failed to save into memcached
Only after a lot of debugging it turns out that the default backend
for oslo_cache is dogpile.cache.null, implying that no values get
cached and token validation always fails. Only after adding
[cache]
backend = oslo_cache.memcache_pool
does console authentication start working. Strangely though, the above
warning messages are still being logged in the working setup, which
made debugging this even more difficult.
So I suggest the following fixes:
1. Change the text of the warnings from "failed to save into memcached" to "failed to save into cache", as with the change to using oslo_cache, there may be other backends in use instead of memcached.
2. Either override the default of using the null backend or refuse to run with it or at the very least give a big fat warning that the configuration can not work.
3. Stop generating the warning messages when the data got in fact saved into cache properly.
Package versions for reference:
# dpkg -l | grep nova
ii nova-api-metadata 2:13.0.0-0ubuntu2 all OpenStack Compute - metadata API frontend
ii nova-api-os-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - OpenStack Compute API frontend
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii nova-spiceproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - spice html5 proxy
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
# dpkg -l | grep oslo
ii python-oslo.cache 1.6.0-2 all cache storage for Openstack projects - Python 2.7
ii python-oslo.concurrency 3.7.0-2 all concurrency and locks for OpenStack projects - Python 2.x
ii python-oslo.config 1:3.9.0-3 all Common code for Openstack Projects (configuration API) - Python 2.x
ii python-oslo.context 2.2.0-2 all WSGI context helpers for OpenStack - Python 2.x
ii python-oslo.db 4.7.0-2ubuntu1 all database connectivity to the different backends and helper utils - Python 2.x
ii python-oslo.i18n 3.5.0-2 all Oslo Internationalization Utilities - Python 2.x
ii python-oslo.log 3.2.0-2 all OpenStack logging configuration library - Python 2.x
ii python-oslo.messaging 4.6.1-2ubuntu1 all oslo messaging library - Python 2.x
ii python-oslo.middleware 3.8.0-2 all various WSGI middleware components for OpenStack - Python 2.x
ii python-oslo.policy 1.6.0-2 all RBAC policy enforcement library for OpenStack - Python 2.x
ii python-oslo.reports 1.7.0-2 all reports serialized in various data types - Python 2.7
ii python-oslo.rootwrap 4.1.0-2 all allows fine filtering of shell commands to run as root - Python 2.x
ii python-oslo.serialization 2.4.0-2 all utilities for serialization , especially JSON - Python 2.x
ii python-oslo.service 1.8.0-1ubuntu1 all library for running OpenStack services - Python 2.x
ii python-oslo.utils 3.8.0-2 all set of utility functions for OpenStack - Python 2.x
ii python-oslo.versionedobjects 1.8.0-1 all deals with DB schema versions and code expectations - Python 2.x
ii python-oslo.vmware 2.5.0-2 all VMware library for OpenStack projects - Python 2.7
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1572062/+subscriptions
Follow ups
