yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1794812] [NEW] Redis caching backend throws stacktraces

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Christoph Manns <christoph.manns@xxxxxxxxx>
Date: Thu, 27 Sep 2018 15:00:51 -0000
Reply-to: Bug 1794812 <1794812@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If you use redis as a caching backend and you delete a server with
no consoleauth tokens you'll get a stacktrace as an empty list is
passed down to the redis client and ultimately the redis server
which responds with an error, complaining about a wrong number
of arguments for the del command.

2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server [req-cc86f95f-00aa-428a-8920-4f0ea478887b a6f683db07b747188336d9a2ebe1c1c2 0e8a0eaa143c4af3938db81fc9df5abf - default default] Exception during message handling: ResponseError: wrong number of arguments for 'del' command
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/consoleauth/manager.py", line 142, in delete_tokens_for_instance
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     [tok.encode('UTF-8') for tok in tokens])
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/cache_utils.py", line 130, in delete_multi
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     return self.region.delete_multi(keys)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1017, in delete_multi
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     self.backend.delete_multi(keys)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/redis.py", line 183, in delete_multi
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     self.client.delete(*keys)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/redis/client.py", line 824, in delete
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     return self.execute_command('DEL', *names)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/redis/client.py", line 565, in execute_command
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     return self.parse_response(connection, command_name, **options)
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/redis/client.py", line 577, in parse_response
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     response = connection.read_response()
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/redis/connection.py", line 574, in read_response
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     raise response
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server ResponseError: wrong number of arguments for 'del' command
2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server

** Affects: nova
     Importance: Low
     Assignee: Christoph Manns (christoph-manns)
         Status: In Progress


** Tags: consoleauth redis

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1794812

Title:
  Redis caching backend throws stacktraces

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  If you use redis as a caching backend and you delete a server with
  no consoleauth tokens you'll get a stacktrace as an empty list is
  passed down to the redis client and ultimately the redis server
  which responds with an error, complaining about a wrong number
  of arguments for the del command.

  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server [req-cc86f95f-00aa-428a-8920-4f0ea478887b a6f683db07b747188336d9a2ebe1c1c2 0e8a0eaa143c4af3938db81fc9df5abf - default default] Exception during message handling: ResponseError: wrong number of arguments for 'del' command
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/consoleauth/manager.py", line 142, in delete_tokens_for_instance
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     [tok.encode('UTF-8') for tok in tokens])
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/cache_utils.py", line 130, in delete_multi
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     return self.region.delete_multi(keys)
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1017, in delete_multi
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     self.backend.delete_multi(keys)
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/redis.py", line 183, in delete_multi
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     self.client.delete(*keys)
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/redis/client.py", line 824, in delete
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     return self.execute_command('DEL', *names)
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/redis/client.py", line 565, in execute_command
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     return self.parse_response(connection, command_name, **options)
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/redis/client.py", line 577, in parse_response
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     response = connection.read_response()
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/redis/connection.py", line 574, in read_response
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server     raise response
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server ResponseError: wrong number of arguments for 'del' command
  2018-09-27 12:42:57.201 5014 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1794812/+subscriptions
Follow ups

[Bug 1794812] Re: Redis caching backend throws stacktraces
From: OpenStack Infra, 2018-09-30