yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1822251] [NEW] grenade job fail due to missing memcache module

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1822251@xxxxxxxxxxxxxxxxxx>
Date: Fri, 29 Mar 2019 07:18:14 -0000
Reply-to: Bug 1822251 <1822251@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

We saw it quite often recently that grenade jobs are failing due to
missing memcache module while keystone is starting.

Error message:

2019-03-29 00:22:39.361 | + lib/keystone:bootstrap_keystone:564      :   /usr/local/bin/keystone-manage bootstrap --bootstrap-username admin --bootstrap-password secretadmin --bootstrap-project-name admin --bootstrap-role-name admin --bootstrap-service-name keystone --bootstrap-region-id RegionOne --bootstrap-admin-url http://10.209.34.128/identity --bootstrap-public-url http://10.209.34.128/identity
2019-03-29 00:22:41.201 | [01;31mCRITICAL keystone [[00;36m-[01;31m] [01;35m[01;31mUnhandled error[00m: ModuleNotFoundError: No module named 'memcache'
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00mTraceback (most recent call last):
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/usr/local/bin/keystone-manage", line 10, in <module>
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    sys.exit(main())
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/manage.py", line 41, in main
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    cli.main(argv=sys.argv, developer_config_file=developer_config)
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 1216, in main
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    CONF.command.cmd_class.main()
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 177, in main
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    klass = cls()
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 66, in __init__
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    self.bootstrapper = bootstrap.Bootstrapper()
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/bootstrap.py", line 31, in __init__
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    backends.load_backends()
2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/server/backends.py", line 41, in load_backends
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    cache.configure_cache()
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/common/cache/core.py", line 124, in configure_cache
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    cache.configure_cache_region(CONF, region)
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/oslo_cache/core.py", line 235, in configure_cache_region
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    '%s.' % conf.cache.config_prefix)
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 591, in configure_from_config
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    "%sreplace_existing_backend" % prefix, False),
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 423, in configure
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    _config_prefix
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/api.py", line 89, in from_config_dict
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    for key in config_dict
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 208, in __init__
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    super(MemcacheArgs, self).__init__(arguments)
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 108, in __init__
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    self._imports()
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 287, in _imports
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    import memcache  # noqa
2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00mModuleNotFoundError: No module named 'memcache'


Logstash query which I used to find similar examples: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'memcache'%5C%22

Looks that it happens at least for 10 runs in last 7 days.

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1822251

Title:
  grenade job fail due to missing memcache module

Status in OpenStack Identity (keystone):
  New

Bug description:
  We saw it quite often recently that grenade jobs are failing due to
  missing memcache module while keystone is starting.

  Error message:

  2019-03-29 00:22:39.361 | + lib/keystone:bootstrap_keystone:564      :   /usr/local/bin/keystone-manage bootstrap --bootstrap-username admin --bootstrap-password secretadmin --bootstrap-project-name admin --bootstrap-role-name admin --bootstrap-service-name keystone --bootstrap-region-id RegionOne --bootstrap-admin-url http://10.209.34.128/identity --bootstrap-public-url http://10.209.34.128/identity
  2019-03-29 00:22:41.201 | [01;31mCRITICAL keystone [[00;36m-[01;31m] [01;35m[01;31mUnhandled error[00m: ModuleNotFoundError: No module named 'memcache'
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00mTraceback (most recent call last):
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/usr/local/bin/keystone-manage", line 10, in <module>
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    sys.exit(main())
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/manage.py", line 41, in main
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    cli.main(argv=sys.argv, developer_config_file=developer_config)
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 1216, in main
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    CONF.command.cmd_class.main()
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 177, in main
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    klass = cls()
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 66, in __init__
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    self.bootstrapper = bootstrap.Bootstrapper()
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/cmd/bootstrap.py", line 31, in __init__
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m    backends.load_backends()
  2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/server/backends.py", line 41, in load_backends
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    cache.configure_cache()
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/opt/stack/old/keystone/keystone/common/cache/core.py", line 124, in configure_cache
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    cache.configure_cache_region(CONF, region)
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/oslo_cache/core.py", line 235, in configure_cache_region
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    '%s.' % conf.cache.config_prefix)
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 591, in configure_from_config
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    "%sreplace_existing_backend" % prefix, False),
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 423, in configure
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    _config_prefix
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/api.py", line 89, in from_config_dict
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    for key in config_dict
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 208, in __init__
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    super(MemcacheArgs, self).__init__(arguments)
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 108, in __init__
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    self._imports()
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 287, in _imports
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m    import memcache  # noqa
  2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00mModuleNotFoundError: No module named 'memcache'

  
  Logstash query which I used to find similar examples: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'memcache'%5C%22

  Looks that it happens at least for 10 runs in last 7 days.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1822251/+subscriptions
Follow ups

[Bug 1822251] Re: grenade job fail due to missing memcache module
From: Matt Riedemann, 2019-04-01
[Bug 1822251] Re: grenade job fail due to missing memcache module
From: Colleen Murphy, 2019-03-29