yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #57094
[Bug 1627696] Re: keystonemiddleware throws OSError
As per discussion in IRC channel with Adam Young, this could be caused
due to SeLinux permission problems (though the audit logs do not have
anything on this). I will try with permissive and see if this can be
reproduced.
** Project changed: keystone => keystonemiddleware
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1627696
Title:
keystonemiddleware throws OSError
Status in keystonemiddleware:
New
Bug description:
I see glance and neutron REST calls failing in some cases and the log
shows this. I have the services running on RHEL 7.2 . The problem
disappears temporarily once the services are restarted.
/tmp directory looks like this:
drwx------. 2 nova nova 64 Sep 26 06:49 keystone-signing-IMigWf
drwx------. 2 cinder cinder 64 Sep 26 06:03 keystone-signing-JVBe80
drwx------. 2 ceilometer ceilometer 64 Sep 26 04:44 keystone-signing-Lb30NH
drwx------. 2 cinder cinder 64 Sep 26 06:21 keystone-signing-pOJym0
drwx------. 2 cinder cinder 64 Sep 26 06:43 keystone-signing-QH89cP
drwx------. 2 nova nova 64 Sep 26 06:49 keystone-signing-_qlc31
drwx------. 2 cinder cinder 64 Sep 26 06:46 keystone-signing-qvCp__
drwx------. 2 nova nova 64 Sep 26 06:49 keystone-signing-RS0Shu
drwx------. 2 swift swift 64 Sep 26 06:49 keystone-signing-swift
drwx------. 2 nova nova 64 Sep 26 06:49 keystone-signing-tioqkc
drwx------. 2 cinder cinder 64 Sep 26 06:35 keystone-signing-TJj0U2
drwx------. 2 cinder cinder 64 Sep 26 06:47 keystone-signing-U56FSw
drwx------. 2 cinder cinder 64 Sep 26 06:19 keystone-signing-_uy4Dy
drwx------. 2 cinder cinder 64 Sep 26 06:49 keystone-signing-YqsCSA
Stacktrace from glance
2016-09-20 05:38:47.310 35161 INFO eventlet.wsgi.server [req-b503be53-cbe9-4ae6-a5b3-6d8f264c4702 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 088c8d5388c4437f93b40f195bfaed05 - default default] 127.0.0.1 - - [20/Sep/2016 05:38:47] "GET /v2/schemas/image HTTP/1.1" 200 4352 0.002869
2016-09-20 05:38:47.706 35160 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 470, in handle_one_response
result = self.application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 126, in __call__
response = req.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 126, in __call__
response = req.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 126, in __call__
response = req.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 804, in __call__
response = req.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 108, in __call__
return request.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 320, in __call__
response = self.process_request(req)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 552, in process_request
resp = super(AuthProtocol, self).process_request(request)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 348, in process_request
data, user_auth_ref = self._do_fetch_token(request.user_token)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 388, in _do_fetch_token
data = self.fetch_token(token)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 670, in fetch_token
self._revocations.check(token_hashes)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 104, in check
if self._any_revoked(token_ids):
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 99, in _any_revoked
if self._is_revoked(token_id):
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 90, in _is_revoked
revoked_tokens = self._list.get('revoked', None)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 74, in _list
self._list = self._fetch()
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 61, in _fetch
return self._cms_verify(revocation_list_data)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 771, in _cms_verify
self._fetch_signing_cert()
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 786, in _fetch_signing_cert
self._identity_server.fetch_signing_cert())
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_signing_dir.py", line 54, in write_file
self._verify_signing_dir()
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_signing_dir.py", line 90, in _verify_signing_dir
os.makedirs(self._directory_name, stat.S_IRWXU)
File "/usr/lib64/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 2] No such file or directory: '/tmp/keystone-signing-9zB_p6'
Stacktrace from neutron:
X-Auth-Token: gAAAAABX6NfMz4Lj4sYIDHu0eXr9oxymDrJTDOOrKztp0NElSiZcs9Umr-v8P-s8VP_lz_aVKPobfoj1ROP9X9amp8ACqwa4FNRvFX5IatzwmjAKR42AZZnuD4jxoJoC05iT-UKIY81gqHsOY8v7DbqTLSE2eOFwrFKZIMQBUDlDaeqwpce0LDp-dZrM2JIta9tOz99aOH5CShyu-ihMy3F87CN3cMdK5qHIr7oM1UiXc97zgzbDOTA
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/oslo_middleware/catch_errors.py", line 38, in __call__
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors response = req.get_response(self.application)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **self.kwargs)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 320, in __call__
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors response = self.process_request(req)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 552, in process_request
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors resp = super(AuthProtocol, self).process_request(request)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 348, in process_request
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors data, user_auth_ref = self._do_fetch_token(request.user_token)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 388, in _do_fetch_token
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors data = self.fetch_token(token)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 670, in fetch_token
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors self._revocations.check(token_hashes)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 104, in check
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors if self._any_revoked(token_ids):
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 99, in _any_revoked
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors if self._is_revoked(token_id):
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 90, in _is_revoked
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors revoked_tokens = self._list.get('revoked', None)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 74, in _list
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors self._list = self._fetch()
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_revocations.py", line 61, in _fetch
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors return self._cms_verify(revocation_list_data)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 771, in _cms_verify
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors self._fetch_signing_cert()
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 786, in _fetch_signing_cert
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors self._identity_server.fetch_signing_cert())
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_signing_dir.py", line 54, in write_file
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors self._verify_signing_dir()
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_signing_dir.py", line 90, in _verify_signing_dir
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors os.makedirs(self._directory_name, stat.S_IRWXU)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib64/python2.7/os.py", line 157, in makedirs
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors mkdir(name, mode)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors OSError: [Errno 2] No such file or directory: '/tmp/keystone-signing-EVEBkJ'
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystonemiddleware/+bug/1627696/+subscriptions
References