yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1856119] [NEW] FileNotFoundError: [Errno 2] No such file or directory: '/etc/nova/policy.json' when policy file removed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: melanie witt <1856119@xxxxxxxxxxxxxxxxxx>
Date: Thu, 12 Dec 2019 03:51:08 -0000
Reply-to: Bug 1856119 <1856119@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Reported in the #openstack-nova channel today, that if a policy.json
file is added without restarting the nova-api service, things work fine,
but if the file is removed without restarting the service, the following
error is raised:

2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack [req-46b41bde-4689-438f-805b-23b0a500066e 1ea06b07c73149ca9c6753e07c30383a 2962d44b73db4e1d884498b8ce000a69 - 5080f063d9f84290a8233e16a0ff39a2 5080f063d9f84290a8233e16a0ff39a2] Caught error: [Errno 2] No such file or directory: '/etc/nova/policy.json': FileNotFoundError: [Errno 2] No such file or directory: '/etc/nova/policy.json'
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack Traceback (most recent call last):
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/nova/api/openstack/__init__.py", line 95, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return req.get_response(self.application)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/request.py", line 1314, in send
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     application, catch_exc_info=False)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return resp(environ, start_response)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     response = req.get_response(self._app)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/request.py", line 1314, in send
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     application, catch_exc_info=False)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/nova/api/auth.py", line 99, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     service_catalog=service_catalog)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return wrapped(*args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return wrapped(*args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return wrapped(*args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   [Previous line repeated 2 more times]
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_context/context.py", line 466, in from_environ
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return cls(**kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/nova/context.py", line 141, in __init__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     self.is_admin = policy.check_is_admin(self)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/nova/policy.py", line 192, in check_is_admin
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return _ENFORCER.authorize('context_is_admin', target, credentials)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 1043, in authorize
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     *args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 855, in enforce
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     self.load_rules()
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 568, in load_rules
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     overwrite=self.overwrite)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 797, in _load_policy_file
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     self._file_cache, path, force_reload=force_reload)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/_cache_handler.py", line 36, in read_cached_file
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     mtime = os.path.getmtime(filename)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3.6/genericpath.py", line 55, in getmtime
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return os.stat(filename).st_mtime
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack FileNotFoundError: [Errno 2] No such file or directory: '/etc/nova/policy.json'

Happening due to some file caching done by oslo.policy. Looks like we
need to handle that somehow in nova.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: low-hanging-fruit

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1856119

Title:
  FileNotFoundError: [Errno 2] No such file or directory:
  '/etc/nova/policy.json' when policy file removed

Status in OpenStack Compute (nova):
  New

Bug description:
  Reported in the #openstack-nova channel today, that if a policy.json
  file is added without restarting the nova-api service, things work
  fine, but if the file is removed without restarting the service, the
  following error is raised:

  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack [req-46b41bde-4689-438f-805b-23b0a500066e 1ea06b07c73149ca9c6753e07c30383a 2962d44b73db4e1d884498b8ce000a69 - 5080f063d9f84290a8233e16a0ff39a2 5080f063d9f84290a8233e16a0ff39a2] Caught error: [Errno 2] No such file or directory: '/etc/nova/policy.json': FileNotFoundError: [Errno 2] No such file or directory: '/etc/nova/policy.json'
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack Traceback (most recent call last):
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/nova/api/openstack/__init__.py", line 95, in __call__
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return req.get_response(self.application)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/request.py", line 1314, in send
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     application, catch_exc_info=False)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return resp(environ, start_response)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     response = req.get_response(self._app)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/request.py", line 1314, in send
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     application, catch_exc_info=False)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     app_iter = application(self.environ, start_response)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     resp = self.call_func(req, *args, **kw)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return self.func(req, *args, **kwargs)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/nova/api/auth.py", line 99, in __call__
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     service_catalog=service_catalog)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return wrapped(*args, **kwargs)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return wrapped(*args, **kwargs)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return wrapped(*args, **kwargs)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   [Previous line repeated 2 more times]
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_context/context.py", line 466, in from_environ
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return cls(**kwargs)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/nova/context.py", line 141, in __init__
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     self.is_admin = policy.check_is_admin(self)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/nova/policy.py", line 192, in check_is_admin
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return _ENFORCER.authorize('context_is_admin', target, credentials)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 1043, in authorize
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     *args, **kwargs)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 855, in enforce
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     self.load_rules()
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 568, in load_rules
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     overwrite=self.overwrite)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 797, in _load_policy_file
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     self._file_cache, path, force_reload=force_reload)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3/dist-packages/oslo_policy/_cache_handler.py", line 36, in read_cached_file
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     mtime = os.path.getmtime(filename)
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack   File "/usr/lib/python3.6/genericpath.py", line 55, in getmtime
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack     return os.stat(filename).st_mtime
  2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack FileNotFoundError: [Errno 2] No such file or directory: '/etc/nova/policy.json'

  Happening due to some file caching done by oslo.policy. Looks like we
  need to handle that somehow in nova.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1856119/+subscriptions