yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1915494] Re: Neutron run with uwsgi is broken with oslo_policy 3.6.2

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1915494@xxxxxxxxxxxxxxxxxx>
Date: Tue, 23 Feb 2021 09:05:21 -0000
Reply-to: Bug 1915494 <1915494@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: Confirmed => Fix Released

** Tags removed: neutron-proactive-backport-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1915494

Title:
  Neutron run with uwsgi is broken with oslo_policy 3.6.2

Status in neutron:
  Fix Released

Bug description:
  There is error:

  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     result = method(request=request, **args)
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 437, in create
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     raise value
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
  Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     raise value
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     self.force_reraise()
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     raise value
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 460, in _create
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     policy.init()
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/policy.py", line 94, in init
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     register_rules(_ENFORCER)
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/policy.py", line 76, in register_rules
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     enforcer.register_defaults(itertools.chain(*policies))
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py", line 1119, in register_defaults
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     self.register_default(default)
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py", line 1108, in register_default
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     self.registered_rules[default.name] = copy.deepcopy(default)
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y = _reconstruct(x, memo, *rv)
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     state = deepcopy(state, memo)
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y = copier(x, memo)
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y[deepcopy(key, memo)] = deepcopy(value, memo)
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y = _reconstruct(x, memo, *rv)
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
  Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     state = deepcopy(state, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y = copier(x, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y[deepcopy(key, memo)] = deepcopy(value, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y = _reconstruct(x, memo, *rv)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     state = deepcopy(state, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y = copier(x, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y[deepcopy(key, memo)] = deepcopy(value, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y = _reconstruct(x, memo, *rv)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     state = deepcopy(state, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y = copier(x, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     y[deepcopy(key, memo)] = deepcopy(value, memo)
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource   File "/usr/lib/python3.8/copy.py", line 161, in deepcopy
  Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource     rv = reductor(4)
  Feb 12 01:11:30.866920 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource TypeError: cannot pickle '_thread.RLock' object
  Feb 12 01:11:30.866920 ubuntu-focal-rax-iad-0022985630 devstack@neutron-api.service[68621]: ERROR neutron.api.v2.resource 

  while doing API request. E.g.:

  2021-02-12 01:11:28.892252 | controller | ++
  lib/neutron_plugins/services/l3:create_neutron_initial_network:176 :
  oscwrap --os-cloud devstack-admin --os-region RegionOne subnet pool
  create shared-default-subnetpool-v4 --default-prefix-length 26 --pool-
  prefix 10.1.0.0/20 --share --default -f value -c id

  Logs:
  https://zuul.opendev.org/t/openstack/build/5e75395d4ef346de9f813ae678cd6fab/logs

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1915494/+subscriptions
References

[Bug 1915494] [NEW] Neutron run with uwsgi is broken with oslo_policy 3.6.2
From: Slawek Kaplonski, 2021-02-12