yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1906500] Re: [ovn] Tempest tests failing while creating security group driver with KeyError: 'remote_address_group_id'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Brian Haley <1906500@xxxxxxxxxxxxxxxxxx>
Date: Tue, 06 Dec 2022 20:40:39 -0000
Reply-to: Bug 1906500 <1906500@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1906500

Title:
  [ovn] Tempest tests failing while creating security group driver with
  KeyError: 'remote_address_group_id'

Status in neutron:
  Fix Released

Bug description:
  This is failing post
  https://review.opendev.org/c/openstack/neutron/+/751110, detected in a
  packstack job, it fails with below Traceback:-

  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 437, in create
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     self.force_reraise()
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     raise value
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     self.force_reraise()
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     raise value
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     self.force_reraise()
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     raise value
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 547, in _create
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     objs = do_create(body, bulk=True)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 543, in do_create
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     self.force_reraise()
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     raise value
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 536, in do_create
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/db/securitygroups_rpc_base.py", line 67, in create_security_group_rule_bulk
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     context, security_group_rules)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     self.force_reraise()
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     raise value
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     self.force_reraise()
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     raise value
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     self.force_reraise()
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     raise value
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/db/securitygroups_db.py", line 387, in create_security_group_rule_bulk_native
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     context, security_group_rules)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/db/securitygroups_db.py", line 666, in _validate_security_group_rules
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     self._validate_security_group_rule(context, rule)
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/db/securitygroups_db.py", line 640, in _validate_security_group_rule
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource     remote_address_group_id = rule['remote_address_group_id']
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource KeyError: 'remote_address_group_id'
  2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource 

  Example logs:-
  Example:- https://logserver.rdoproject.org/ci.centos.org/weirdo-generic-packstack-scenario001/11909/weirdo-project/logs/latest/tempest.log.txt.gz
  https://logserver.rdoproject.org/ci.centos.org/weirdo-generic-packstack-scenario001/11909/weirdo-project/logs/neutron/server.log.txt.gz


  The issue didn't got detected in neutron devstack jobs running with
  ovn as extension security-groups-remote-address-group got loaded via
  'logger' mechanism driver. security-groups-remote-address-group needs
  to be loaded with ovn driver too to get security group creation
  working

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1906500/+subscriptions
References

[Bug 1906500] [NEW] [ovn] Tempest tests failing while creating security group driver with KeyError: 'remote_address_group_id'
From: yatin, 2020-12-02