yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1839658] Re: "subnet" register in the DB can have network_id=NULL

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1839658@xxxxxxxxxxxxxxxxxx>
Date: Tue, 27 Aug 2019 19:24:25 -0000
Reply-to: Bug 1839658 <1839658@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/677221
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4b6c2246c09eccbb193b4a59a8dfee9e24f97bed
Submitter: Zuul
Branch:    master

commit 4b6c2246c09eccbb193b4a59a8dfee9e24f97bed
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Mon Aug 19 13:09:20 2019 +0000

    Force "network_id" in "subnet" DB registers
    
    The "subnet" OVO does not allow to have an empty (None) "network_id" but
    the DB "subnet" table allows to have this parameter empty (NULL) in the
    database. In order to avoid any problem like the one described in the
    bug, this patch ensures the database "subnet" register does always have
    a "network_id" value and if a "network" register is being deleted, all
    related "subnet" registers are checked first.
    
    Change-Id: Iad210f0585b4201fdb87187b44a9b42267b58db4
    Closes-Bug: #1839658


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1839658

Title:
  "subnet" register in the DB can have network_id=NULL

Status in neutron:
  Fix Released

Bug description:
  (A bit more readable version of the following text is found at
  http://paste.openstack.org/show/756523/)

  When deleting a cluster with several networks and subnets, the
  following error was found when executing the shell command 'openstack
  subnet list':

  019-08-06 08:26:10.000 28 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200  len: 248 time: 0.0012720
  2019-08-06 08:26:10.803 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.15', 41484) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id'
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return self.get(*args, **kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     **query_params)}
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1075, in get_subnets
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     for subnet_object in subnet_objs
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     'network_id': subnet['network_id'],
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return getattr(self, name)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation     return getattr(self, attrname)
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id'
  2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation
  2019-08-06 08:26:11.523 28 INFO neutron.wsgi [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] 172.17.1.15 "GET /v2.0/subnets HTTP/1.1" status: 500  len: 344 time: 0.7188911
  2019-08-06 08:26:12.002 34 DEBUG neutron.wsgi [-] (34) accepted ('172.17.1.15', 41734) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
  2019-08-06 08:26:12.004 34 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200  len: 248 time: 0.0012681

  Which means that some subnets where left behind without an associated network. In this case ns/openshift-kube-controller-manager-subnet:
  MariaDB [ovs_neutron]> select * from subnets;
  +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+
  | project_id                       | id                                   | name                                        | network_id                           | ip_version | cidr           | gateway_ip  | enable_dhcp | ipv6_ra_mode | ipv6_address_mode | subnetpool_id                        | standard_attr_id | segment_id |
  +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+
  | 755fea49ab0643d0b731073e87e739e0 | 26953b3b-fa6c-46a0-9e60-6a7d5cadf580 | lb-mgmt-subnet                              | 4bfa167b-fcf9-4ad8-b5f3-99c6141b20a4 |          4 | 172.24.0.0/16  | 172.24.0.1  |           1 | NULL         | NULL              | NULL                                 |                8 | NULL       |
  | dcb34bf98dea46928294ff6762d9c167 | 5ea30bc8-6eb1-4494-96e6-b37d9a70fb09 | ns/openshift-kube-controller-manager-subnet | NULL                                 |          4 | 10.128.34.0/24 | 10.128.34.1 |           0 | NULL         | NULL              | cc48743c-e44c-48ee-b29c-5ceb53f60fbd |             9493 | NULL       |
  | 755fea49ab0643d0b731073e87e739e0 | 8f4a9baf-ad79-4730-9449-671706f24d7a | external_subnet                             | 004f76a6-3867-46f4-8718-5498634945a3 |          4 | 10.0.0.0/24    | 10.0.0.1    |           0 | NULL         | NULL              | NULL                                 |               27 | NULL       |
  +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+

  This is only happening rarely, not 100% of the time.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1839658/+subscriptions
References

[Bug 1839658] [NEW] "subnet" register in the DB can have network_id=NULL
From: Rodolfo Alonso, 2019-08-09