yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2061350] [NEW] [ovn] The 'ovn_client' of the L3 plugin is occasionally NoneType

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Liu Xie <2061350@xxxxxxxxxxxxxxxxxx>
Date: Mon, 15 Apr 2024 03:52:03 -0000
Reply-to: Bug 2061350 <2061350@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

I don't know how to reproduce it, and we have encountered it twice. The
errors are as follows:

2024-04-01 10:53:46.185 252 INFO neutron.wsgi [req-3ed42a43-8895-4ad3-b675-af89c6d47631 9862b8ef431c43489ada1bb12f97be59 c2c9335342874b758aa6af2166fca837 - default default] 192.168.10.5 "GET /v2.0/ports?device_id=ae622d1a-416f-4676-b340-be0af8e0626a HTTP/1.1" status: 200  len: 7306 time: 0.0576138
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource [req-e0dcd168-ad4e-43d8-b88c-47393077aa64 b255e17db7c5499c8c8e45e363ae3b47 12edc48812c94bc3a861bc8bafcf8fa6 - default default] index failed: No details.: AttributeError: 'NoneType' object has no attribute 'get_lrouter'
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource Traceback (most recent call last):
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/base.py", line 369, in index
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return self._items(request, True, parent_id)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/base.py", line 304, in _items
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     obj_list = obj_getter(request.context, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 217, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/l3_db.py", line 601, in get_routers
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     page_reverse=page_reverse)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 319, in get_collection
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     for c in query
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 319, in <listcomp>
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     for c in query
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/l3_db.py", line 231, in _make_router_dict
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     resource_extend.apply_funcs(l3_apidef.ROUTERS, res, router)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/timeutils.py", line 383, in wrapper
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     result = func(*args, **kwargs)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/resource_extend.py", line 91, in apply_funcs
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     resolved_func(response, db_object)
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/availability_zone/router.py", line 40, in _add_az_to_response
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     l3_plugin.get_router_availability_zones(router_db))
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/services/ovn_l3/plugin.py", line 516, in get_router_availability_zones
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     lr = self._ovn.get_lrouter(router['id'])
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource AttributeError: 'NoneType' object has no attribute 'get_lrouter'
2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource

I suspect that at the completion of ovn_l3 plugin instantiation, the ovn
client has not yet been fully instantiated.

neutron version: wallaby.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: ovn

** Description changed:

  I don't know how to reproduce it, and we have encountered it twice. The
  errors are as follows:
  
  2024-04-01 10:53:46.185 252 INFO neutron.wsgi [req-3ed42a43-8895-4ad3-b675-af89c6d47631 9862b8ef431c43489ada1bb12f97be59 c2c9335342874b758aa6af2166fca837 - default default] 192.168.10.5 "GET /v2.0/ports?device_id=ae622d1a-416f-4676-b340-be0af8e0626a HTTP/1.1" status: 200  len: 7306 time: 0.0576138
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource [req-e0dcd168-ad4e-43d8-b88c-47393077aa64 b255e17db7c5499c8c8e45e363ae3b47 12edc48812c94bc3a861bc8bafcf8fa6 - default default] index failed: No details.: AttributeError: 'NoneType' object has no attribute 'get_lrouter'
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/base.py", line 369, in index
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return self._items(request, True, parent_id)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/base.py", line 304, in _items
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     obj_list = obj_getter(request.context, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 217, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/l3_db.py", line 601, in get_routers
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     page_reverse=page_reverse)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 319, in get_collection
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     for c in query
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 319, in <listcomp>
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     for c in query
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/l3_db.py", line 231, in _make_router_dict
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     resource_extend.apply_funcs(l3_apidef.ROUTERS, res, router)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/timeutils.py", line 383, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     result = func(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/resource_extend.py", line 91, in apply_funcs
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     resolved_func(response, db_object)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/availability_zone/router.py", line 40, in _add_az_to_response
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     l3_plugin.get_router_availability_zones(router_db))
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/services/ovn_l3/plugin.py", line 516, in get_router_availability_zones
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     lr = self._ovn.get_lrouter(router['id'])
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource AttributeError: 'NoneType' object has no attribute 'get_lrouter'
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource
  
- 
- I suspect that at the completion of ovn_l3 plugin instantiation, the client has not yet been fully instantiated.
- 
+ I suspect that at the completion of ovn_l3 plugin instantiation, the ovn
+ client has not yet been fully instantiated.
  
  neutron version: wallaby.

** Tags added: ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2061350

Title:
  [ovn]  The 'ovn_client' of the L3 plugin is occasionally NoneType

Status in neutron:
  New

Bug description:
  I don't know how to reproduce it, and we have encountered it twice.
  The errors are as follows:

  2024-04-01 10:53:46.185 252 INFO neutron.wsgi [req-3ed42a43-8895-4ad3-b675-af89c6d47631 9862b8ef431c43489ada1bb12f97be59 c2c9335342874b758aa6af2166fca837 - default default] 192.168.10.5 "GET /v2.0/ports?device_id=ae622d1a-416f-4676-b340-be0af8e0626a HTTP/1.1" status: 200  len: 7306 time: 0.0576138
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource [req-e0dcd168-ad4e-43d8-b88c-47393077aa64 b255e17db7c5499c8c8e45e363ae3b47 12edc48812c94bc3a861bc8bafcf8fa6 - default default] index failed: No details.: AttributeError: 'NoneType' object has no attribute 'get_lrouter'
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/base.py", line 369, in index
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return self._items(request, True, parent_id)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/api/v2/base.py", line 304, in _items
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     obj_list = obj_getter(request.context, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 217, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     self.force_reraise()
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     raise self.value
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/l3_db.py", line 601, in get_routers
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     page_reverse=page_reverse)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 319, in get_collection
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     for c in query
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 319, in <listcomp>
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     for c in query
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/l3_db.py", line 231, in _make_router_dict
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     resource_extend.apply_funcs(l3_apidef.ROUTERS, res, router)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/oslo_utils/timeutils.py", line 383, in wrapper
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     result = func(*args, **kwargs)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron_lib/db/resource_extend.py", line 91, in apply_funcs
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     resolved_func(response, db_object)
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/db/availability_zone/router.py", line 40, in _add_az_to_response
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     l3_plugin.get_router_availability_zones(router_db))
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/site-packages/neutron/services/ovn_l3/plugin.py", line 516, in get_router_availability_zones
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource     lr = self._ovn.get_lrouter(router['id'])
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource AttributeError: 'NoneType' object has no attribute 'get_lrouter'
  2024-04-01 10:53:46.184 245 ERROR neutron.api.v2.resource

  I suspect that at the completion of ovn_l3 plugin instantiation, the
  ovn client has not yet been fully instantiated.

  neutron version: wallaby.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2061350/+subscriptions
Follow ups

[Bug 2061350] Re: [ovn] The 'ovn_client' of the L3 plugin is occasionally NoneType
From: Launchpad Bug Tracker, 2024-06-17