yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1898860] [NEW] neutron-ovn-rally-task failing 100% of times since 06.10.2020

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1898860@xxxxxxxxxxxxxxxxxx>
Date: Wed, 07 Oct 2020 11:14:10 -0000
Reply-to: Bug 1898860 <1898860@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Example of failure:
https://zuul.opendev.org/t/openstack/build/0708d0d6e0ee4016b7c32aae98fc98c6/logs

In neutron-server logs there is error:

Oct 06 10:46:49.068700 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.api.extensions [None req-
e44a37df-c419-46c2-82ef-a9696baa8ed0 None None] Unable to process
extensions (qos-fip) because the configured plugins do not satisfy their
requirements. Some features will not work as expected.

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service [None req-e44a37df-c419-46c2-82ef-
a9696baa8ed0 None None] Unrecoverable error: please check log for
details.: neutron_lib.exceptions.ExtensionsNotFound: Extensions not
found: ['qos-fip'].

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service Traceback (most recent call last):

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/service.py", line 88, in serve_wsgi

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     service.start()

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/service.py", line 64, in start

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     self.wsgi_app =
_run_wsgi(self.app_name)

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/service.py", line 321, in _run_wsgi

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     app =
config.load_paste_app(app_name)

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/common/config.py", line 133, in
load_paste_app

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     app =
loader.load_app(app_name)

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/oslo_service/wsgi.py", line 351, in load_app

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return
deploy.loadapp("config:%s" % self.config_path, name=name)

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 253, in loadapp

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return loadobj(APP, uri,
name=name, **kw)

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 278, in loadobj

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return context.create()

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 715, in create

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return
self.object_type.invoke(self)

Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 148, in invoke

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return
fix_call(context.object,

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/util.py", line 55, in fix_call

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     val = callable(*args, **kw)

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/urlmap.py", line 31, in urlmap_factory

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     app = loader.get_app(app_name,
global_conf=global_conf)

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 355, in get_app

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return self.app_context(

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 715, in create

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return
self.object_type.invoke(self)

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 148, in invoke

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return
fix_call(context.object,

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/util.py", line 55, in fix_call

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     val = callable(*args, **kw)

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/auth.py", line 47, in pipeline_factory

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     app =
loader.get_app(pipeline[-1])

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 355, in get_app

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return self.app_context(

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 715, in create

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return
self.object_type.invoke(self)

Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/loadwsgi.py", line 152, in invoke

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return
fix_call(context.object, context.global_conf, **context.local_conf)

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
/dist-packages/paste/deploy/util.py", line 55, in fix_call

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     val = callable(*args, **kw)

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/api/v2/router.py", line 25, in _factory

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     return
pecan_app.v2_factory(global_config, **local_config)

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/pecan_wsgi/app.py", line 47, in v2_factory

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     startup.initialize_all()

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/pecan_wsgi/startup.py", line 41, in
initialize_all

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service
ext_mgr.extend_resources("2.0", attributes.RESOURCES)

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/api/extensions.py", line 376, in
extend_resources

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service
self._check_faulty_extensions(unloadable_extensions)

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service   File
"/opt/stack/neutron/neutron/api/extensions.py", line 395, in
_check_faulty_extensions

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service     raise
exceptions.ExtensionsNotFound(

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service
neutron_lib.exceptions.ExtensionsNotFound: Extensions not found: ['qos-
fip'].

Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron.service

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: CRITICAL neutron [None req-e44a37df-c419-46c2-82ef-
a9696baa8ed0 None None] Unhandled error:
neutron_lib.exceptions.ExtensionsNotFound: Extensions not found: ['qos-
fip'].

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron Traceback (most recent call last):

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/bin/neutron-server",
line 10, in <module>

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     sys.exit(main())

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/cmd/eventlet/server/__init__.py", line 19,
in main

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron
server.boot_server(wsgi_eventlet.eventlet_wsgi_server)

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/server/__init__.py", line 68, in boot_server

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     server_func()

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/server/wsgi_eventlet.py", line 24, in
eventlet_wsgi_server

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     neutron_api =
service.serve_wsgi(service.NeutronApiService)

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/service.py", line 91, in serve_wsgi

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     LOG.exception('Unrecoverable error:
please check log '

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/oslo_utils/excutils.py", line 220, in __exit__

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     self.force_reraise()

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/oslo_utils/excutils.py", line 196, in force_reraise

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     six.reraise(self.type_, self.value,
self.tb)

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/six.py", line 703, in reraise

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     raise value

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/service.py", line 88, in serve_wsgi

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     service.start()

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/service.py", line 64, in start

Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     self.wsgi_app =
_run_wsgi(self.app_name)

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/service.py", line 321, in _run_wsgi

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     app = config.load_paste_app(app_name)

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/common/config.py", line 133, in
load_paste_app

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     app = loader.load_app(app_name)

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/oslo_service/wsgi.py", line 351, in load_app

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return deploy.loadapp("config:%s" %
self.config_path, name=name)

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 253, in loadapp

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return loadobj(APP, uri, name=name,
**kw)

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 278, in loadobj

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return context.create()

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 715, in create

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return self.object_type.invoke(self)

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 148, in invoke

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return fix_call(context.object,

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/util.py", line 55, in fix_call

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     val = callable(*args, **kw)

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/urlmap.py", line 31, in urlmap_factory

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     app = loader.get_app(app_name,
global_conf=global_conf)

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 355, in get_app

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return self.app_context(

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 715, in create

Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return self.object_type.invoke(self)

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 148, in invoke

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return fix_call(context.object,

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/util.py", line 55, in fix_call

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     val = callable(*args, **kw)

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/auth.py", line 47, in pipeline_factory

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     app = loader.get_app(pipeline[-1])

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 355, in get_app

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return self.app_context(

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 715, in create

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return self.object_type.invoke(self)

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/loadwsgi.py", line 152, in invoke

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return fix_call(context.object,
context.global_conf, **context.local_conf)

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
packages/paste/deploy/util.py", line 55, in fix_call

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     val = callable(*args, **kw)

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/api/v2/router.py", line 25, in _factory

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     return
pecan_app.v2_factory(global_config, **local_config)

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/pecan_wsgi/app.py", line 47, in v2_factory

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     startup.initialize_all()

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/pecan_wsgi/startup.py", line 41, in
initialize_all

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     ext_mgr.extend_resources("2.0",
attributes.RESOURCES)

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/api/extensions.py", line 376, in
extend_resources

Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron
self._check_faulty_extensions(unloadable_extensions)

Oct 06 10:46:49.085183 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron   File
"/opt/stack/neutron/neutron/api/extensions.py", line 395, in
_check_faulty_extensions

Oct 06 10:46:49.085183 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron     raise exceptions.ExtensionsNotFound(

Oct 06 10:46:49.085183 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron neutron_lib.exceptions.ExtensionsNotFound:
Extensions not found: ['qos-fip'].

Oct 06 10:46:49.085183 ubuntu-focal-inap-mtl01-0020312575 neutron-
server[107711]: ERROR neutron

** Affects: neutron
     Importance: Critical
         Status: Confirmed


** Tags: gate-failure ovn rally

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1898860

Title:
  neutron-ovn-rally-task failing 100% of times since 06.10.2020

Status in neutron:
  Confirmed

Bug description:
  Example of failure:
  https://zuul.opendev.org/t/openstack/build/0708d0d6e0ee4016b7c32aae98fc98c6/logs

  In neutron-server logs there is error:

  Oct 06 10:46:49.068700 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.api.extensions [None req-
  e44a37df-c419-46c2-82ef-a9696baa8ed0 None None] Unable to process
  extensions (qos-fip) because the configured plugins do not satisfy
  their requirements. Some features will not work as expected.

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service [None req-e44a37df-c419-46c2
  -82ef-a9696baa8ed0 None None] Unrecoverable error: please check log
  for details.: neutron_lib.exceptions.ExtensionsNotFound: Extensions
  not found: ['qos-fip'].

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service Traceback (most recent call
  last):

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/service.py", line 88, in serve_wsgi

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     service.start()

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/service.py", line 64, in start

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     self.wsgi_app =
  _run_wsgi(self.app_name)

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/service.py", line 321, in _run_wsgi

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     app =
  config.load_paste_app(app_name)

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/common/config.py", line 133, in
  load_paste_app

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     app =
  loader.load_app(app_name)

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/oslo_service/wsgi.py", line 351, in load_app

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return
  deploy.loadapp("config:%s" % self.config_path, name=name)

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 253, in loadapp

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return loadobj(APP, uri,
  name=name, **kw)

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 278, in loadobj

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return context.create()

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 715, in create

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return
  self.object_type.invoke(self)

  Oct 06 10:46:49.072799 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 148, in invoke

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return
  fix_call(context.object,

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/util.py", line 55, in fix_call

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     val = callable(*args, **kw)

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/urlmap.py", line 31, in urlmap_factory

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     app =
  loader.get_app(app_name, global_conf=global_conf)

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 355, in get_app

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return self.app_context(

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 715, in create

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return
  self.object_type.invoke(self)

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 148, in invoke

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return
  fix_call(context.object,

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/util.py", line 55, in fix_call

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     val = callable(*args, **kw)

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/auth.py", line 47, in pipeline_factory

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     app =
  loader.get_app(pipeline[-1])

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 355, in get_app

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return self.app_context(

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 715, in create

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return
  self.object_type.invoke(self)

  Oct 06 10:46:49.074866 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/loadwsgi.py", line 152, in invoke

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return
  fix_call(context.object, context.global_conf, **context.local_conf)

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File "/usr/local/lib/python3.8
  /dist-packages/paste/deploy/util.py", line 55, in fix_call

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     val = callable(*args, **kw)

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/api/v2/router.py", line 25, in _factory

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     return
  pecan_app.v2_factory(global_config, **local_config)

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/pecan_wsgi/app.py", line 47, in v2_factory

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     startup.initialize_all()

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/pecan_wsgi/startup.py", line 41, in
  initialize_all

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service
  ext_mgr.extend_resources("2.0", attributes.RESOURCES)

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/api/extensions.py", line 376, in
  extend_resources

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service
  self._check_faulty_extensions(unloadable_extensions)

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service   File
  "/opt/stack/neutron/neutron/api/extensions.py", line 395, in
  _check_faulty_extensions

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service     raise
  exceptions.ExtensionsNotFound(

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service
  neutron_lib.exceptions.ExtensionsNotFound: Extensions not found:
  ['qos-fip'].

  Oct 06 10:46:49.076840 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron.service

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: CRITICAL neutron [None req-e44a37df-c419-46c2-82ef-
  a9696baa8ed0 None None] Unhandled error:
  neutron_lib.exceptions.ExtensionsNotFound: Extensions not found:
  ['qos-fip'].

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron Traceback (most recent call last):

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/bin/neutron-server",
  line 10, in <module>

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     sys.exit(main())

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/cmd/eventlet/server/__init__.py", line 19,
  in main

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron
  server.boot_server(wsgi_eventlet.eventlet_wsgi_server)

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/server/__init__.py", line 68, in
  boot_server

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     server_func()

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/server/wsgi_eventlet.py", line 24, in
  eventlet_wsgi_server

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     neutron_api =
  service.serve_wsgi(service.NeutronApiService)

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/service.py", line 91, in serve_wsgi

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     LOG.exception('Unrecoverable error:
  please check log '

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/oslo_utils/excutils.py", line 220, in __exit__

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     self.force_reraise()

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/oslo_utils/excutils.py", line 196, in force_reraise

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     six.reraise(self.type_, self.value,
  self.tb)

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/six.py", line 703, in reraise

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     raise value

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/service.py", line 88, in serve_wsgi

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     service.start()

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/service.py", line 64, in start

  Oct 06 10:46:49.078577 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     self.wsgi_app =
  _run_wsgi(self.app_name)

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/service.py", line 321, in _run_wsgi

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     app =
  config.load_paste_app(app_name)

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/common/config.py", line 133, in
  load_paste_app

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     app = loader.load_app(app_name)

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/oslo_service/wsgi.py", line 351, in load_app

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return deploy.loadapp("config:%s" %
  self.config_path, name=name)

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 253, in loadapp

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return loadobj(APP, uri, name=name,
  **kw)

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 278, in loadobj

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return context.create()

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 715, in create

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return self.object_type.invoke(self)

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 148, in invoke

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return fix_call(context.object,

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/util.py", line 55, in fix_call

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     val = callable(*args, **kw)

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/urlmap.py", line 31, in urlmap_factory

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     app = loader.get_app(app_name,
  global_conf=global_conf)

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 355, in get_app

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return self.app_context(

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 715, in create

  Oct 06 10:46:49.080780 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return self.object_type.invoke(self)

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 148, in invoke

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return fix_call(context.object,

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/util.py", line 55, in fix_call

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     val = callable(*args, **kw)

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/auth.py", line 47, in pipeline_factory

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     app = loader.get_app(pipeline[-1])

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 355, in get_app

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return self.app_context(

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 715, in create

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return self.object_type.invoke(self)

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/loadwsgi.py", line 152, in invoke

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return fix_call(context.object,
  context.global_conf, **context.local_conf)

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File "/usr/local/lib/python3.8/dist-
  packages/paste/deploy/util.py", line 55, in fix_call

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     val = callable(*args, **kw)

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/api/v2/router.py", line 25, in _factory

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     return
  pecan_app.v2_factory(global_config, **local_config)

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/pecan_wsgi/app.py", line 47, in v2_factory

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     startup.initialize_all()

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/pecan_wsgi/startup.py", line 41, in
  initialize_all

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     ext_mgr.extend_resources("2.0",
  attributes.RESOURCES)

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/api/extensions.py", line 376, in
  extend_resources

  Oct 06 10:46:49.082979 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron
  self._check_faulty_extensions(unloadable_extensions)

  Oct 06 10:46:49.085183 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron   File
  "/opt/stack/neutron/neutron/api/extensions.py", line 395, in
  _check_faulty_extensions

  Oct 06 10:46:49.085183 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron     raise exceptions.ExtensionsNotFound(

  Oct 06 10:46:49.085183 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron
  neutron_lib.exceptions.ExtensionsNotFound: Extensions not found:
  ['qos-fip'].

  Oct 06 10:46:49.085183 ubuntu-focal-inap-mtl01-0020312575 neutron-
  server[107711]: ERROR neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1898860/+subscriptions