yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #86070
[Bug 1881424] Re: Neutron ovs agent fails on rpc_loop iteration:1
Looks like this was fixed by Rodolfo's singleton patch
** No longer affects: ovsdbapp
** Changed in: neutron
Status: New => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1881424
Title:
Neutron ovs agent fails on rpc_loop iteration:1
Status in kolla-ansible:
Invalid
Status in kolla-ansible victoria series:
Invalid
Status in neutron:
Fix Released
Bug description:
Hi Neutrinos!
This is from Kolla-Ansible CI, it started happening in Victoria on May 28.
It affects all distros deb-ubu-centos and makes the jobs fail (ovs agent is dead).
It does *not* affect OVN though.
br-ex exists before iteration:0 and acts fine in iteration:0 but not
in iteration:1.
The only quasi-relevant change in neutron seems to be
https://review.opendev.org/721554 but still it should only affect DVR
which we are not running. Odd.
Full logs with config and debug are attached.
The relevant logs look like this (bottom of the file):
2020-05-30 15:27:59.003 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent rpc_loop - iteration:1 started
2020-05-30 15:27:59.006 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] ofctl request version=0x4,msg_type=0x12,msg_len=0x38,xid=0x15f04540,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=23,type=1) result [OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=371283074374527098,duration_nsec=718000000,duration_sec=5,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_count=0,priority=0,table_id=23)],flags=0,type=1)] _send_msg /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:113
2020-05-30 15:27:59.008 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Physical bridge br-ex was just re-created.
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.010 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index target autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.011 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Mapping physical network physnet1 to bridge br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Error executing command: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 38, in execute
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command self.run_idl(None)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 214, in run_idl
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command record = self.api.lookup(self.table, self.record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 171, in lookup
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command return self._lookup(table, record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 218, in _lookup
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command row = idlutils.row_by_value(self, rl.table, rl.column, record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 95, in row_by_value
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command raise RowNotFound(table=table, col=column, match=match)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command
2020-05-30 15:27:59.015 7 DEBUG neutron.agent.common.async_process [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Halting async process [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]. stop /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/common/async_process.py:135
2020-05-30 15:27:59.015 7 DEBUG neutron.agent.linux.utils [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Running command: ['kill', '-9', '75'] create_process /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent main thread died of an exception: TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp Traceback (most recent call last):
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp ovs_agent.main(bridge_classes)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2780, in main
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp agent.daemon_loop()
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2674, in daemon_loop
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp self.rpc_loop(polling_manager=pm)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2545, in rpc_loop
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp added_bridges)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1416, in _reconfigure_physical_bridges
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp sync = self._do_reconfigure_physical_bridges(bridges)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1439, in _do_reconfigure_physical_bridges
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp self.setup_physical_bridges(bridge_mappings)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1495, in setup_physical_bridges
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp self._check_bridge_datapath_id(br, datapath_ids_set)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1457, in _check_bridge_datapath_id
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp dpid = int(bridge.get_datapath_id(), 16)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
2020-05-30 15:27:59.048 7 DEBUG neutron.agent.common.async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: None _read_stdout /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/common/async_process.py:262
2020-05-30 15:27:59.048 7 ERROR neutron.agent.common.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: None
2020-05-30 15:27:59.052 7 CRITICAL neutron [-] Unhandled error: TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.052 7 ERROR neutron Traceback (most recent call last):
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/bin/neutron-openvswitch-agent", line 8, in <module>
2020-05-30 15:27:59.052 7 ERROR neutron sys.exit(main())
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2020-05-30 15:27:59.052 7 ERROR neutron agent_main.main()
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 37, in main
2020-05-30 15:27:59.052 7 ERROR neutron of_main.main()
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
2020-05-30 15:27:59.052 7 ERROR neutron 'neutron.plugins.ml2.drivers.openvswitch.agent.'
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/base/app_manager.py", line 375, in run_apps
2020-05-30 15:27:59.052 7 ERROR neutron hub.joinall(services)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 102, in joinall
2020-05-30 15:27:59.052 7 ERROR neutron t.wait()
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 181, in wait
2020-05-30 15:27:59.052 7 ERROR neutron return self._exit_event.wait()
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/event.py", line 132, in wait
2020-05-30 15:27:59.052 7 ERROR neutron current.throw(*self._exc)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
2020-05-30 15:27:59.052 7 ERROR neutron result = function(*args, **kwargs)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 64, in _launch
2020-05-30 15:27:59.052 7 ERROR neutron raise e
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 59, in _launch
2020-05-30 15:27:59.052 7 ERROR neutron return func(*args, **kwargs)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 43, in agent_main_wrapper
2020-05-30 15:27:59.052 7 ERROR neutron LOG.exception("Agent main thread died of an exception")
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-30 15:27:59.052 7 ERROR neutron self.force_reraise()
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-30 15:27:59.052 7 ERROR neutron six.reraise(self.type_, self.value, self.tb)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-05-30 15:27:59.052 7 ERROR neutron raise value
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
2020-05-30 15:27:59.052 7 ERROR neutron ovs_agent.main(bridge_classes)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2780, in main
2020-05-30 15:27:59.052 7 ERROR neutron agent.daemon_loop()
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2674, in daemon_loop
2020-05-30 15:27:59.052 7 ERROR neutron self.rpc_loop(polling_manager=pm)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2545, in rpc_loop
2020-05-30 15:27:59.052 7 ERROR neutron added_bridges)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1416, in _reconfigure_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron sync = self._do_reconfigure_physical_bridges(bridges)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1439, in _do_reconfigure_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron self.setup_physical_bridges(bridge_mappings)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1495, in setup_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron self._check_bridge_datapath_id(br, datapath_ids_set)
2020-05-30 15:27:59.052 7 ERROR neutron File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1457, in _check_bridge_datapath_id
2020-05-30 15:27:59.052 7 ERROR neutron dpid = int(bridge.get_datapath_id(), 16)
2020-05-30 15:27:59.052 7 ERROR neutron TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.052 7 ERROR neutron
To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1881424/+subscriptions
References