yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1881424] [NEW] Neutron ovs agent fails on rpc_loop iteration:1

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Radosław Piliszek <1881424@xxxxxxxxxxxxxxxxxx>
Date: Sat, 30 May 2020 17:13:29 -0000
Reply-to: Bug 1881424 <1881424@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi Neutrinos!

This is from Kolla-Ansible CI, it started happening in Victoria on May 28.
It affects all distros deb-ubu-centos and makes the jobs fail (ovs agent is dead).
It does *not* affect OVN though.

br-ex exists before iteration:0 and acts fine in iteration:0 but not in
iteration:1.

The only quasi-relevant change in neutron seems to be
https://review.opendev.org/721554 but still it should only affect DVR
which we are not running. Odd.

Full logs with config and debug are attached.

The relevant logs look like this (bottom of the file):
2020-05-30 15:27:59.003 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent rpc_loop - iteration:1 started
2020-05-30 15:27:59.006 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] ofctl request version=0x4,msg_type=0x12,msg_len=0x38,xid=0x15f04540,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=23,type=1) result [OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=371283074374527098,duration_nsec=718000000,duration_sec=5,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_count=0,priority=0,table_id=23)],flags=0,type=1)] _send_msg /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:113
2020-05-30 15:27:59.008 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Physical bridge br-ex was just re-created.
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.010 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index target autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.011 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Mapping physical network physnet1 to bridge br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Error executing command: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 38, in execute
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     self.run_idl(None)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 214, in run_idl
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     record = self.api.lookup(self.table, self.record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 171, in lookup
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     return self._lookup(table, record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 218, in _lookup
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 95, in row_by_value
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     raise RowNotFound(table=table, col=column, match=match)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command 
2020-05-30 15:27:59.015 7 DEBUG neutron.agent.common.async_process [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Halting async process [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]. stop /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/common/async_process.py:135
2020-05-30 15:27:59.015 7 DEBUG neutron.agent.linux.utils [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Running command: ['kill', '-9', '75'] create_process /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent main thread died of an exception: TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp Traceback (most recent call last):
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     ovs_agent.main(bridge_classes)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2780, in main
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     agent.daemon_loop()
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2674, in daemon_loop
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     self.rpc_loop(polling_manager=pm)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2545, in rpc_loop
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     added_bridges)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1416, in _reconfigure_physical_bridges
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     sync = self._do_reconfigure_physical_bridges(bridges)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1439, in _do_reconfigure_physical_bridges
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     self.setup_physical_bridges(bridge_mappings)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1495, in setup_physical_bridges
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     self._check_bridge_datapath_id(br, datapath_ids_set)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1457, in _check_bridge_datapath_id
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     dpid = int(bridge.get_datapath_id(), 16)
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp 
2020-05-30 15:27:59.048 7 DEBUG neutron.agent.common.async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: None _read_stdout /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/common/async_process.py:262
2020-05-30 15:27:59.048 7 ERROR neutron.agent.common.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: None
2020-05-30 15:27:59.052 7 CRITICAL neutron [-] Unhandled error: TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.052 7 ERROR neutron Traceback (most recent call last):
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/bin/neutron-openvswitch-agent", line 8, in <module>
2020-05-30 15:27:59.052 7 ERROR neutron     sys.exit(main())
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2020-05-30 15:27:59.052 7 ERROR neutron     agent_main.main()
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 37, in main
2020-05-30 15:27:59.052 7 ERROR neutron     of_main.main()
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
2020-05-30 15:27:59.052 7 ERROR neutron     'neutron.plugins.ml2.drivers.openvswitch.agent.'
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/base/app_manager.py", line 375, in run_apps
2020-05-30 15:27:59.052 7 ERROR neutron     hub.joinall(services)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 102, in joinall
2020-05-30 15:27:59.052 7 ERROR neutron     t.wait()
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 181, in wait
2020-05-30 15:27:59.052 7 ERROR neutron     return self._exit_event.wait()
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/event.py", line 132, in wait
2020-05-30 15:27:59.052 7 ERROR neutron     current.throw(*self._exc)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
2020-05-30 15:27:59.052 7 ERROR neutron     result = function(*args, **kwargs)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 64, in _launch
2020-05-30 15:27:59.052 7 ERROR neutron     raise e
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 59, in _launch
2020-05-30 15:27:59.052 7 ERROR neutron     return func(*args, **kwargs)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 43, in agent_main_wrapper
2020-05-30 15:27:59.052 7 ERROR neutron     LOG.exception("Agent main thread died of an exception")
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-30 15:27:59.052 7 ERROR neutron     self.force_reraise()
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-30 15:27:59.052 7 ERROR neutron     six.reraise(self.type_, self.value, self.tb)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-05-30 15:27:59.052 7 ERROR neutron     raise value
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
2020-05-30 15:27:59.052 7 ERROR neutron     ovs_agent.main(bridge_classes)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2780, in main
2020-05-30 15:27:59.052 7 ERROR neutron     agent.daemon_loop()
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2674, in daemon_loop
2020-05-30 15:27:59.052 7 ERROR neutron     self.rpc_loop(polling_manager=pm)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2545, in rpc_loop
2020-05-30 15:27:59.052 7 ERROR neutron     added_bridges)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1416, in _reconfigure_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron     sync = self._do_reconfigure_physical_bridges(bridges)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1439, in _do_reconfigure_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron     self.setup_physical_bridges(bridge_mappings)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1495, in setup_physical_bridges
2020-05-30 15:27:59.052 7 ERROR neutron     self._check_bridge_datapath_id(br, datapath_ids_set)
2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1457, in _check_bridge_datapath_id
2020-05-30 15:27:59.052 7 ERROR neutron     dpid = int(bridge.get_datapath_id(), 16)
2020-05-30 15:27:59.052 7 ERROR neutron TypeError: int() can't convert non-string with explicit base
2020-05-30 15:27:59.052 7 ERROR neutron

** Affects: kolla-ansible
     Importance: Critical
         Status: Triaged

** Affects: kolla-ansible/victoria
     Importance: Critical
         Status: Triaged

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: ovs

** Attachment added: "full log"
   https://bugs.launchpad.net/bugs/1881424/+attachment/5378811/+files/neutron-openvswitch-agent.txt

** Also affects: kolla-ansible
   Importance: Undecided
       Status: New

** Changed in: kolla-ansible
       Status: New => Triaged

** Changed in: kolla-ansible
   Importance: Undecided => Critical

** Also affects: kolla-ansible/victoria
   Importance: Critical
       Status: Triaged

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1881424

Title:
  Neutron ovs agent fails on rpc_loop iteration:1

Status in kolla-ansible:
  Triaged
Status in kolla-ansible victoria series:
  Triaged
Status in neutron:
  New

Bug description:
  Hi Neutrinos!

  This is from Kolla-Ansible CI, it started happening in Victoria on May 28.
  It affects all distros deb-ubu-centos and makes the jobs fail (ovs agent is dead).
  It does *not* affect OVN though.

  br-ex exists before iteration:0 and acts fine in iteration:0 but not
  in iteration:1.

  The only quasi-relevant change in neutron seems to be
  https://review.opendev.org/721554 but still it should only affect DVR
  which we are not running. Odd.

  Full logs with config and debug are attached.

  The relevant logs look like this (bottom of the file):
  2020-05-30 15:27:59.003 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent rpc_loop - iteration:1 started
  2020-05-30 15:27:59.006 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] ofctl request version=0x4,msg_type=0x12,msg_len=0x38,xid=0x15f04540,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=23,type=1) result [OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=371283074374527098,duration_nsec=718000000,duration_sec=5,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_count=0,priority=0,table_id=23)],flags=0,type=1)] _send_msg /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:113
  2020-05-30 15:27:59.008 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Physical bridge br-ex was just re-created.
  2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
  2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
  2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
  2020-05-30 15:27:59.010 7 DEBUG ovsdbapp.backend.ovs_idl [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index target autocreate_indices /var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
  2020-05-30 15:27:59.011 7 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Mapping physical network physnet1 to bridge br-ex
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Error executing command: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-ex
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 38, in execute
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     self.run_idl(None)
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 214, in run_idl
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     record = self.api.lookup(self.table, self.record)
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 171, in lookup
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     return self._lookup(table, record)
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 218, in _lookup
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 95, in row_by_value
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command     raise RowNotFound(table=table, col=column, match=match)
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-ex
  2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command 
  2020-05-30 15:27:59.015 7 DEBUG neutron.agent.common.async_process [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Halting async process [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]. stop /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/common/async_process.py:135
  2020-05-30 15:27:59.015 7 DEBUG neutron.agent.linux.utils [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Running command: ['kill', '-9', '75'] create_process /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/utils.py:87
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp [req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent main thread died of an exception: TypeError: int() can't convert non-string with explicit base
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp Traceback (most recent call last):
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     ovs_agent.main(bridge_classes)
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2780, in main
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     agent.daemon_loop()
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2674, in daemon_loop
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     self.rpc_loop(polling_manager=pm)
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2545, in rpc_loop
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     added_bridges)
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1416, in _reconfigure_physical_bridges
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     sync = self._do_reconfigure_physical_bridges(bridges)
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1439, in _do_reconfigure_physical_bridges
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     self.setup_physical_bridges(bridge_mappings)
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1495, in setup_physical_bridges
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     self._check_bridge_datapath_id(br, datapath_ids_set)
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1457, in _check_bridge_datapath_id
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp     dpid = int(bridge.get_datapath_id(), 16)
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp TypeError: int() can't convert non-string with explicit base
  2020-05-30 15:27:59.036 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp 
  2020-05-30 15:27:59.048 7 DEBUG neutron.agent.common.async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: None _read_stdout /var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/common/async_process.py:262
  2020-05-30 15:27:59.048 7 ERROR neutron.agent.common.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: None
  2020-05-30 15:27:59.052 7 CRITICAL neutron [-] Unhandled error: TypeError: int() can't convert non-string with explicit base
  2020-05-30 15:27:59.052 7 ERROR neutron Traceback (most recent call last):
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/bin/neutron-openvswitch-agent", line 8, in <module>
  2020-05-30 15:27:59.052 7 ERROR neutron     sys.exit(main())
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
  2020-05-30 15:27:59.052 7 ERROR neutron     agent_main.main()
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 37, in main
  2020-05-30 15:27:59.052 7 ERROR neutron     of_main.main()
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
  2020-05-30 15:27:59.052 7 ERROR neutron     'neutron.plugins.ml2.drivers.openvswitch.agent.'
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/base/app_manager.py", line 375, in run_apps
  2020-05-30 15:27:59.052 7 ERROR neutron     hub.joinall(services)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 102, in joinall
  2020-05-30 15:27:59.052 7 ERROR neutron     t.wait()
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 181, in wait
  2020-05-30 15:27:59.052 7 ERROR neutron     return self._exit_event.wait()
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/event.py", line 132, in wait
  2020-05-30 15:27:59.052 7 ERROR neutron     current.throw(*self._exc)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
  2020-05-30 15:27:59.052 7 ERROR neutron     result = function(*args, **kwargs)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 64, in _launch
  2020-05-30 15:27:59.052 7 ERROR neutron     raise e
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/os_ken/lib/hub.py", line 59, in _launch
  2020-05-30 15:27:59.052 7 ERROR neutron     return func(*args, **kwargs)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 43, in agent_main_wrapper
  2020-05-30 15:27:59.052 7 ERROR neutron     LOG.exception("Agent main thread died of an exception")
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-05-30 15:27:59.052 7 ERROR neutron     self.force_reraise()
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-05-30 15:27:59.052 7 ERROR neutron     six.reraise(self.type_, self.value, self.tb)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 703, in reraise
  2020-05-30 15:27:59.052 7 ERROR neutron     raise value
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
  2020-05-30 15:27:59.052 7 ERROR neutron     ovs_agent.main(bridge_classes)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2780, in main
  2020-05-30 15:27:59.052 7 ERROR neutron     agent.daemon_loop()
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2674, in daemon_loop
  2020-05-30 15:27:59.052 7 ERROR neutron     self.rpc_loop(polling_manager=pm)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2545, in rpc_loop
  2020-05-30 15:27:59.052 7 ERROR neutron     added_bridges)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1416, in _reconfigure_physical_bridges
  2020-05-30 15:27:59.052 7 ERROR neutron     sync = self._do_reconfigure_physical_bridges(bridges)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1439, in _do_reconfigure_physical_bridges
  2020-05-30 15:27:59.052 7 ERROR neutron     self.setup_physical_bridges(bridge_mappings)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1495, in setup_physical_bridges
  2020-05-30 15:27:59.052 7 ERROR neutron     self._check_bridge_datapath_id(br, datapath_ids_set)
  2020-05-30 15:27:59.052 7 ERROR neutron   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1457, in _check_bridge_datapath_id
  2020-05-30 15:27:59.052 7 ERROR neutron     dpid = int(bridge.get_datapath_id(), 16)
  2020-05-30 15:27:59.052 7 ERROR neutron TypeError: int() can't convert non-string with explicit base
  2020-05-30 15:27:59.052 7 ERROR neutron

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla-ansible/+bug/1881424/+subscriptions
Follow ups

[Bug 1881424] Re: Neutron ovs agent fails on rpc_loop iteration:1
From: Terry Wilson, 2021-05-13
[Bug 1881424] Re: Neutron ovs agent fails on rpc_loop iteration:1
From: Mark Goddard, 2020-06-15
[Bug 1881424] Re: Neutron ovs agent fails on rpc_loop iteration:1
From: Radosław Piliszek, 2020-05-31