yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1965599] [NEW] python-ovs package 2.17 introduce table.condition attribute which is overwriten by neutron ovsdb code

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mirek <1965599@xxxxxxxxxxxxxxxxxx>
Date: Fri, 18 Mar 2022 22:12:03 -0000
Reply-to: Bug 1965599 <1965599@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

I've opened it up initially with openvswitch team before I dug a bit more into it. It looks like changes in 2.17 code introduce condition attribute to sync changes. However, neutron also use condition attribute, e.g. neutron/agent/ovn/metadata/ovsdb.py
class MetadataAgentOvnSbIdl(ovsdb_monitor.OvnIdl):
...
        if chassis:
            for table in set(tables).intersection({'Chassis',
                                                   'Chassis_Private'}):
                self.tables[table].condition = [['name', '==', chassis]]

So what happen is ovs idl code is falling with the following error:
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: ''list' object has no attribute 'request''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: AttributeError: 'list' object has no attribute 'request'
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 127, in start_connection
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 83, in start
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 53, in wait_for_change
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     while idl_.change_seqno == seqno and not idl_.run():
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 421, in run
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.restart_fsm()
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 383, in restart_fsm
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.sync_conditions()
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 372, in sync_conditions
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     table.condition.request()
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn AttributeError: 'list' object has no attribute 'request'


When I've added a few print lines to find out what is going on, I found out that ovs is expecting ConditionState object, but neutron code is also using condition attribute and when I've override __setattr__ func to find out who is overwrite the ovs code. It turns out ovsdb.py also use that attribute.
    
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 150, in publish
    errors = self._notify_loop(resource, event, trigger, payload)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
    callback(resource, event, trigger, payload=payload)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/agent/ovn/metadata/server.py", line 74, in post_fork_initialize
    self.sb_idl = ovsdb.MetadataAgentOvnSbIdl(
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/agent/ovn/metadata/ovsdb.py", line 53, in __init__
    self.tables[table].condition = [['name', '==', chassis]]
  File "/usr/lib/python3/dist-packages/ovs/db/schema.py", line 190, in __setattr__
    vlog.err(''.join(traceback.format_stack()))
    
    
https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/ovn/metadata/ovsdb.py
class MetadataAgentOvnSbIdl(ovsdb_monitor.OvnIdl):

    SCHEMA = 'OVN_Southbound'

    def __init__(self, chassis=None, events=None, tables=None):
        connection_string = config.get_ovn_sb_connection()
        ovsdb_monitor._check_and_set_ssl_files(self.SCHEMA)
        helper = self._get_ovsdb_helper(connection_string)
        if tables is None:
            tables = ('Chassis', 'Encap', 'Port_Binding', 'Datapath_Binding',
                      'SB_Global')
        for table in tables:
            helper.register_table(table)
        try:
            super(MetadataAgentOvnSbIdl, self).__init__(
                None, connection_string, helper, leader_only=False)
        except TypeError:
            # TODO(twilson) We can remove this when we require ovs>=2.12.0
            super(MetadataAgentOvnSbIdl, self).__init__(
                None, connection_string, helper)
        if chassis:
            for table in set(tables).intersection({'Chassis',
                                                   'Chassis_Private'}):
                self.tables[table].condition = [['name', '==', chassis]]


The workaround at the moment is to downgrade to 2.16, however, it's problematic as neutron requiments.txt are only setting lower limit for ovs package. 
ovs>=2.10.0 # Apache-2.0

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1965599

Title:
  python-ovs package 2.17 introduce table.condition attribute which is
  overwriten by neutron ovsdb code

Status in neutron:
  New

Bug description:
  I've opened it up initially with openvswitch team before I dug a bit more into it. It looks like changes in 2.17 code introduce condition attribute to sync changes. However, neutron also use condition attribute, e.g. neutron/agent/ovn/metadata/ovsdb.py
  class MetadataAgentOvnSbIdl(ovsdb_monitor.OvnIdl):
  ...
          if chassis:
              for table in set(tables).intersection({'Chassis',
                                                     'Chassis_Private'}):
                  self.tables[table].condition = [['name', '==', chassis]]

  So what happen is ovs idl code is falling with the following error:
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: ''list' object has no attribute 'request''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: AttributeError: 'list' object has no attribute 'request'
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 127, in start_connection
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 83, in start
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 53, in wait_for_change
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     while idl_.change_seqno == seqno and not idl_.run():
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 421, in run
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.restart_fsm()
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 383, in restart_fsm
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.sync_conditions()
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 372, in sync_conditions
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     table.condition.request()
  2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn AttributeError: 'list' object has no attribute 'request'

  
  When I've added a few print lines to find out what is going on, I found out that ovs is expecting ConditionState object, but neutron code is also using condition attribute and when I've override __setattr__ func to find out who is overwrite the ovs code. It turns out ovsdb.py also use that attribute.
      
    File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 150, in publish
      errors = self._notify_loop(resource, event, trigger, payload)
    File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
      callback(resource, event, trigger, payload=payload)
    File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/agent/ovn/metadata/server.py", line 74, in post_fork_initialize
      self.sb_idl = ovsdb.MetadataAgentOvnSbIdl(
    File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/agent/ovn/metadata/ovsdb.py", line 53, in __init__
      self.tables[table].condition = [['name', '==', chassis]]
    File "/usr/lib/python3/dist-packages/ovs/db/schema.py", line 190, in __setattr__
      vlog.err(''.join(traceback.format_stack()))
      
      
  https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/ovn/metadata/ovsdb.py
  class MetadataAgentOvnSbIdl(ovsdb_monitor.OvnIdl):

      SCHEMA = 'OVN_Southbound'

      def __init__(self, chassis=None, events=None, tables=None):
          connection_string = config.get_ovn_sb_connection()
          ovsdb_monitor._check_and_set_ssl_files(self.SCHEMA)
          helper = self._get_ovsdb_helper(connection_string)
          if tables is None:
              tables = ('Chassis', 'Encap', 'Port_Binding', 'Datapath_Binding',
                        'SB_Global')
          for table in tables:
              helper.register_table(table)
          try:
              super(MetadataAgentOvnSbIdl, self).__init__(
                  None, connection_string, helper, leader_only=False)
          except TypeError:
              # TODO(twilson) We can remove this when we require ovs>=2.12.0
              super(MetadataAgentOvnSbIdl, self).__init__(
                  None, connection_string, helper)
          if chassis:
              for table in set(tables).intersection({'Chassis',
                                                     'Chassis_Private'}):
                  self.tables[table].condition = [['name', '==', chassis]]

  
  The workaround at the moment is to downgrade to 2.16, however, it's problematic as neutron requiments.txt are only setting lower limit for ovs package. 
  ovs>=2.10.0 # Apache-2.0

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1965599/+subscriptions