yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1882020] [NEW] neutron-ovn-db-sync-util fails with Traceback when notify_nova config is present

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Frode Nordahl <1882020@xxxxxxxxxxxxxxxxxx>
Date: Thu, 04 Jun 2020 06:14:57 -0000
Reply-to: Bug 1882020 <1882020@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If either ``notify_nova_on_port_status_changes`` or
``notify_nova_on_port_data_changes`` is set in the neutron configuration
provided to the ``neutron-ovn-db-sync-util`` tool it will stop with the
following Traceback on startup:

neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini 
2020-06-03 13:12:17.066 484772 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [-] Started Neutron OVN db sync
2020-06-03 13:12:17.407 484772 INFO neutron.manager [-] Loading core plugin: neutron.cmd.ovn.neutron_ovn_db_sync_util.Ml2Plugin
2020-06-03 13:12:17.412 484772 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['geneve', 'vxlan', 'vlan', 'flat', 'local']
2020-06-03 13:12:17.417 484772 INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['physnet1']
2020-06-03 13:12:17.419 484772 INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
2020-06-03 13:12:17.421 484772 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict([('physnet1', [(1000, 2000)])])
2020-06-03 13:12:17.423 484772 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan', 'vxlan']
2020-06-03 13:12:17.423 484772 INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan', 'vxlan'])
2020-06-03 13:12:17.424 484772 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['geneve', 'vxlan', 'vlan', 'flat', 'local']
2020-06-03 13:12:17.424 484772 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: []
2020-06-03 13:12:17.425 484772 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: []
2020-06-03 13:12:17.425 484772 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: []
2020-06-03 13:12:17.425 484772 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn-sync']
2020-06-03 13:12:17.426 484772 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn-sync']
2020-06-03 13:12:17.426 484772 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn-sync']
2020-06-03 13:12:17.426 484772 INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling.
2020-06-03 13:12:17.428 484772 CRITICAL neutron_ovn_db_sync_util [-] Unhandled error: oslo_config.cfg.NoSuchOptError: no such option keystone_authtoken in group [DEFAULT]
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2197, in __getattr__
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     return self._get(name)
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2631, in _get
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     value, loc = self._do_get(name, group, namespace)
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2649, in _do_get
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     info = self._get_opt_info(name, group)
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2849, in _get_opt_info
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     raise NoSuchOptError(opt_name, group)
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util oslo_config.cfg.NoSuchOptError: no such option keystone_authtoken in group [DEFAULT]
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util 
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred:
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util 
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     sys.exit(main())
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 201, in main
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     manager.init()
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 304, in init
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     NeutronManager.get_instance()
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 255, in get_instance
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     cls._create_instance()
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     return f(*args, **kwargs)
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 241, in _create_instance
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     cls._instance = cls()
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 129, in __init__
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     plugin = self._get_plugin_instance(CORE_PLUGINS_NAMESPACE,
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 165, in _get_plugin_instance
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     plugin_inst = plugin_class()
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/quota/resource_registry.py", line 123, in wrapper
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     return f(*args, **kwargs)
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/plugin.py", line 250, in __init__
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     super(Ml2Plugin, self).__init__()
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/db/db_base_plugin_v2.py", line 169, in __init__
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     self.nova_notifier = nova.Notifier.get_instance()
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/notifiers/nova.py", line 55, in get_instance
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     cls._instance = cls()
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/notifiers/nova.py", line 59, in __init__
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     auth = ks_loading.load_auth_from_conf_options(cfg.CONF, 'nova')
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/keystoneauth1/loading/conf.py", line 122, in load_from_conf_options
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     name = conf[group].auth_type
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2205, in __getitem__
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     return self.__getattr__(key)
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2201, in __getattr__
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     raise NoSuchOptError(name)
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util oslo_config.cfg.NoSuchOptError: no such option keystone_authtoken in group [DEFAULT]
2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util

The tool is already overriding the ML2 AgentNotifierApi to avoid
unwanted notifications sent during the sync process so I would suggest
we just override the above mentioned configuration options as they will
not effect the operation of the tool.

** Affects: neutron
     Importance: Undecided
     Assignee: Frode Nordahl (fnordahl)
         Status: In Progress

** Changed in: neutron
       Status: New => In Progress

** Changed in: neutron
     Assignee: (unassigned) => Frode Nordahl (fnordahl)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1882020

Title:
  neutron-ovn-db-sync-util fails with Traceback when notify_nova config
  is present

Status in neutron:
  In Progress

Bug description:
  If either ``notify_nova_on_port_status_changes`` or
  ``notify_nova_on_port_data_changes`` is set in the neutron
  configuration provided to the ``neutron-ovn-db-sync-util`` tool it
  will stop with the following Traceback on startup:

  neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini 
  2020-06-03 13:12:17.066 484772 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [-] Started Neutron OVN db sync
  2020-06-03 13:12:17.407 484772 INFO neutron.manager [-] Loading core plugin: neutron.cmd.ovn.neutron_ovn_db_sync_util.Ml2Plugin
  2020-06-03 13:12:17.412 484772 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['geneve', 'vxlan', 'vlan', 'flat', 'local']
  2020-06-03 13:12:17.417 484772 INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['physnet1']
  2020-06-03 13:12:17.419 484772 INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
  2020-06-03 13:12:17.421 484772 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict([('physnet1', [(1000, 2000)])])
  2020-06-03 13:12:17.423 484772 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan', 'vxlan']
  2020-06-03 13:12:17.423 484772 INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan', 'vxlan'])
  2020-06-03 13:12:17.424 484772 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['geneve', 'vxlan', 'vlan', 'flat', 'local']
  2020-06-03 13:12:17.424 484772 INFO neutron.plugins.ml2.managers [-] Configured extension driver names: []
  2020-06-03 13:12:17.425 484772 INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: []
  2020-06-03 13:12:17.425 484772 INFO neutron.plugins.ml2.managers [-] Registered extension drivers: []
  2020-06-03 13:12:17.425 484772 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn-sync']
  2020-06-03 13:12:17.426 484772 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn-sync']
  2020-06-03 13:12:17.426 484772 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn-sync']
  2020-06-03 13:12:17.426 484772 INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling.
  2020-06-03 13:12:17.428 484772 CRITICAL neutron_ovn_db_sync_util [-] Unhandled error: oslo_config.cfg.NoSuchOptError: no such option keystone_authtoken in group [DEFAULT]
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2197, in __getattr__
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     return self._get(name)
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2631, in _get
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     value, loc = self._do_get(name, group, namespace)
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2649, in _do_get
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     info = self._get_opt_info(name, group)
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2849, in _get_opt_info
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     raise NoSuchOptError(opt_name, group)
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util oslo_config.cfg.NoSuchOptError: no such option keystone_authtoken in group [DEFAULT]
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util 
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred:
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util 
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     sys.exit(main())
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/cmd/ovn/neutron_ovn_db_sync_util.py", line 201, in main
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     manager.init()
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 304, in init
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     NeutronManager.get_instance()
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 255, in get_instance
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     cls._create_instance()
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 359, in inner
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     return f(*args, **kwargs)
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 241, in _create_instance
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     cls._instance = cls()
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 129, in __init__
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     plugin = self._get_plugin_instance(CORE_PLUGINS_NAMESPACE,
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/manager.py", line 165, in _get_plugin_instance
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     plugin_inst = plugin_class()
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/quota/resource_registry.py", line 123, in wrapper
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     return f(*args, **kwargs)
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/plugin.py", line 250, in __init__
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     super(Ml2Plugin, self).__init__()
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/db/db_base_plugin_v2.py", line 169, in __init__
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     self.nova_notifier = nova.Notifier.get_instance()
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/notifiers/nova.py", line 55, in get_instance
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     cls._instance = cls()
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/neutron/notifiers/nova.py", line 59, in __init__
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     auth = ks_loading.load_auth_from_conf_options(cfg.CONF, 'nova')
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/keystoneauth1/loading/conf.py", line 122, in load_from_conf_options
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     name = conf[group].auth_type
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2205, in __getitem__
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     return self.__getattr__(key)
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2201, in __getattr__
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util     raise NoSuchOptError(name)
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util oslo_config.cfg.NoSuchOptError: no such option keystone_authtoken in group [DEFAULT]
  2020-06-03 13:12:17.428 484772 ERROR neutron_ovn_db_sync_util

  The tool is already overriding the ML2 AgentNotifierApi to avoid
  unwanted notifications sent during the sync process so I would suggest
  we just override the above mentioned configuration options as they
  will not effect the operation of the tool.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1882020/+subscriptions
Follow ups

[Bug 1882020] Re: neutron-ovn-db-sync-util fails with Traceback when notify_nova config is present
From: OpenStack Infra, 2020-06-05