yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1842517] [NEW] neutron-sanity-check command fails if netdev datapath is used

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Deepak Tiwari <1842517@xxxxxxxxxxxxxxxxxx>
Date: Tue, 03 Sep 2019 21:36:13 -0000
Reply-to: Bug 1842517 <1842517@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If ovs-dpdk is being used, in containerized openstack deployment,
restarting the neutron pods sometimes leads to neutron-sanity-check
command getting failed. This script tries to create several bridges but
with 'system' datapath_type. Even though 'netdev' datapath_type bridges
were already created by core neutron_ovs_agent process.

steps:
------
1. Deploy ovs-dpdk in a containerized environment (using openstack-helm for ex.)
2. Deploy neutron pods
3. First time neutron-ovs-agent pods shall come up fine (and it shall create ovs bridges with netdev datapath_type)
4. Then restart neutron pods multiple times unless following issue is observed
5. The init containers shall run neutron-sanity-check which tries to create ovs bridges with 'system' datapath_type which fails randomly....

Logs:
------
+ OVS_SOCKET=/run/openvswitch/db.sock
+ chown neutron: /run/openvswitch/db.sock
+ DPDK_CONFIG_FILE=/tmp/dpdk.conf
+ DPDK_CONFIG=
+ '[' '!' -f /tmp/dpdk.conf ']'
++ cat /tmp/dpdk.conf
+ DPDK_CONFIG='{"bonds":[{"bridge":"br-phy-bond0","migrate_ip":false,"mtu":9000,"n_rxq":4,"n_rxq_size":1024,"n_txq_size":1024,"name":"dpdkbond0","nics":[{"name":"dpdk_b0s0","pci_id":"0000:5e:00.0","vf_index":0},{"name":"dpdk_b0s1","pci_id":"0000:87:00.1","vf_index":0}],"ovs_options":"bond_mode=active-backup"}],"bridges":[{"name":"br-phy-bond0"}],"driver":"vfio-pci","enabled":true,"nics":[]}'
+ neutron-sanity-check --version
+ timeout 3m neutron-sanity-check --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --ovsdb_native --nokeepalived_ipv6_support
Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2019-08-31 04:07:15.696 1161 INFO neutron.common.config [-] Logging enabled!
2019-08-31 04:07:15.696 1161 INFO neutron.common.config [-] /var/lib/openstack/bin/neutron-sanity-check version 10.0.8.dev105
2019-08-31 04:07:16.572 1161 INFO neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connecting...
2019-08-31 04:07:16.572 1161 INFO neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connected
2019-08-31 04:07:26.612 1161 CRITICAL neutron [-] TimeoutException: Commands [AddBridgeCommand(datapath_type=system, may_exist=True, name=patchtest-4e35d), DbAddCommand(column=protocols, record=patchtest-4e35d, values=('OpenFlow10',), table=Bridge), DbSetCommand(table=Bridge, col_values=(('other_config', {'mac-table-size': '50000'}),), record=patchtest-4e35d)] exceeded timeout 10 seconds
2019-08-31 04:07:26.612 1161 ERROR neutron Traceback (most recent call last):
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/bin/neutron-sanity-check", line 10, in <module>
2019-08-31 04:07:26.612 1161 ERROR neutron     sys.exit(main())
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity_check.py", line 394, in main
2019-08-31 04:07:26.612 1161 ERROR neutron     return 0 if all_tests_passed() else 1
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity_check.py", line 381, in all_tests_passed
2019-08-31 04:07:26.612 1161 ERROR neutron     return all(opt.callback() for opt in OPTS if cfg.CONF.get(opt.name))
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity_check.py", line 381, in <genexpr>
2019-08-31 04:07:26.612 1161 ERROR neutron     return all(opt.callback() for opt in OPTS if cfg.CONF.get(opt.name))
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity_check.py", line 79, in check_ovs_patch
2019-08-31 04:07:26.612 1161 ERROR neutron     result = checks.patch_supported()
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity/checks.py", line 74, in patch_supported
2019-08-31 04:07:26.612 1161 ERROR neutron     with ovs_lib.OVSBridge(name) as br:
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 692, in __enter__
2019-08-31 04:07:26.612 1161 ERROR neutron     self.create()
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 258, in create
2019-08-31 04:07:26.612 1161 ERROR neutron     FAILMODE_SECURE))
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py", line 79, in __exit__
2019-08-31 04:07:26.612 1161 ERROR neutron     self.result = self.commit()
2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 73, in commit
2019-08-31 04:07:26.612 1161 ERROR neutron     'timeout': self.timeout})
2019-08-31 04:07:26.612 1161 ERROR neutron TimeoutException: Commands [AddBridgeCommand(datapath_type=system, may_exist=True, name=patchtest-4e35d), DbAddCommand(column=protocols, record=patchtest-4e35d, values=('OpenFlow10',), table=Bridge), DbSetCommand(table=Bridge, col_values=(('other_config', {'mac-table-size': '50000'}),), record=patchtest-4e35d)] exceeded timeout 10 seconds

** Affects: neutron
     Importance: Undecided
     Assignee: Deepak Tiwari (deepak.tiwari)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Deepak Tiwari (deepak.tiwari)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1842517

Title:
  neutron-sanity-check command fails if netdev datapath is used

Status in neutron:
  In Progress

Bug description:
  If ovs-dpdk is being used, in containerized openstack deployment,
  restarting the neutron pods sometimes leads to neutron-sanity-check
  command getting failed. This script tries to create several bridges
  but with 'system' datapath_type. Even though 'netdev' datapath_type
  bridges were already created by core neutron_ovs_agent process.

  steps:
  ------
  1. Deploy ovs-dpdk in a containerized environment (using openstack-helm for ex.)
  2. Deploy neutron pods
  3. First time neutron-ovs-agent pods shall come up fine (and it shall create ovs bridges with netdev datapath_type)
  4. Then restart neutron pods multiple times unless following issue is observed
  5. The init containers shall run neutron-sanity-check which tries to create ovs bridges with 'system' datapath_type which fails randomly....

  Logs:
  ------
  + OVS_SOCKET=/run/openvswitch/db.sock
  + chown neutron: /run/openvswitch/db.sock
  + DPDK_CONFIG_FILE=/tmp/dpdk.conf
  + DPDK_CONFIG=
  + '[' '!' -f /tmp/dpdk.conf ']'
  ++ cat /tmp/dpdk.conf
  + DPDK_CONFIG='{"bonds":[{"bridge":"br-phy-bond0","migrate_ip":false,"mtu":9000,"n_rxq":4,"n_rxq_size":1024,"n_txq_size":1024,"name":"dpdkbond0","nics":[{"name":"dpdk_b0s0","pci_id":"0000:5e:00.0","vf_index":0},{"name":"dpdk_b0s1","pci_id":"0000:87:00.1","vf_index":0}],"ovs_options":"bond_mode=active-backup"}],"bridges":[{"name":"br-phy-bond0"}],"driver":"vfio-pci","enabled":true,"nics":[]}'
  + neutron-sanity-check --version
  + timeout 3m neutron-sanity-check --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --ovsdb_native --nokeepalived_ipv6_support
  Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
  2019-08-31 04:07:15.696 1161 INFO neutron.common.config [-] Logging enabled!
  2019-08-31 04:07:15.696 1161 INFO neutron.common.config [-] /var/lib/openstack/bin/neutron-sanity-check version 10.0.8.dev105
  2019-08-31 04:07:16.572 1161 INFO neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connecting...
  2019-08-31 04:07:16.572 1161 INFO neutron.agent.ovsdb.native.vlog [-] tcp:127.0.0.1:6640: connected
  2019-08-31 04:07:26.612 1161 CRITICAL neutron [-] TimeoutException: Commands [AddBridgeCommand(datapath_type=system, may_exist=True, name=patchtest-4e35d), DbAddCommand(column=protocols, record=patchtest-4e35d, values=('OpenFlow10',), table=Bridge), DbSetCommand(table=Bridge, col_values=(('other_config', {'mac-table-size': '50000'}),), record=patchtest-4e35d)] exceeded timeout 10 seconds
  2019-08-31 04:07:26.612 1161 ERROR neutron Traceback (most recent call last):
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/bin/neutron-sanity-check", line 10, in <module>
  2019-08-31 04:07:26.612 1161 ERROR neutron     sys.exit(main())
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity_check.py", line 394, in main
  2019-08-31 04:07:26.612 1161 ERROR neutron     return 0 if all_tests_passed() else 1
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity_check.py", line 381, in all_tests_passed
  2019-08-31 04:07:26.612 1161 ERROR neutron     return all(opt.callback() for opt in OPTS if cfg.CONF.get(opt.name))
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity_check.py", line 381, in <genexpr>
  2019-08-31 04:07:26.612 1161 ERROR neutron     return all(opt.callback() for opt in OPTS if cfg.CONF.get(opt.name))
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity_check.py", line 79, in check_ovs_patch
  2019-08-31 04:07:26.612 1161 ERROR neutron     result = checks.patch_supported()
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/cmd/sanity/checks.py", line 74, in patch_supported
  2019-08-31 04:07:26.612 1161 ERROR neutron     with ovs_lib.OVSBridge(name) as br:
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 692, in __enter__
  2019-08-31 04:07:26.612 1161 ERROR neutron     self.create()
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 258, in create
  2019-08-31 04:07:26.612 1161 ERROR neutron     FAILMODE_SECURE))
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py", line 79, in __exit__
  2019-08-31 04:07:26.612 1161 ERROR neutron     self.result = self.commit()
  2019-08-31 04:07:26.612 1161 ERROR neutron   File "/var/lib/openstack/local/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 73, in commit
  2019-08-31 04:07:26.612 1161 ERROR neutron     'timeout': self.timeout})
  2019-08-31 04:07:26.612 1161 ERROR neutron TimeoutException: Commands [AddBridgeCommand(datapath_type=system, may_exist=True, name=patchtest-4e35d), DbAddCommand(column=protocols, record=patchtest-4e35d, values=('OpenFlow10',), table=Bridge), DbSetCommand(table=Bridge, col_values=(('other_config', {'mac-table-size': '50000'}),), record=patchtest-4e35d)] exceeded timeout 10 seconds

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1842517/+subscriptions
Follow ups

[Bug 1842517] Re: neutron-sanity-check command fails if netdev datapath is used
From: OpenStack Infra, 2023-01-12