yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1623876] [NEW] nova is not setting the MTU provided by Neutron

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Benton <1623876@xxxxxxxxxxxxxxxxxx>
Date: Thu, 15 Sep 2016 10:08:36 -0000
Reply-to: Bug 1623876 <1623876@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Spotted in gate grenade job. We can see neutron MTU is 1450 but the mtu set calls in privsep use 1500.
http://logs.openstack.org/56/369956/3/gate/gate-grenade-dsvm-neutron-ubuntu-trusty/83daad8/logs/new/screen-n-cpu.txt.gz#_2016-09-15_01_16_57_512

Relevant log snippet:

2016-09-15 01:16:57.512 25573 DEBUG nova.network.os_vif_util [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Converting VIF {"profile": {}, "ovs_interfaceid": "8dfdfd9b-da9d-4215-abbd-4dffdc48494b", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "10.1.0.9"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "10.1.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.1.0.1"}}], "meta": {"injected": false, "tenant_id": "563ca55619b1402ebf0c792ec604a774", "mtu": 1450}, "id": "6e1f0d14-a238-4da9-a2d5-659a0f28479c", "label": "tempest-AttachInterfacesTestJSON-32449395-network"}, "devname": "tap8dfdfd9b-da", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true, "ovs_hybrid_plug": true}, "address": "fa:16:3e:38:52:12", "active": false, "type": "ovs", "id": "8dfdfd9b-da9d-4215-abbd-4dffdc48494b", "qbg_params": null} nova_to_osvif_vif /opt/stack/new/nova/nova/network/os_vif_util.py:362
2016-09-15 01:16:57.513 25573 DEBUG nova.network.os_vif_util [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Converted object VIFBridge(active=False,address=fa:16:3e:38:52:12,bridge_name='qbr8dfdfd9b-da',has_traffic_filtering=True,id=8dfdfd9b-da9d-4215-abbd-4dffdc48494b,network=Network(6e1f0d14-a238-4da9-a2d5-659a0f28479c),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap8dfdfd9b-da') nova_to_osvif_vif /opt/stack/new/nova/nova/network/os_vif_util.py:374
2016-09-15 01:16:57.514 25573 DEBUG os_vif [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Plugging vif VIFBridge(active=False,address=fa:16:3e:38:52:12,bridge_name='qbr8dfdfd9b-da',has_traffic_filtering=True,id=8dfdfd9b-da9d-4215-abbd-4dffdc48494b,network=Network(6e1f0d14-a238-4da9-a2d5-659a0f28479c),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap8dfdfd9b-da') plug /usr/local/lib/python2.7/dist-packages/os_vif/__init__.py:76
2016-09-15 01:16:57.515 25573 DEBUG oslo.privsep.daemon [-] privsep: request[140021949493072]: (3, 'vif_plug_ovs.linux_net.ensure_bridge', (u'qbr8dfdfd9b-da',), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.515 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl addbr qbr8dfdfd9b-da out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.517 25573 DEBUG neutronclient.v2_0.client [req-6754757c-066c-488a-bc16-6bd451c28cdc tempest-ServerActionsTestJSON-1197364963 tempest-ServerActionsTestJSON-1197364963] GET call to neutron for http://127.0.0.1:9696/v2.0/subnets.json?id=2b899e3c-17dc-478a-bd39-91132bb057ab used request id req-e82dd927-a0f8-48ba-bbbc-56724a10a29d _append_request_id /usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:127
2016-09-15 01:16:57.521 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl addbr qbr8dfdfd9b-da" returned: 0 in 0.005s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.521 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl setfd qbr8dfdfd9b-da 0 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.524 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl setfd qbr8dfdfd9b-da 0" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.524 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl stp qbr8dfdfd9b-da off out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.528 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl stp qbr8dfdfd9b-da off" returned: 0 in 0.003s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.528 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): tee /sys/class/net/qbr8dfdfd9b-da/bridge/multicast_snooping out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.532 25573 DEBUG oslo.privsep.daemon [-] CMD "tee /sys/class/net/qbr8dfdfd9b-da/bridge/multicast_snooping" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.532 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): tee /proc/sys/net/ipv6/conf/qbr8dfdfd9b-da/disable_ipv6 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] CMD "tee /proc/sys/net/ipv6/conf/qbr8dfdfd9b-da/disable_ipv6" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] privsep: reply[140021949493072]: (4, None) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] privsep: request[140021949493072]: (3, 'vif_plug_ovs.linux_net.create_veth_pair', ('qvb8dfdfd9b-da', 'qvo8dfdfd9b-da', 1500), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.538 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link add qvb8dfdfd9b-da type veth peer name qvo8dfdfd9b-da out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.546 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link add qvb8dfdfd9b-da type veth peer name qvo8dfdfd9b-da" returned: 0 in 0.007s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.547 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da up out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.550 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da up" returned: 0 in 0.003s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.550 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da promisc on out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.556 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da promisc on" returned: 0 in 0.006s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.556 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da mtu 1500 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.560 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da mtu 1500" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-

** Affects: nova
     Importance: Undecided
     Assignee: Kevin Benton (kevinbenton)
         Status: New

** Affects: os-vif
     Importance: Undecided
     Assignee: Kevin Benton (kevinbenton)
         Status: In Progress

** Changed in: nova
     Assignee: (unassigned) => Kevin Benton (kevinbenton)

** Also affects: os-vif
   Importance: Undecided
       Status: New

** Changed in: os-vif
     Assignee: (unassigned) => Kevin Benton (kevinbenton)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1623876

Title:
  nova is not setting the MTU provided by Neutron

Status in OpenStack Compute (nova):
  New
Status in os-vif:
  In Progress

Bug description:
  Spotted in gate grenade job. We can see neutron MTU is 1450 but the mtu set calls in privsep use 1500.
  http://logs.openstack.org/56/369956/3/gate/gate-grenade-dsvm-neutron-ubuntu-trusty/83daad8/logs/new/screen-n-cpu.txt.gz#_2016-09-15_01_16_57_512

  Relevant log snippet:

  2016-09-15 01:16:57.512 25573 DEBUG nova.network.os_vif_util [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Converting VIF {"profile": {}, "ovs_interfaceid": "8dfdfd9b-da9d-4215-abbd-4dffdc48494b", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "10.1.0.9"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "10.1.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.1.0.1"}}], "meta": {"injected": false, "tenant_id": "563ca55619b1402ebf0c792ec604a774", "mtu": 1450}, "id": "6e1f0d14-a238-4da9-a2d5-659a0f28479c", "label": "tempest-AttachInterfacesTestJSON-32449395-network"}, "devname": "tap8dfdfd9b-da", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true, "ovs_hybrid_plug": true}, "address": "fa:16:3e:38:52:12", "active": false, "type": "ovs", "id": "8dfdfd9b-da9d-4215-abbd-4dffdc48494b", "qbg_params": null} nova_to_osvif_vif /opt/stack/new/nova/nova/network/os_vif_util.py:362
  2016-09-15 01:16:57.513 25573 DEBUG nova.network.os_vif_util [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Converted object VIFBridge(active=False,address=fa:16:3e:38:52:12,bridge_name='qbr8dfdfd9b-da',has_traffic_filtering=True,id=8dfdfd9b-da9d-4215-abbd-4dffdc48494b,network=Network(6e1f0d14-a238-4da9-a2d5-659a0f28479c),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap8dfdfd9b-da') nova_to_osvif_vif /opt/stack/new/nova/nova/network/os_vif_util.py:374
  2016-09-15 01:16:57.514 25573 DEBUG os_vif [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Plugging vif VIFBridge(active=False,address=fa:16:3e:38:52:12,bridge_name='qbr8dfdfd9b-da',has_traffic_filtering=True,id=8dfdfd9b-da9d-4215-abbd-4dffdc48494b,network=Network(6e1f0d14-a238-4da9-a2d5-659a0f28479c),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap8dfdfd9b-da') plug /usr/local/lib/python2.7/dist-packages/os_vif/__init__.py:76
  2016-09-15 01:16:57.515 25573 DEBUG oslo.privsep.daemon [-] privsep: request[140021949493072]: (3, 'vif_plug_ovs.linux_net.ensure_bridge', (u'qbr8dfdfd9b-da',), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.515 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl addbr qbr8dfdfd9b-da out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.517 25573 DEBUG neutronclient.v2_0.client [req-6754757c-066c-488a-bc16-6bd451c28cdc tempest-ServerActionsTestJSON-1197364963 tempest-ServerActionsTestJSON-1197364963] GET call to neutron for http://127.0.0.1:9696/v2.0/subnets.json?id=2b899e3c-17dc-478a-bd39-91132bb057ab used request id req-e82dd927-a0f8-48ba-bbbc-56724a10a29d _append_request_id /usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:127
  2016-09-15 01:16:57.521 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl addbr qbr8dfdfd9b-da" returned: 0 in 0.005s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.521 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl setfd qbr8dfdfd9b-da 0 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.524 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl setfd qbr8dfdfd9b-da 0" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.524 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl stp qbr8dfdfd9b-da off out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.528 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl stp qbr8dfdfd9b-da off" returned: 0 in 0.003s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.528 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): tee /sys/class/net/qbr8dfdfd9b-da/bridge/multicast_snooping out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.532 25573 DEBUG oslo.privsep.daemon [-] CMD "tee /sys/class/net/qbr8dfdfd9b-da/bridge/multicast_snooping" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.532 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): tee /proc/sys/net/ipv6/conf/qbr8dfdfd9b-da/disable_ipv6 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] CMD "tee /proc/sys/net/ipv6/conf/qbr8dfdfd9b-da/disable_ipv6" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] privsep: reply[140021949493072]: (4, None) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] privsep: request[140021949493072]: (3, 'vif_plug_ovs.linux_net.create_veth_pair', ('qvb8dfdfd9b-da', 'qvo8dfdfd9b-da', 1500), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.538 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link add qvb8dfdfd9b-da type veth peer name qvo8dfdfd9b-da out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.546 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link add qvb8dfdfd9b-da type veth peer name qvo8dfdfd9b-da" returned: 0 in 0.007s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.547 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da up out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.550 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da up" returned: 0 in 0.003s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.550 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da promisc on out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.556 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da promisc on" returned: 0 in 0.006s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.556 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da mtu 1500 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
  2016-09-15 01:16:57.560 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da mtu 1500" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1623876/+subscriptions
Follow ups

[Bug 1623876] Re: nova is not setting the MTU provided by Neutron
From: OpenStack Infra, 2016-09-16
[Bug 1623876] Re: nova is not setting the MTU provided by Neutron
From: OpenStack Infra, 2016-09-16