← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #08214

[Bug 1268955] [NEW] OVS agent updates the wrong port when using Xen + Neutron

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Environment
==========
- Xen Server 6.2
- OpenStack Havana installed with Packstack
- Neutron OVS agent using VLAN

>From time to time, when an instance is started, it fails to get network
connectivity. As a result the instance cannot get its IP address from
DHCP and it remains unreachable.

After further investigation, it appears that the OVS agent running on
the compute node is updating the wrong OVS port because on startup, 2
ports exist for the same instance: vifX.0 and tapX.0. The agent updates
whatever port is returned in first position (see logs below). Note that
the tapX.0 port is only transient and disappears after a few seconds.

Workaround
==========

Manually update the OVS port on dom0:

$ ovs-vsctl set Port vif17.0 tag=1

OVS Agent logs
============

2014-01-14 14:15:11.382 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43
2014-01-14 14:15:11.403 18268 DEBUG qpid.messaging.io.raw [-] SENT[3350c68]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x81' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480
2014-01-14 14:15:11.649 18268 DEBUG neutron.agent.linux.utils [-]
Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"']
Exit code: 0
Stdout: 'external_ids        : {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=active, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname                : "tap29.0"\nofport              : 52\n\nexternal_ids        : {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=inactive, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname                : "vif29.0"\nofport              : 51\n\n'
Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60
2014-01-14 14:15:11.650 18268 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port 98679ab6-b879-4b1b-a524-01696959d468 updated. Details: {u'admin_state_up': True, u'network_id': u'ad37f107-074b-4c58-8f36-4705533afb8d', u'segmentation_id': 100, u'physical_network': u'default', u'device': u'98679ab6-b879-4b1b-a524-01696959d468', u'port_id': u'98679ab6-b879-4b1b-a524-01696959d468', u'network_type': u'vlan'}
2014-01-14 14:15:11.650 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43
2014-01-14 14:15:11.913 18268 DEBUG neutron.agent.linux.utils [-]
Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1']
Exit code: 0
Stdout: '\n'
Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60

** Affects: neutron
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: xenserver

** Also affects: nova
   Importance: Undecided
       Status: New

** Tags added: xenserver

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1268955

Title:
  OVS agent updates the wrong port when using Xen + Neutron

Status in OpenStack Neutron (virtual network service):
  New
Status in OpenStack Compute (Nova):
  New

Bug description:
  Environment
  ==========
  - Xen Server 6.2
  - OpenStack Havana installed with Packstack
  - Neutron OVS agent using VLAN

  From time to time, when an instance is started, it fails to get
  network connectivity. As a result the instance cannot get its IP
  address from DHCP and it remains unreachable.

  After further investigation, it appears that the OVS agent running on
  the compute node is updating the wrong OVS port because on startup, 2
  ports exist for the same instance: vifX.0 and tapX.0. The agent
  updates whatever port is returned in first position (see logs below).
  Note that the tapX.0 port is only transient and disappears after a few
  seconds.

  Workaround
  ==========

  Manually update the OVS port on dom0:

  $ ovs-vsctl set Port vif17.0 tag=1

  OVS Agent logs
  ============

  2014-01-14 14:15:11.382 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43
  2014-01-14 14:15:11.403 18268 DEBUG qpid.messaging.io.raw [-] SENT[3350c68]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x81' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480
  2014-01-14 14:15:11.649 18268 DEBUG neutron.agent.linux.utils [-]
  Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"']
  Exit code: 0
  Stdout: 'external_ids        : {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=active, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname                : "tap29.0"\nofport              : 52\n\nexternal_ids        : {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=inactive, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname                : "vif29.0"\nofport              : 51\n\n'
  Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60
  2014-01-14 14:15:11.650 18268 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port 98679ab6-b879-4b1b-a524-01696959d468 updated. Details: {u'admin_state_up': True, u'network_id': u'ad37f107-074b-4c58-8f36-4705533afb8d', u'segmentation_id': 100, u'physical_network': u'default', u'device': u'98679ab6-b879-4b1b-a524-01696959d468', u'port_id': u'98679ab6-b879-4b1b-a524-01696959d468', u'network_type': u'vlan'}
  2014-01-14 14:15:11.650 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43
  2014-01-14 14:15:11.913 18268 DEBUG neutron.agent.linux.utils [-]
  Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1']
  Exit code: 0
  Stdout: '\n'
  Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1268955/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next