yahoo-eng-team team mailing list archive

[Victor Tapia <victor.tapia@xxxxxxxxxxxxx>]

Public bug reported:

For some reason, some compute nodes are missing ovs flows of various
tenants (not all of them), resulting in vm isolation (no dhcp/metadata
on boot). A particular tenant "A" might have issues with node "B" whilst
tenant "B" might have problems with node "A" and not "B". All of the
affected tenant VMs running in an affected node are isolated.

In those compute nodes, the ovs-vswitchd process has crashed previously:
ubuntu@niagara:~$ ps -ef | grep vswitchd
root      1959     1  0 Feb12 ?        00:41:15 ovs-vswitchd: monitoring pid 62005 (4 crashes: pid 59408 died, killed (Segmentation fault), core dumped)

After restarting the openvswitch-switch service, ps shows that the missing flows are being created by neutron (e.g. sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovs-ofctl mod-flows br-tun table=21,dl_vlan=11,actions=strip_vlan,set_tunnel:4,output:5,6,2,4,7,3).
--- 
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
DistroRelease: Ubuntu 14.04
Package: neutron-common 1:2014.1.3-0ubuntu1
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: User Name 3.13.0-45.74-generic 3.13.11-ckt13
Tags:  trusty uec-images
Uname: Linux 3.13.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy libvirtd netdev plugdev sudo video
_MarkForUpload: True
modified.conffile..etc.neutron.api.paste.ini: [deleted]
modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
modified.conffile..etc.neutron.l3.agent.ini: [deleted]
modified.conffile..etc.neutron.neutron.conf: [deleted]
modified.conffile..etc.neutron.policy.json: [deleted]
modified.conffile..etc.neutron.rootwrap.conf: [deleted]
modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: cts

** Tags added: apport-collected trusty uec-images

** Description changed:

  For some reason, some compute nodes are missing ovs flows of various
  tenants (not all of them), resulting in vm isolation (no dhcp/metadata
  on boot). A particular tenant "A" might have issues with node "B" whilst
  tenant "B" might have problems with node "A" and not "B". All of the
  affected tenant VMs running in an affected node are isolated.
  
  In those compute nodes, the ovs-vswitchd process has crashed previously:
  ubuntu@niagara:~$ ps -ef | grep vswitchd
  root      1959     1  0 Feb12 ?        00:41:15 ovs-vswitchd: monitoring pid 62005 (4 crashes: pid 59408 died, killed (Segmentation fault), core dumped)
  
- After restarting the openvswitch-switch service, ps shows that the
- missing flows are being created by neutron (e.g. sudo neutron-rootwrap
- /etc/neutron/rootwrap.conf ovs-ofctl mod-flows br-tun
- table=21,dl_vlan=11,actions=strip_vlan,set_tunnel:4,output:5,6,2,4,7,3).
+ After restarting the openvswitch-switch service, ps shows that the missing flows are being created by neutron (e.g. sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovs-ofctl mod-flows br-tun table=21,dl_vlan=11,actions=strip_vlan,set_tunnel:4,output:5,6,2,4,7,3).
+ --- 
+ ApportVersion: 2.14.1-0ubuntu3.5
+ Architecture: amd64
+ DistroRelease: Ubuntu 14.04
+ Package: neutron-common 1:2014.1.3-0ubuntu1
+ PackageArchitecture: all
+ ProcEnviron:
+  TERM=xterm
+  PATH=(custom, no user)
+  XDG_RUNTIME_DIR=<set>
+  LANG=en_US.UTF-8
+  SHELL=/bin/bash
+ ProcVersionSignature: User Name 3.13.0-45.74-generic 3.13.11-ckt13
+ Tags:  trusty uec-images
+ Uname: Linux 3.13.0-45-generic x86_64
+ UpgradeStatus: No upgrade log present (probably fresh install)
+ UserGroups: adm audio cdrom dialout dip floppy libvirtd netdev plugdev sudo video
+ _MarkForUpload: True
+ modified.conffile..etc.neutron.api.paste.ini: [deleted]
+ modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
+ modified.conffile..etc.neutron.l3.agent.ini: [deleted]
+ modified.conffile..etc.neutron.neutron.conf: [deleted]
+ modified.conffile..etc.neutron.policy.json: [deleted]
+ modified.conffile..etc.neutron.rootwrap.conf: [deleted]
+ modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
+ modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
+ modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
+ modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
+ modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
+ modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1431958

Title:
  Missing ovs flows results in vm isolation for particular tenants

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  For some reason, some compute nodes are missing ovs flows of various
  tenants (not all of them), resulting in vm isolation (no dhcp/metadata
  on boot). A particular tenant "A" might have issues with node "B"
  whilst tenant "B" might have problems with node "A" and not "B". All
  of the affected tenant VMs running in an affected node are isolated.

  In those compute nodes, the ovs-vswitchd process has crashed previously:
  ubuntu@niagara:~$ ps -ef | grep vswitchd
  root      1959     1  0 Feb12 ?        00:41:15 ovs-vswitchd: monitoring pid 62005 (4 crashes: pid 59408 died, killed (Segmentation fault), core dumped)

  After restarting the openvswitch-switch service, ps shows that the missing flows are being created by neutron (e.g. sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovs-ofctl mod-flows br-tun table=21,dl_vlan=11,actions=strip_vlan,set_tunnel:4,output:5,6,2,4,7,3).
  --- 
  ApportVersion: 2.14.1-0ubuntu3.5
  Architecture: amd64
  DistroRelease: Ubuntu 14.04
  Package: neutron-common 1:2014.1.3-0ubuntu1
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcVersionSignature: User Name 3.13.0-45.74-generic 3.13.11-ckt13
  Tags:  trusty uec-images
  Uname: Linux 3.13.0-45-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: adm audio cdrom dialout dip floppy libvirtd netdev plugdev sudo video
  _MarkForUpload: True
  modified.conffile..etc.neutron.api.paste.ini: [deleted]
  modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
  modified.conffile..etc.neutron.l3.agent.ini: [deleted]
  modified.conffile..etc.neutron.neutron.conf: [deleted]
  modified.conffile..etc.neutron.policy.json: [deleted]
  modified.conffile..etc.neutron.rootwrap.conf: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
  modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
  modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1431958/+subscriptions