yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1264408] Re: Network Node kernel panics when booting nova instance

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mark McClain <1264408@xxxxxxxxxxxxxxxxxx>
Date: Wed, 05 Mar 2014 18:19:15 -0000
Reply-to: Bug 1264408 <1264408@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: New => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1264408

Title:
  Network Node kernel panics when booting nova instance

Status in OpenStack Neutron (virtual network service):
  Won't Fix

Bug description:
  Hi, I am using OpenStack Grizzly release on Ubuntu Server 12.04 with three node setup Controller, Network (with OpenvSwtich OVS plugin) & Compute (with qemu hypervisor). Network node is setup as outlined in http://docs.openstack.org/grizzly/openstack-network/admin/content/.  When I instantiate nova instance my network node panics showing traces of openvswitch ovs_tnl. Refer attached screenshot "kernel-panic" for more info. The nova instance gets created with active status (but unable to login). 
  The network node recovers from panic only after quantum network is deleted.

  ovsdb-server.log shows :

  Dec 26 14:03:45|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
  Dec 26 14:03:45|00002|ovsdb_file|ERR|I/O error: /etc/openvswitch/conf.db: error reading 254 bytes starting at offset 217078 (End of file)
  Dec 26 14:07:09|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
  Dec 26 14:07:09|00002|ovsdb_file|ERR|I/O error: /etc/openvswitch/conf.db: error reading 548 bytes starting at offset 217078 (End of file)
  Dec 26 14:38:48|00002|daemon|INFO|pid 1494 died, killed (Terminated), exiting
  Dec 26 14:39:10|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
  Dec 26 15:19:57|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
  Dec 26 15:19:57|00002|ovsdb_file|ERR|I/O error: /etc/openvswitch/conf.db: error reading 93 bytes starting at offset 233449 (End of file)
  Dec 26 15:23:10|00002|daemon|INFO|pid 1515 died, killed (Terminated), exiting
  Dec 26 15:31:25|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
  Dec 26 16:02:59|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
  Dec 26 16:02:59|00002|ovsdb_file|ERR|I/O error: /etc/openvswitch/conf.db: error reading 982 bytes starting at offset 23995 (End of file)
  Dec 26 16:17:26|00002|daemon|INFO|pid 1445 died, killed (Terminated), exiting
  Dec 26 16:17:42|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log
  Dec 26 16:47:04|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log

  
  ovs-vswitchd.log shows :

  Dec 26 16:47:04|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
  Dec 26 16:47:04|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
  Dec 26 16:47:04|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
  Dec 26 16:47:04|00004|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:04|00005|bridge|INFO|created port br-int on bridge br-int
  Dec 26 16:47:04|00006|bridge|INFO|created port patch-tun on bridge br-int
  Dec 26 16:47:04|00007|bridge|INFO|created port patch-int on bridge br-tun
  Dec 26 16:47:04|00008|bridge|INFO|created port gre-1 on bridge br-tun
  Dec 26 16:47:04|00009|bridge|INFO|created port br-tun on bridge br-tun
  Dec 26 16:47:04|00010|bridge|INFO|created port eth1 on bridge br-ex
  Dec 26 16:47:04|00011|bridge|INFO|created port br-ex on bridge br-ex
  Dec 26 16:47:04|00012|ofproto|INFO|using datapath ID 0000002320b0e672
  Dec 26 16:47:04|00013|ofproto|INFO|using datapath ID 0000002320acd925
  Dec 26 16:47:04|00014|ofproto|INFO|using datapath ID 0000002320d7ea10
  Dec 26 16:47:04|00015|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:04|00016|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:04|00017|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:04|00018|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:04|00019|xenserver|INFO|not running on a XenServer
  Dec 26 16:47:04|00020|ofproto|INFO|datapath ID changed to 0000068c873e694a
  Dec 26 16:47:04|00021|bridge|WARN|bridge br-tun: using default bridge Ethernet address c6:a4:7b:6e:63:4d
  Dec 26 16:47:04|00022|ofproto|INFO|datapath ID changed to 0000c6a47b6e634d
  Dec 26 16:47:04|00023|ofproto|INFO|datapath ID changed to 000000505683b2d3
  Dec 26 16:47:04|00024|bridge|INFO|destroyed port patch-int on bridge br-tun
  Dec 26 16:47:04|00025|bridge|INFO|destroyed port gre-1 on bridge br-tun
  Dec 26 16:47:04|00026|bridge|INFO|destroyed port br-tun on bridge br-tun
  Dec 26 16:47:04|00027|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:04|00028|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:04|00029|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:04|00030|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:04|00031|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:04|00032|bridge|INFO|created port br-tun on bridge br-tun
  Dec 26 16:47:04|00033|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:04|00034|ofproto|INFO|using datapath ID 0000002320d6eb85
  Dec 26 16:47:04|00035|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:04|00036|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:04|00037|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:04|00038|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:04|00039|ofproto|INFO|datapath ID changed to 00007ae99ad97f43
  Dec 26 16:47:04|00040|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:05|00041|bridge|INFO|created port patch-int on bridge br-tun
  Dec 26 16:47:05|00042|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00043|bridge|WARN|could not open network device patch-int (No such device)
  Dec 26 16:47:05|00044|bridge|WARN|patch-int port has no interfaces, dropping
  Dec 26 16:47:05|00045|bridge|INFO|destroyed port patch-int on bridge br-tun
  Dec 26 16:47:05|00046|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:05|00047|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:05|00048|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00049|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:05|00050|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:05|00051|bridge|INFO|created port patch-int on bridge br-tun
  Dec 26 16:47:05|00052|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00053|netdev_vport|ERR|patch-int: patch type requires valid 'peer' argument
  Dec 26 16:47:05|00054|bridge|WARN|could not configure network device patch-int (Invalid argument)
  Dec 26 16:47:05|00055|bridge|WARN|patch-int port has no interfaces, dropping
  Dec 26 16:47:05|00056|bridge|INFO|destroyed port patch-int on bridge br-tun
  Dec 26 16:47:05|00057|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:05|00058|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:05|00059|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00060|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:05|00061|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:05|00062|bridge|INFO|created port patch-int on bridge br-tun
  Dec 26 16:47:05|00063|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00064|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:05|00065|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:05|00066|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00067|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:05|00068|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:05|00069|bridge|INFO|created port gre-1 on bridge br-tun
  Dec 26 16:47:05|00070|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00071|bridge|WARN|could not open network device gre-1 (No such device)
  Dec 26 16:47:05|00072|bridge|WARN|gre-1 port has no interfaces, dropping
  Dec 26 16:47:05|00073|bridge|INFO|destroyed port gre-1 on bridge br-tun
  Dec 26 16:47:05|00074|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:05|00075|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:05|00076|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00077|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:05|00078|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:05|00079|bridge|INFO|created port gre-1 on bridge br-tun
  Dec 26 16:47:05|00080|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00081|netdev_vport|ERR|gre-1: gre type requires valid 'remote_ip' argument
  Dec 26 16:47:05|00082|bridge|WARN|could not configure network device gre-1 (Invalid argument)
  Dec 26 16:47:05|00083|bridge|WARN|gre-1 port has no interfaces, dropping
  Dec 26 16:47:05|00084|bridge|INFO|destroyed port gre-1 on bridge br-tun
  Dec 26 16:47:05|00085|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:05|00086|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:05|00087|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00088|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:05|00089|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:05|00090|bridge|INFO|created port gre-1 on bridge br-tun
  Dec 26 16:47:05|00091|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00092|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:05|00093|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:05|00094|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00095|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:05|00096|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:05|00097|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00098|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:05|00099|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:05|00100|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:05|00101|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:05|00102|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a
  Dec 26 16:47:06|00103|bridge|INFO|created port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:06|00104|bridge|WARN|could not open network device tap13a6499d-a7 (No such device)
  Dec 26 16:47:06|00105|bridge|WARN|tap13a6499d-a7 port has no interfaces, dropping
  Dec 26 16:47:06|00106|bridge|INFO|destroyed port tap13a6499d-a7 on bridge br-int
  Dec 26 16:47:06|00107|bridge|WARN|bridge br-tun: using default bridge Ethernet address 7a:e9:9a:d9:7f:43
  Dec 26 16:47:06|00108|bridge|WARN|bridge br-int: using default bridge Ethernet address 06:8c:87:3e:69:4a

  
  openvswitch-agent.log shows :

  
  2013-12-26 16:47:04    ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-vsctl', '--timeout=2', '--', '--if-exists', 'del-port', 'br-int', 'patch-tun']. Exception:
  Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--if-exists', 'del-port', 'br-int', 'patch-tun']
  Exit code: 242
  Stdout: ''
  Stderr: 'Dec 26 16:47:02|00002|stream_unix|ERR|/tmp/stream-unix.1290.0: connection to /var/run/openvswitch/db.sock failed: No such file or directory\nDec 26 16:47:02|00003|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\nDec 26 16:47:03|00004|stream_unix|ERR|/tmp/stream-unix.1290.1: connection to /var/run/openvswitch/db.sock failed: No such file or directory\nDec 26 16:47:03|00005|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\n'
  2013-12-26 16:47:05    ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-vsctl', '--timeout=2', 'add-port', 'br-int', 'patch-tun']. Exception:
  Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'add-port', 'br-int', 'patch-tun']
  Exit code: 1
  Stdout: ''
  Stderr: 'ovs-vsctl: cannot create a port named patch-tun because a port named patch-tun already exists on bridge br-int\n'

  
  Output of ovs-vsctl show 

  
  33c3075d-8208-42c9-bc53-e0a62317ef46
      Bridge br-int
          Port patch-tun
              Interface patch-tun
                  type: patch
                  options: {peer=patch-int}
          Port "tap13a6499d-a7"
              tag: 1
              Interface "tap13a6499d-a7"
          Port br-int
              Interface br-int
                  type: internal
      Bridge br-ex
          Port br-ex
              Interface br-ex
                  type: internal
          Port "eth1"
              Interface "eth1"
      Bridge br-tun
          Port patch-int
              Interface patch-int
                  type: patch
                  options: {peer=patch-tun}
          Port "gre-1"
              Interface "gre-1"
                  type: gre
                  options: {in_key=flow, out_key=flow, remote_ip="192.168.100.180"}
          Port br-tun
              Interface br-tun
                  type: internal
      ovs_version: "1.4.0+build0"

  I am unsure as to what would be causing the panic. So any help would
  be highly appreciated.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1264408/+subscriptions
References

[Bug 1264408] [NEW] Network Node kernel panics when booting nova instance
From: Nitz, 2013-12-27