← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1266592] Re: unable to ping virtual instance with quantum+ovs plugin

 

[Expired for neutron because there has been no activity for 60 days.]

** Changed in: neutron
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1266592

Title:
  unable to ping virtual instance with quantum+ovs plugin

Status in OpenStack Neutron (virtual network service):
  Expired

Bug description:
  Hi,
  I am using Grizzly on Ubuntu 12.04 with three node setup : Controller, Compute (qemu hypervisor), Network node (quantum with OpenvSwitch plugin). The network node is setup as directed by http://docs.openstack.org/grizzly/openstack-network/admin/content/ch_install.html.
  I am able to launch virtual instance. The virtual instance get created with status 'active'. When I console into VM, I see it doesnot have the IP assigned (unless DHCP agent is installed on compute node). 

  All three nodes are on Ubuntu 12.04 with same kernel version. 
   3.5.0-23-generic #35~precise1-Ubuntu 

  Openvswitch version - ovs-vsctl (Open vSwitch) 1.4.0+build0

  Compute Node openvswitch agent log

  2014-01-06 13:38:03    ERROR [quantum.openstack.common.rpc.common] Failed to consume message from queue: Socket closed
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 563, in ensure
      return method(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 643, in _consume
      return self.connection.drain_events(timeout=timeout)
    File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
      return self.transport.drain_events(self.connection, **kwargs)
    File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
      return connection.drain_events(**kwargs)
    File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
      return self.wait_multi(self.channels.values(), timeout=timeout)
    File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
      chanmap.keys(), allowed_methods, timeout=timeout)
    File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
      channel, method_sig, args, content = read_timeout(timeout)
    File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 88, in read_timeout
      return self.method_reader.read_method()
    File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
      raise m
  IOError: Socket closed
  2014-01-06 13:38:03    ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
  2014-01-06 13:38:28    ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-vsctl', '--timeout=2', '--', '--if-exists', 'del-port', 'br-int', 'patch-tun']. Exception:
  Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--if-exists', 'del-port', 'br-int', 'patch-tun']
  Exit code: 242
  Stdout: ''
  Stderr: 'Jan 06 13:38:26|00002|stream_unix|ERR|/tmp/stream-unix.1565.0: connection to /var/run/openvswitch/db.sock failed: No such file or directory\nJan 06 13:38:26|00003|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\nJan 06 13:38:27|00004|stream_unix|ERR|/tmp/stream-unix.1565.1: connection to /var/run/openvswitch/db.sock failed: No such file or directory\nJan 06 13:38:27|00005|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\n'
  2014-01-06 13:38:28    ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-ofctl', 'del-flows', 'br-int']. Exception:
  Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'del-flows', 'br-int']
  Exit code: 1
  Stdout: ''
  Stderr: 'ovs-ofctl: br-int is not a bridge or a socket\n'
  2014-01-06 13:38:28    ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0,idle_timeout=0,priority=1,actions=normal']. Exception:
  Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0,idle_timeout=0,priority=1,actions=normal']
  Exit code: 1
  Stdout: ''
  Stderr: 'ovs-ofctl: br-int is not a bridge or a socket\n'
  2014-01-06 13:38:30    ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
  2014-01-06 13:38:31    ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
  2014-01-06 13:38:34    ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
  2014-01-06 13:44:53    ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Cannot provision GRE network for net-id=419f8275-b688-4fa5-83dc-03221de1191a - tunneling disabled

  
  Openvswitch service is running on both compute & network node
  ovsdb-server is running with pid 1764
  ovs-vswitchd is running with pid 1773


  quantum agent-list output shows
  +--------------------------------------+--------------------+-------------------------------+-------+----------------+
  | id                                   | agent_type         | host                          | alive | admin_state_up |
  +--------------------------------------+--------------------+-------------------------------+-------+----------------+
  | 972363a1-1945-405c-b2ae-e96c98fa803a | DHCP agent         | os-networknode.eng.vmware.com | :-)   | True           |
  | 97a29293-e1a8-4246-9997-ad60c7e21102 | Open vSwitch agent | os-networknode.eng.vmware.com | :-)   | True           |
  | 99988262-23b6-48af-9301-4e5497832c44 | Open vSwitch agent | oscompute-kvm.eng.vmware.com  | :-)   | True           |
  | d71c2b1b-dd16-4670-9a39-5d84bcfa39f4 | L3 agent           | os-networknode.eng.vmware.com | :-)   | True           |

  
  ovs-vsctl output on compute node

  598ceaf1-6a07-4f9e-930c-5ab333d358e3
      Bridge br-tun
          Port br-tun
              Interface br-tun
                  type: internal
      Bridge br-int
          Port br-int
              Interface br-int
                  type: internal
          Port "qvo783946bb-65"
              tag: 1
              Interface "qvo783946bb-65"
      ovs_version: "1.4.0+build0"

  ovs-vsctl output on network node

  33c3075d-8208-42c9-bc53-e0a62317ef46
      Bridge br-tun
          Port br-tun
              Interface br-tun
                  type: internal
          Port patch-int
              Interface patch-int
                  type: patch
                  options: {peer=patch-tun}
      Bridge br-int
          Port patch-tun
              Interface patch-tun
                  type: patch
                  options: {peer=patch-int}
          Port "tap6e0602ce-b6"
              tag: 1
              Interface "tap6e0602ce-b6"
          Port br-int
              Interface br-int
                  type: internal
      Bridge br-ex
          Port "eth1"
              Interface "eth1"
          Port br-ex
              Interface br-ex
                  type: internal
      ovs_version: "1.4.0+build0"

  
  Endpoints in quantum db
  mysql> select * from ovs_tunnel_endpoints;
  +-----------------+----+
  | ip_address      | id |
  +-----------------+----+
  | 192.168.100.190 |  1 |
  +-----------------+----+
  1 row in set (0.00 sec)

  I am seeing GRE tunneling issues in openvswitch agent log. Any
  pointers as to whats causing the problem ? Any help would be highly
  appreciated.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1266592/+subscriptions


References