yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #07972
[Bug 1266592] [NEW] unable to ping virtual instance with quantum+ovs plugin
Public bug reported:
Hi,
I am using Grizzly on Ubuntu 12.04 with three node setup : Controller, Compute (qemu hypervisor), Network node (quantum with OpenvSwitch plugin). The network node is setup as directed by http://docs.openstack.org/grizzly/openstack-network/admin/content/ch_install.html.
I am able to launch virtual instance. The virtual instance get created with status 'active'. When I console into VM, I see it doesnot have the IP assigned (unless DHCP agent is installed on compute node).
All three nodes are on Ubuntu 12.04 with same kernel version.
3.5.0-23-generic #35~precise1-Ubuntu
Openvswitch version - ovs-vsctl (Open vSwitch) 1.4.0+build0
Compute Node openvswitch agent log
2014-01-06 13:38:03 ERROR [quantum.openstack.common.rpc.common] Failed to consume message from queue: Socket closed
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 563, in ensure
return method(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 643, in _consume
return self.connection.drain_events(timeout=timeout)
File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
return connection.drain_events(**kwargs)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
return self.wait_multi(self.channels.values(), timeout=timeout)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
chanmap.keys(), allowed_methods, timeout=timeout)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
channel, method_sig, args, content = read_timeout(timeout)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 88, in read_timeout
return self.method_reader.read_method()
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
raise m
IOError: Socket closed
2014-01-06 13:38:03 ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-01-06 13:38:28 ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-vsctl', '--timeout=2', '--', '--if-exists', 'del-port', 'br-int', 'patch-tun']. Exception:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--if-exists', 'del-port', 'br-int', 'patch-tun']
Exit code: 242
Stdout: ''
Stderr: 'Jan 06 13:38:26|00002|stream_unix|ERR|/tmp/stream-unix.1565.0: connection to /var/run/openvswitch/db.sock failed: No such file or directory\nJan 06 13:38:26|00003|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\nJan 06 13:38:27|00004|stream_unix|ERR|/tmp/stream-unix.1565.1: connection to /var/run/openvswitch/db.sock failed: No such file or directory\nJan 06 13:38:27|00005|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\n'
2014-01-06 13:38:28 ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-ofctl', 'del-flows', 'br-int']. Exception:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'del-flows', 'br-int']
Exit code: 1
Stdout: ''
Stderr: 'ovs-ofctl: br-int is not a bridge or a socket\n'
2014-01-06 13:38:28 ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0,idle_timeout=0,priority=1,actions=normal']. Exception:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0,idle_timeout=0,priority=1,actions=normal']
Exit code: 1
Stdout: ''
Stderr: 'ovs-ofctl: br-int is not a bridge or a socket\n'
2014-01-06 13:38:30 ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-01-06 13:38:31 ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2014-01-06 13:38:34 ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2014-01-06 13:44:53 ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Cannot provision GRE network for net-id=419f8275-b688-4fa5-83dc-03221de1191a - tunneling disabled
Openvswitch service is running on both compute & network node
ovsdb-server is running with pid 1764
ovs-vswitchd is running with pid 1773
quantum agent-list output shows
+--------------------------------------+--------------------+-------------------------------+-------+----------------+
| id | agent_type | host | alive | admin_state_up |
+--------------------------------------+--------------------+-------------------------------+-------+----------------+
| 972363a1-1945-405c-b2ae-e96c98fa803a | DHCP agent | os-networknode.eng.vmware.com | :-) | True |
| 97a29293-e1a8-4246-9997-ad60c7e21102 | Open vSwitch agent | os-networknode.eng.vmware.com | :-) | True |
| 99988262-23b6-48af-9301-4e5497832c44 | Open vSwitch agent | oscompute-kvm.eng.vmware.com | :-) | True |
| d71c2b1b-dd16-4670-9a39-5d84bcfa39f4 | L3 agent | os-networknode.eng.vmware.com | :-) | True |
ovs-vsctl output on compute node
598ceaf1-6a07-4f9e-930c-5ab333d358e3
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Bridge br-int
Port br-int
Interface br-int
type: internal
Port "qvo783946bb-65"
tag: 1
Interface "qvo783946bb-65"
ovs_version: "1.4.0+build0"
ovs-vsctl output on network node
33c3075d-8208-42c9-bc53-e0a62317ef46
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Bridge br-int
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "tap6e0602ce-b6"
tag: 1
Interface "tap6e0602ce-b6"
Port br-int
Interface br-int
type: internal
Bridge br-ex
Port "eth1"
Interface "eth1"
Port br-ex
Interface br-ex
type: internal
ovs_version: "1.4.0+build0"
Endpoints in quantum db
mysql> select * from ovs_tunnel_endpoints;
+-----------------+----+
| ip_address | id |
+-----------------+----+
| 192.168.100.190 | 1 |
+-----------------+----+
1 row in set (0.00 sec)
I am seeing GRE tunneling issues in openvswitch agent log. Any pointers
as to whats causing the problem ? Any help would be highly appreciated.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1266592
Title:
unable to ping virtual instance with quantum+ovs plugin
Status in OpenStack Neutron (virtual network service):
New
Bug description:
Hi,
I am using Grizzly on Ubuntu 12.04 with three node setup : Controller, Compute (qemu hypervisor), Network node (quantum with OpenvSwitch plugin). The network node is setup as directed by http://docs.openstack.org/grizzly/openstack-network/admin/content/ch_install.html.
I am able to launch virtual instance. The virtual instance get created with status 'active'. When I console into VM, I see it doesnot have the IP assigned (unless DHCP agent is installed on compute node).
All three nodes are on Ubuntu 12.04 with same kernel version.
3.5.0-23-generic #35~precise1-Ubuntu
Openvswitch version - ovs-vsctl (Open vSwitch) 1.4.0+build0
Compute Node openvswitch agent log
2014-01-06 13:38:03 ERROR [quantum.openstack.common.rpc.common] Failed to consume message from queue: Socket closed
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 563, in ensure
return method(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 643, in _consume
return self.connection.drain_events(timeout=timeout)
File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
return connection.drain_events(**kwargs)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
return self.wait_multi(self.channels.values(), timeout=timeout)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
chanmap.keys(), allowed_methods, timeout=timeout)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
channel, method_sig, args, content = read_timeout(timeout)
File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 88, in read_timeout
return self.method_reader.read_method()
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
raise m
IOError: Socket closed
2014-01-06 13:38:03 ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-01-06 13:38:28 ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-vsctl', '--timeout=2', '--', '--if-exists', 'del-port', 'br-int', 'patch-tun']. Exception:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--if-exists', 'del-port', 'br-int', 'patch-tun']
Exit code: 242
Stdout: ''
Stderr: 'Jan 06 13:38:26|00002|stream_unix|ERR|/tmp/stream-unix.1565.0: connection to /var/run/openvswitch/db.sock failed: No such file or directory\nJan 06 13:38:26|00003|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\nJan 06 13:38:27|00004|stream_unix|ERR|/tmp/stream-unix.1565.1: connection to /var/run/openvswitch/db.sock failed: No such file or directory\nJan 06 13:38:27|00005|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\n'
2014-01-06 13:38:28 ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-ofctl', 'del-flows', 'br-int']. Exception:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'del-flows', 'br-int']
Exit code: 1
Stdout: ''
Stderr: 'ovs-ofctl: br-int is not a bridge or a socket\n'
2014-01-06 13:38:28 ERROR [quantum.agent.linux.ovs_lib] Unable to execute ['ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0,idle_timeout=0,priority=1,actions=normal']. Exception:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0,idle_timeout=0,priority=1,actions=normal']
Exit code: 1
Stdout: ''
Stderr: 'ovs-ofctl: br-int is not a bridge or a socket\n'
2014-01-06 13:38:30 ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-01-06 13:38:31 ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2014-01-06 13:38:34 ERROR [quantum.openstack.common.rpc.common] AMQP server on 10.115.200.171:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2014-01-06 13:44:53 ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Cannot provision GRE network for net-id=419f8275-b688-4fa5-83dc-03221de1191a - tunneling disabled
Openvswitch service is running on both compute & network node
ovsdb-server is running with pid 1764
ovs-vswitchd is running with pid 1773
quantum agent-list output shows
+--------------------------------------+--------------------+-------------------------------+-------+----------------+
| id | agent_type | host | alive | admin_state_up |
+--------------------------------------+--------------------+-------------------------------+-------+----------------+
| 972363a1-1945-405c-b2ae-e96c98fa803a | DHCP agent | os-networknode.eng.vmware.com | :-) | True |
| 97a29293-e1a8-4246-9997-ad60c7e21102 | Open vSwitch agent | os-networknode.eng.vmware.com | :-) | True |
| 99988262-23b6-48af-9301-4e5497832c44 | Open vSwitch agent | oscompute-kvm.eng.vmware.com | :-) | True |
| d71c2b1b-dd16-4670-9a39-5d84bcfa39f4 | L3 agent | os-networknode.eng.vmware.com | :-) | True |
ovs-vsctl output on compute node
598ceaf1-6a07-4f9e-930c-5ab333d358e3
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Bridge br-int
Port br-int
Interface br-int
type: internal
Port "qvo783946bb-65"
tag: 1
Interface "qvo783946bb-65"
ovs_version: "1.4.0+build0"
ovs-vsctl output on network node
33c3075d-8208-42c9-bc53-e0a62317ef46
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Bridge br-int
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "tap6e0602ce-b6"
tag: 1
Interface "tap6e0602ce-b6"
Port br-int
Interface br-int
type: internal
Bridge br-ex
Port "eth1"
Interface "eth1"
Port br-ex
Interface br-ex
type: internal
ovs_version: "1.4.0+build0"
Endpoints in quantum db
mysql> select * from ovs_tunnel_endpoints;
+-----------------+----+
| ip_address | id |
+-----------------+----+
| 192.168.100.190 | 1 |
+-----------------+----+
1 row in set (0.00 sec)
I am seeing GRE tunneling issues in openvswitch agent log. Any
pointers as to whats causing the problem ? Any help would be highly
appreciated.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1266592/+subscriptions
Follow ups
References
