← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #33636

[Bug 1464178] [NEW] A hostname change breaks neutron-openvswitch-agent / neutron-server tunneling updates.

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

When using tunnelling, if one of the hosts changed the hostname and
tries to sync tunnels to neutron-server, this will throw an exception
due to an unnecessary constraint, breaking the network.

Hostname changes are something neutron-server may survive to. Probably a
log warning is enough, and the old hostname endpoint should be deleted.

This was found in HA deployments with pacemaker, where the hostname is
roamed to the active node, or it's set dynamically on the nodes based on
the clone ID provided by pacemaker, that's used to allow architectures
like A/A/A/P/P for neutron, where one of the active nodes could die, and
a passive takes the resources of the old active by roaming it's hostname
(which is the logical ID where neutron agent resources are tied to).


neutron-server log:

015-06-10 05:44:48.151 24546 ERROR oslo_messaging._drivers.common [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] Returning exception Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}). to caller
2015-06-10 05:44:48.152 24546 ERROR oslo_messaging._drivers.common [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 248, in tunnel_sync\n    raise exc.InvalidInput(error_message=msg)\n', "InvalidInput: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}).\n"]
2015-06-10 05:44:52.152 24546 ERROR oslo_messaging.rpc.dispatcher [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] Exception during message handling: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}).
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 248, in tunnel_sync
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher     raise exc.InvalidInput(error_message=msg)
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher InvalidInput: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}).
2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher 
2015-06-10 05:44:52.152 24546 ERROR oslo_messaging._drivers.common [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] Returning exception Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}). to caller
2015-06-10 05:44:52.153 24546 ERROR oslo_messaging._drivers.common [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 248, in tunnel_sync\n    raise exc.InvalidInput(error_message=msg)\n', "InvalidInput: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}).\n"]

How to reproduce:

1) Install a single node AIO with tunnelling for tenant networks. 
2) openstack-config --set /etc/neutron/neutron.conf DEFAULT host newhostname
3) service neutron-openvswitch-agent restart
4) The exceptions keep happening for neutron-server in a loop as the agent tries to sync the tunnel, and fail.

This new behaviour was introduced in Kilo by this patch:
https://github.com/openstack/neutron/commit/3db0e619c83892a7aab61807969205253833ff8d

** Affects: neutron
     Importance: Undecided
     Assignee: Miguel Angel Ajo (mangelajo)
         Status: Confirmed

** Changed in: neutron
     Assignee: (unassigned) => Miguel Angel Ajo (mangelajo)

** Changed in: neutron
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1464178

Title:
  A hostname change breaks neutron-openvswitch-agent / neutron-server
  tunneling updates.

Status in OpenStack Neutron (virtual network service):
  Confirmed

Bug description:
  When using tunnelling, if one of the hosts changed the hostname and
  tries to sync tunnels to neutron-server, this will throw an exception
  due to an unnecessary constraint, breaking the network.

  Hostname changes are something neutron-server may survive to. Probably
  a log warning is enough, and the old hostname endpoint should be
  deleted.

  This was found in HA deployments with pacemaker, where the hostname is
  roamed to the active node, or it's set dynamically on the nodes based
  on the clone ID provided by pacemaker, that's used to allow
  architectures like A/A/A/P/P for neutron, where one of the active
  nodes could die, and a passive takes the resources of the old active
  by roaming it's hostname (which is the logical ID where neutron agent
  resources are tied to).

  
  neutron-server log:

  015-06-10 05:44:48.151 24546 ERROR oslo_messaging._drivers.common [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] Returning exception Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}). to caller
  2015-06-10 05:44:48.152 24546 ERROR oslo_messaging._drivers.common [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 248, in tunnel_sync\n    raise exc.InvalidInput(error_message=msg)\n', "InvalidInput: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}).\n"]
  2015-06-10 05:44:52.152 24546 ERROR oslo_messaging.rpc.dispatcher [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] Exception during message handling: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}).
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 248, in tunnel_sync
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher     raise exc.InvalidInput(error_message=msg)
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher InvalidInput: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}).
  2015-06-10 05:44:52.152 24546 TRACE oslo_messaging.rpc.dispatcher 
  2015-06-10 05:44:52.152 24546 ERROR oslo_messaging._drivers.common [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] Returning exception Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}). to caller
  2015-06-10 05:44:52.153 24546 ERROR oslo_messaging._drivers.common [req-751f3392-9915-49b9-bb0b-2dec63a6649a ] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 248, in tunnel_sync\n    raise exc.InvalidInput(error_message=msg)\n', "InvalidInput: Invalid input for operation: (u'Tunnel IP %(ip)s in use with host %(host)s', {'ip': u'192.168.16.105', 'host': u'neutron-n-2'}).\n"]

  How to reproduce:

  1) Install a single node AIO with tunnelling for tenant networks. 
  2) openstack-config --set /etc/neutron/neutron.conf DEFAULT host newhostname
  3) service neutron-openvswitch-agent restart
  4) The exceptions keep happening for neutron-server in a loop as the agent tries to sync the tunnel, and fail.

  This new behaviour was introduced in Kilo by this patch:
  https://github.com/openstack/neutron/commit/3db0e619c83892a7aab61807969205253833ff8d

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1464178/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next