← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #21577

[Bug 1370795] [NEW] neutron floatingip-associate on port can cause server exception

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Associating a floating IP address with a port, when it's not itself
associated with an instance, can cause the neutron server to throw an
exception, leaving neutron completely unusable.

Here's how to reproduce it:

1. Start-up devstack, having it clone the latest upstream code, making sure to enable dvr by setting Q_DVR_MODE=dvr_snat
    (this will create a network, subnet, and router and attach it to private and ext-nets)
2. neutron net-list
3. neutron port-create $private_network_id
4. neutron floatingip-create $public_network_id
5. neutron floatingip-associate $floatingip_id $port_id

You'll start seeting this in screen-q-svc.log:

2014-09-17 20:56:17.758 5423 DEBUG neutron.db.l3_dvr_db [req-3faea024-ab6c-46f2-8706-e8b1028616ab None] Floating IP host: None _process_floating_ips /opt/stack/neutron/neutron/db/l3_dvr_db.py:296
2014-09-17 20:56:17.760 5423 ERROR oslo.messaging.rpc.dispatcher [req-3faea024-ab6c-46f2-8706-e8b1028616ab ] Exception during message handling: Agent with agent_type=L3 agent and host=None could not be found
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 78, in sync_routers
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     context, host, router_ids))
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/l3_agentschedulers_db.py", line 299, in list_active_sync_routers_on_active_l3_agent
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     active=True)
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 458, in get_ha_sync_data_for_host
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     active)
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 330, in get_sync_data
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     self._process_floating_ips(context, routers_dict, floating_ips)
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 299, in _process_floating_ips
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     floating_ip['host'])
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 157, in _get_agent_by_type_and_host
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     host=host)
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=None could not be found
2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher
2014-09-17 20:56:17.768 5423 ERROR oslo.messaging._drivers.common [req-3faea024-ab6c-46f2-8706-e8b1028616ab ] Returning exception Agent with agent_type=L3 agent and host=None could not be found to caller

And it will just keep repeating as the l3-agent retries the call.

The result is the l3-agent won't be able to do any work.

I have a fix I'll send out for review.

** Affects: neutron
     Importance: Undecided
     Assignee: Brian Haley (brian-haley)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Brian Haley (brian-haley)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1370795

Title:
  neutron floatingip-associate on port can cause server exception

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Associating a floating IP address with a port, when it's not itself
  associated with an instance, can cause the neutron server to throw an
  exception, leaving neutron completely unusable.

  Here's how to reproduce it:

  1. Start-up devstack, having it clone the latest upstream code, making sure to enable dvr by setting Q_DVR_MODE=dvr_snat
      (this will create a network, subnet, and router and attach it to private and ext-nets)
  2. neutron net-list
  3. neutron port-create $private_network_id
  4. neutron floatingip-create $public_network_id
  5. neutron floatingip-associate $floatingip_id $port_id

  You'll start seeting this in screen-q-svc.log:

  2014-09-17 20:56:17.758 5423 DEBUG neutron.db.l3_dvr_db [req-3faea024-ab6c-46f2-8706-e8b1028616ab None] Floating IP host: None _process_floating_ips /opt/stack/neutron/neutron/db/l3_dvr_db.py:296
  2014-09-17 20:56:17.760 5423 ERROR oslo.messaging.rpc.dispatcher [req-3faea024-ab6c-46f2-8706-e8b1028616ab ] Exception during message handling: Agent with agent_type=L3 agent and host=None could not be found
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 78, in sync_routers
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     context, host, router_ids))
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/l3_agentschedulers_db.py", line 299, in list_active_sync_routers_on_active_l3_agent
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     active=True)
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 458, in get_ha_sync_data_for_host
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     active)
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 330, in get_sync_data
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     self._process_floating_ips(context, routers_dict, floating_ips)
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 299, in _process_floating_ips
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     floating_ip['host'])
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 157, in _get_agent_by_type_and_host
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher     host=host)
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=None could not be found
  2014-09-17 20:56:17.760 5423 TRACE oslo.messaging.rpc.dispatcher
  2014-09-17 20:56:17.768 5423 ERROR oslo.messaging._drivers.common [req-3faea024-ab6c-46f2-8706-e8b1028616ab ] Returning exception Agent with agent_type=L3 agent and host=None could not be found to caller

  And it will just keep repeating as the l3-agent retries the call.

  The result is the l3-agent won't be able to do any work.

  I have a fix I'll send out for review.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1370795/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next