yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1253344] Re: IpAddressGenerationFailure while handling dhcp ports

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Wed, 22 Jan 2014 15:07:35 -0000
Reply-to: Bug 1253344 <1253344@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1253344

Title:
  IpAddressGenerationFailure while handling dhcp ports

Status in OpenStack Neutron (virtual network service):
  Fix Released

Bug description:
  Due to timing issues, if a subnet is deleted concurrently with a
  request from the dhcp agent the following stacktraces can be seen:

  2013-11-20 18:01:57.523 2727 DEBUG neutron.db.dhcp_rpc_base [-] Create dhcp port {u'port': {u'name': u'', u'admin_state_up': True, u'network_id': u'5c86cd4c-b37b-49fb-b3c4-ca43c4f9835c', u'tenant_id': u'5caf7376a94a483a9b0ba7fbabdd8843', u'fixed_ips': [{u'subnet_id': u'9610bd56-552f-4ad7-a8f0-d9ec2c75cbba'}], u'device_id': u'dhcp83d3530b-d4a1-5fee-8532-f3249772754b-5c86cd4c-b37b-49fb-b3c4-ca43c4f9835c'}} from devstack-precise-check-rax-iad-700557.slave.openstack.org. create_dhcp_port /opt/stack/new/neutron/neutron/db/dhcp_rpc_base.py:228
  2013-11-20 18:01:57.527 2727 DEBUG neutron.plugins.ml2.plugin [-] Deleting subnet record delete_subnet /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:517
  2013-11-20 18:01:57.539 2727 DEBUG neutron.db.db_base_plugin_v2 [-] Generated mac for network 5c86cd4c-b37b-49fb-b3c4-ca43c4f9835c is fa:16:3e:d7:d4:cf _generate_mac /opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py:302
  2013-11-20 18:01:57.547 2727 DEBUG neutron.plugins.ml2.plugin [-] Committing transaction delete_subnet /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:521
  2013-11-20 18:01:57.552 2727 DEBUG neutron.db.db_base_plugin_v2 [-] All IPs from subnet 9610bd56-552f-4ad7-a8f0-d9ec2c75cbba (10.100.0.0/24) allocated _generate_ip /opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py:466
  2013-11-20 18:01:57.553 2727 DEBUG neutron.openstack.common.rpc.amqp [-] Sending subnet.delete.end on notifications.info notify /opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py:598
  2013-11-20 18:01:57.553 2727 DEBUG neutron.openstack.common.rpc.amqp [-] UNIQUE_ID is b697e0b56ef74e44b3f3ea564b21b864. _add_unique_id /opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py:339
  2013-11-20 18:01:57.554 2727 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     **args)
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/common/rpc.py", line 44, in dispatch
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     neutron_ctxt, version, method, namespace, **kwargs)
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/dhcp_rpc_base.py", line 235, in create_dhcp_port
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     return plugin.create_port(context, port)
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 551, in create_port
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     result = super(Ml2Plugin, self).create_port(context, port)
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1332, in create_port
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     ips = self._allocate_ips_for_port(context, network, port)
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 704, in _allocate_ips_for_port
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     ips = self._allocate_fixed_ips(context, network, configured_ips)
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 648, in _allocate_fixed_ips
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     result = self._generate_ip(context, subnets)
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 483, in _generate_ip
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp     raise q_exc.IpAddressGenerationFailure(net_id=subnets[0]['network_id'])
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp IpAddressGenerationFailure: No more IP addresses available on network 5c86cd4c-b37b-49fb-b3c4-ca43c4f9835c.
  2013-11-20 18:01:57.554 2727 TRACE neutron.openstack.common.rpc.amqp 

  Or similarly:

  2013-11-20 17:48:31.865 32606 DEBUG neutron.plugins.ml2.plugin [-] Deleting subnet record delete_subnet /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:525
  2013-11-20 17:48:31.880 32606 DEBUG neutron.plugins.ml2.plugin [-] Committing transaction delete_subnet /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:529
  2013-11-20 17:48:31.882 32606 DEBUG neutron.db.db_base_plugin_v2 [-] Generated mac for network ad5c76d7-96ff-4089-a9d7-d31ef5869f84 is fa:16:3e:7a:cd:2d _generate_mac /opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py:302
  2013-11-20 17:48:31.893 32606 DEBUG neutron.db.db_base_plugin_v2 [-] All IPs from subnet 656f7834-ee5f-4ddf-8ae2-b36025f116e0 (10.100.0.0/24) allocated _generate_ip /opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py:466
  2013-11-20 17:48:31.894 32606 DEBUG neutron.openstack.common.rpc.amqp [-] Sending subnet.delete.end on notifications.info notify /opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py:598
  2013-11-20 17:48:31.894 32606 DEBUG neutron.openstack.common.rpc.amqp [-] UNIQUE_ID is 2a6e2cc37d324a4cafcad46c12cc6754. _add_unique_id /opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py:339
  2013-11-20 17:48:31.895 32606 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     **args)
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/common/rpc.py", line 44, in dispatch
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     neutron_ctxt, version, method, namespace, **kwargs)
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/dhcp_rpc_base.py", line 169, in get_dhcp_port
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     retval = plugin.create_port(context, dict(port=port_dict))
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 557, in create_port
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     result = super(Ml2Plugin, self).create_port(context, port)
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1332, in create_port
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     ips = self._allocate_ips_for_port(context, network, port)
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 704, in _allocate_ips_for_port
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     ips = self._allocate_fixed_ips(context, network, configured_ips)
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 648, in _allocate_fixed_ips
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     result = self._generate_ip(context, subnets)
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 483, in _generate_ip
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp     raise q_exc.IpAddressGenerationFailure(net_id=subnets[0]['network_id'])
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp IpAddressGenerationFailure: No more IP addresses available on network ad5c76d7-96ff-4089-a9d7-d31ef5869f84.
  2013-11-20 17:48:31.895 32606 TRACE neutron.openstack.common.rpc.amqp 

  These are rather harmless but it'd be good to get rid of them

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1253344/+subscriptions