yahoo-eng-team team mailing list archive

[Vish Ishaya <vishvananda@xxxxxxxxx>]

Public bug reported:

There is a race condition in linux_net.get_dhcp_opts when
use_single_default_gateway is set. Because it makes multiple queries, if
an instance is deleted while iterating through the list of fixed ips, it
can cause an exception like the following:

2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1043, in update_dhcp
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     restart_dhcp(context, dev, network_ref)
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1096, in restart_dhcp
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     write_to_file(optsfile, get_dhcp_opts(context, network_ref))
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1016, in get_dhcp_opts
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     context, instance_uuid)
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 110, in wrapper
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     args, kwargs)
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 425, in object_class_action
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     objver=objver, args=args, kwargs=kwargs)
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     wait_for_reply=True, timeout=timeout)
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout)
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     raise result

While the query could be fixed to catch the exception, this code is also
horribly inefficient, so the logic needs to be fixed to only make one db
query.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1359510

Title:
  Network deallocation can fail with use_single_default_gateway

Status in OpenStack Compute (Nova):
  New

Bug description:
  There is a race condition in linux_net.get_dhcp_opts when
  use_single_default_gateway is set. Because it makes multiple queries,
  if an instance is deleted while iterating through the list of fixed
  ips, it can cause an exception like the following:

  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1043, in update_dhcp
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     restart_dhcp(context, dev, network_ref)
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 249, in inner
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1096, in restart_dhcp
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     write_to_file(optsfile, get_dhcp_opts(context, network_ref))
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1016, in get_dhcp_opts
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     context, instance_uuid)
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 110, in wrapper
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     args, kwargs)
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 425, in object_class_action
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     objver=objver, args=args, kwargs=kwargs)
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     wait_for_reply=True, timeout=timeout)
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout)
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     return self._send(target, ctxt, message, wait_for_reply, timeout)
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
  2014-08-20 19:44:31.371 10867 TRACE oslo.messaging.rpc.dispatcher     raise result

  While the query could be fixed to catch the exception, this code is
  also horribly inefficient, so the logic needs to be fixed to only make
  one db query.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1359510/+subscriptions