yahoo-eng-team team mailing list archive

[Bug 1197627] Re: DHCP agent race condition on subnet and network delete

** Changed in: neutron
Status: Fix Committed => Fix Released

--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1197627

Title:
DHCP agent race condition on subnet and network delete

Status in OpenStack Neutron (virtual network service):
Fix Released

Bug description:
When a network with a subnet is deleted from Horizon, Horizon iterates
through and deletes each subnet and then deletes the network [1].
After quantum deletes the subnet, the DHCP agent asynchronously
discovers that its subnet has been deleted and begins the teardown
process for its interface [2]. The result is that the DHCP agent
teardown process and the network delete process end up being called
nearly simultaneously and try to delete the DHCP port at the same time
[3][4].

When the DHCP agent checks to see if the port exists[5], the port is
still present because it hasn't yet been deleted by the network_delete
method. However, by the time it calls delete_port, it has just been
deleted by delete_network. This results in a two calls to _delete_port
that doesn't generate any DB errors because they are so close together
that sqlalchemy hasn't finished removing the port from the database by
the time the DHCP agent checks the database to see if its port exists
before deleting it.

This bug may not have surfaced in various plugins if the _delete_port
method is fast enough to remove the port from the database on the
first request before the DHCP agents checks the database. But in the
case of plugins where _delete_port results in a REST call to a
controller before removing the port from the DB, it results in a big
enough window for the race condition to occur.

I can submit a patch, but I would like to get feedback on the
appropriate approach to fix this for any potential use-case rather
than just one plugin.

1. https://github.com/openstack/horizon/blob/46ebda3d5d155cb2569c8b96f1f83bc97c51d4d6/openstack_dashboard/dashboards/project/networks/tables.py#L52
2. https://github.com/openstack/quantum/blob/0aa5becc005166ecdfeacc288f124c8806d8b2f8/neutron/agent/dhcp_agent.py#L648
3. https://github.com/openstack/quantum/blob/0aa5becc005166ecdfeacc288f124c8806d8b2f8/neutron/db/dhcp_rpc_base.py#L152
4. https://github.com/openstack/quantum/blob/0aa5becc005166ecdfeacc288f124c8806d8b2f8/neutron/db/db_base_plugin_v2.py#L1030
5. https://github.com/openstack/quantum/blob/0aa5becc005166ecdfeacc288f124c8806d8b2f8/neutron/db/dhcp_rpc_base.py#L149

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1197627/+subscriptions