yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1425402] [NEW] race condition when dhcp agent using reserved port

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Wei T <nuaafe@xxxxxxxxx>
Date: Wed, 25 Feb 2015 06:01:31 -0000
Reply-to: Bug 1425402 <1425402@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Because when we detach a network from a dhcp agent , the port's device
owner will goto "reserved_dhcp_port", and next time if we attach the
network to different agent at the same time, both of them will get port
list to see if there is reserved_dhcp_port, and update the port's device
id by change it to dhcp<hostname-hash>-<network_id>. It will end with
both of the two agent will create port with the same profile.


$neutron dhcp-agent-network-remove 47f46f82-5105-4de1-91ab-03faf765c6e3 cec2ca0a-8ced-429b-8a8d-10c0ed1b591d
$neutron dhcp-agent-network-remove d5340feb-1aea-4851-8c9e-d558eb0a37b9  cec2ca0a-8ced-429b-8a8d-10c0ed1b591d

select * from ports where network_id='cec2ca0a-8ced-429b-8a8d-10c0ed1b591d';
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
| tenant_id                        | id                                   | name | network_id                           | mac_address       | admin_state_up | status | device_id                                                                     | device_owner |
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
| c21da7e7d2b142049cc3ec8c551fdb80 | 23e1acb6-5bab-45b5-9d32-67f879771fa7 |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:4a:27:f4 |              1 | ACTIVE | dhcpe82d9294-4fef-5db5-b27e-d3e53cea2856-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | network:dhcp |
| c21da7e7d2b142049cc3ec8c551fdb80 | 25f50954-8bf9-4dd6-ba89-09b2ae169f86 |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:b8:8d:5e |              1 | ACTIVE | dhcp74013012-f6f2-511b-882b-fc394cffe407-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | network:dhcp |
| c21da7e7d2b142049cc3ec8c551fdb80 | 526be157-8c3b-4aff-997b-3f78a59767ad |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:47:da:eb |              1 | ACTIVE | reserved_dhcp_port                                                            | network:dhcp |
| c21da7e7d2b142049cc3ec8c551fdb80 | 825e7437-96a8-4523-95ba-6c3b6302482d |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:b3:f4:0b |              1 | ACTIVE | reserved_dhcp_port                                                            | network:dhcp |
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
4 rows in set (0.00 sec)
$neutron dhcp-agent-network-add 47f46f82-5105-4de1-91ab-03faf765c6e3 cec2ca0a-8ced-429b-8a8d-10c0ed1b591d & neutron dhcp-agent-network-add d5340feb-1aea-4851-8c9e-d558eb0a37b9 cec2ca0a-8ced-429b-8a8d-10c0ed1b591d

"ip netns exec qdhcp-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d ip a|grep tap"

[1] 08:40:34 [SUCCESS] net-001
5819: tap526be157-8c: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 192.168.200.22/24 brd 192.168.200.255 scope global tap526be157-8c
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap526be157-8c
[2] 08:40:34 [SUCCESS] net-002
8683: tap526be157-8c: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 192.168.200.22/24 brd 192.168.200.255 scope global tap526be157-8c
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap526be157-8c


select * from ports where network_id='cec2ca0a-8ced-429b-8a8d-10c0ed1b591d';
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
| tenant_id                        | id                                   | name | network_id                           | mac_address       | admin_state_up | status | device_id                                                                     | device_owner |
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
| c21da7e7d2b142049cc3ec8c551fdb80 | 23e1acb6-5bab-45b5-9d32-67f879771fa7 |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:4a:27:f4 |              1 | ACTIVE | dhcpe82d9294-4fef-5db5-b27e-d3e53cea2856-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | network:dhcp |
| c21da7e7d2b142049cc3ec8c551fdb80 | 25f50954-8bf9-4dd6-ba89-09b2ae169f86 |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:b8:8d:5e |              1 | ACTIVE | dhcp74013012-f6f2-511b-882b-fc394cffe407-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | network:dhcp |
| c21da7e7d2b142049cc3ec8c551fdb80 | 526be157-8c3b-4aff-997b-3f78a59767ad |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:47:da:eb |              1 | ACTIVE | dhcp1d6fb30f-279e-5cad-848f-353561944dc3-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d                                                            | network:dhcp |
| c21da7e7d2b142049cc3ec8c551fdb80 | 825e7437-96a8-4523-95ba-6c3b6302482d |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:b3:f4:0b |              1 | ACTIVE | reserved_dhcp_port                                                            | network:dhcp |
+----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+

The resolution could be add row lock when update port

** Affects: neutron
     Importance: Undecided
     Assignee: Wei T (nuaafe)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Wei T (nuaafe)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1425402

Title:
  race condition when dhcp agent using reserved port

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Because when we detach a network from a dhcp agent , the port's device
  owner will goto "reserved_dhcp_port", and next time if we attach the
  network to different agent at the same time, both of them will get
  port list to see if there is reserved_dhcp_port, and update the port's
  device id by change it to dhcp<hostname-hash>-<network_id>. It will
  end with both of the two agent will create port with the same profile.

  
  $neutron dhcp-agent-network-remove 47f46f82-5105-4de1-91ab-03faf765c6e3 cec2ca0a-8ced-429b-8a8d-10c0ed1b591d
  $neutron dhcp-agent-network-remove d5340feb-1aea-4851-8c9e-d558eb0a37b9  cec2ca0a-8ced-429b-8a8d-10c0ed1b591d

  select * from ports where network_id='cec2ca0a-8ced-429b-8a8d-10c0ed1b591d';
  +----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
  | tenant_id                        | id                                   | name | network_id                           | mac_address       | admin_state_up | status | device_id                                                                     | device_owner |
  +----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
  | c21da7e7d2b142049cc3ec8c551fdb80 | 23e1acb6-5bab-45b5-9d32-67f879771fa7 |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:4a:27:f4 |              1 | ACTIVE | dhcpe82d9294-4fef-5db5-b27e-d3e53cea2856-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | network:dhcp |
  | c21da7e7d2b142049cc3ec8c551fdb80 | 25f50954-8bf9-4dd6-ba89-09b2ae169f86 |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:b8:8d:5e |              1 | ACTIVE | dhcp74013012-f6f2-511b-882b-fc394cffe407-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | network:dhcp |
  | c21da7e7d2b142049cc3ec8c551fdb80 | 526be157-8c3b-4aff-997b-3f78a59767ad |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:47:da:eb |              1 | ACTIVE | reserved_dhcp_port                                                            | network:dhcp |
  | c21da7e7d2b142049cc3ec8c551fdb80 | 825e7437-96a8-4523-95ba-6c3b6302482d |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:b3:f4:0b |              1 | ACTIVE | reserved_dhcp_port                                                            | network:dhcp |
  +----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
  4 rows in set (0.00 sec)
  $neutron dhcp-agent-network-add 47f46f82-5105-4de1-91ab-03faf765c6e3 cec2ca0a-8ced-429b-8a8d-10c0ed1b591d & neutron dhcp-agent-network-add d5340feb-1aea-4851-8c9e-d558eb0a37b9 cec2ca0a-8ced-429b-8a8d-10c0ed1b591d

  "ip netns exec qdhcp-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d ip a|grep
  tap"

  [1] 08:40:34 [SUCCESS] net-001
  5819: tap526be157-8c: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
      inet 192.168.200.22/24 brd 192.168.200.255 scope global tap526be157-8c
      inet 169.254.169.254/16 brd 169.254.255.255 scope global tap526be157-8c
  [2] 08:40:34 [SUCCESS] net-002
  8683: tap526be157-8c: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
      inet 192.168.200.22/24 brd 192.168.200.255 scope global tap526be157-8c
      inet 169.254.169.254/16 brd 169.254.255.255 scope global tap526be157-8c

  
  select * from ports where network_id='cec2ca0a-8ced-429b-8a8d-10c0ed1b591d';
  +----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
  | tenant_id                        | id                                   | name | network_id                           | mac_address       | admin_state_up | status | device_id                                                                     | device_owner |
  +----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+
  | c21da7e7d2b142049cc3ec8c551fdb80 | 23e1acb6-5bab-45b5-9d32-67f879771fa7 |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:4a:27:f4 |              1 | ACTIVE | dhcpe82d9294-4fef-5db5-b27e-d3e53cea2856-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | network:dhcp |
  | c21da7e7d2b142049cc3ec8c551fdb80 | 25f50954-8bf9-4dd6-ba89-09b2ae169f86 |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:b8:8d:5e |              1 | ACTIVE | dhcp74013012-f6f2-511b-882b-fc394cffe407-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | network:dhcp |
  | c21da7e7d2b142049cc3ec8c551fdb80 | 526be157-8c3b-4aff-997b-3f78a59767ad |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:47:da:eb |              1 | ACTIVE | dhcp1d6fb30f-279e-5cad-848f-353561944dc3-cec2ca0a-8ced-429b-8a8d-10c0ed1b591d                                                            | network:dhcp |
  | c21da7e7d2b142049cc3ec8c551fdb80 | 825e7437-96a8-4523-95ba-6c3b6302482d |      | cec2ca0a-8ced-429b-8a8d-10c0ed1b591d | fa:16:3e:b3:f4:0b |              1 | ACTIVE | reserved_dhcp_port                                                            | network:dhcp |
  +----------------------------------+--------------------------------------+------+--------------------------------------+-------------------+----------------+--------+-------------------------------------------------------------------------------+--------------+

  The resolution could be add row lock when update port

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1425402/+subscriptions
Follow ups

[Bug 1425402] Re: race condition when dhcp agent using reserved port
From: Doug Hellmann, 2015-12-03
[Bug 1425402] [NEW] race condition when dhcp agent using reserved port
From: Wei T, 2015-02-25
References

[Bug 1425402] [NEW] race condition when dhcp agent using reserved port
From: Wei T, 2015-02-25