yahoo-eng-team team mailing list archive

[Mohammed Naser <2125500@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

It seems that there is a race condition happening right now in the OVN
code which makes it so that when doing high load operations, the code to
create fixed IPs for ports in the metadata service is racing.

It can easily be reproduced by the following:

```
export NET_ID=$(openstack network create test-subnet-race-condition -c id -f value)
for i in {1..48}; do
openstack subnet create --network ${NET_ID} --subnet-range 10.0.$i.0/24 subnet$i &
done
```

Once you run this, you can see that there isn't actually 48 fixed IPs on
the network and some will be missing, in my case usually 46 instead of
48.

```
openstack port list --network ${NET_ID} | grep ip_address
```

This is happening in a 2024.2 environment though since sleuthing the
code it seems that it would affect later versions too since the code
hasn't changed much, not to mention we upgraded this env to 2024.2 from
Zed to see if it would clear things up, same issue was there.

I believe the code that is introducing the race is here:

https://github.com/openstack/neutron/blob/e0ca9a0d68fbbcb98820f488accf2f84fb8c9639/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2813-L2814

I believe it's getting a not-up-to-date list.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2125500

Title:
  Metadata port fixed IPs missing from port

Status in neutron:
  New

Bug description:
  It seems that there is a race condition happening right now in the OVN
  code which makes it so that when doing high load operations, the code
  to create fixed IPs for ports in the metadata service is racing.

  It can easily be reproduced by the following:

  ```
  export NET_ID=$(openstack network create test-subnet-race-condition -c id -f value)
  for i in {1..48}; do
  openstack subnet create --network ${NET_ID} --subnet-range 10.0.$i.0/24 subnet$i &
  done
  ```

  Once you run this, you can see that there isn't actually 48 fixed IPs
  on the network and some will be missing, in my case usually 46 instead
  of 48.

  ```
  openstack port list --network ${NET_ID} | grep ip_address
  ```

  This is happening in a 2024.2 environment though since sleuthing the
  code it seems that it would affect later versions too since the code
  hasn't changed much, not to mention we upgraded this env to 2024.2
  from Zed to see if it would clear things up, same issue was there.

  I believe the code that is introducing the race is here:

  https://github.com/openstack/neutron/blob/e0ca9a0d68fbbcb98820f488accf2f84fb8c9639/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2813-L2814

  I believe it's getting a not-up-to-date list.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2125500/+subscriptions