yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1879407] [NEW] [OVN] Modifying FIP that is no associated causes ovn_revision_numbers to go stale

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Flavio Fernandes <flavio@xxxxxxxxxxx>
Date: Mon, 18 May 2020 22:32:32 -0000
Reply-to: Bug 1879407 <1879407@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

NOTE: This is a low priority issue, mostly because it eventually gets
fixed by maintenance task

CheckRevisionNumberCommand relies in finding a corresponding entry in OVN's NAT table
in order to update the OVN_REV_NUM_EXT_ID_KEY to keep ovn and neutron databases in sync.

Ref: http://lucasgom.es/posts/neutron_ovn_database_consistency.html

Trouble is that unless the floating ip is associated, there will be no
entries in OVN's NAT table, causing the call to

 db_rev.bump_revision(context, floatingip, ovn_const.TYPE_FLOATINGIPS)

to not take place.

Steps to reproduce it:

# create a floating ip but do not associate it with anything so router_id is None
FIP=172.24.4.8
openstack floating ip create --floating-ip-address ${FIP} public
FIP_UUID=$(openstack floating ip show ${FIP} -f value -c id) ; echo $FIP_UUID

# Mess with its name, which will bump revision on fip object
openstack floating ip set --description foo ${FIP_UUID}

Code when there is no NAT for a given FIP makes line 1044 skip line 1045

https://github.com/openstack/neutron/blob/15088b39bab715e40d8161a85c95ca400708c83f/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1044

check_rev_cmd.result is None

The dbs are now the inconsistent state

mysql> use neutron;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from standardattributes where resource_type="floatingips";
+----+---------------+---------------------+---------------------+-------------+-----------------+
| id | resource_type | created_at          | updated_at          | description | revision_number |
+----+---------------+---------------------+---------------------+-------------+-----------------+
| 49 | floatingips   | 2020-05-18 20:56:51 | 2020-05-18 20:58:58 | foo2        |               2 |
+----+---------------+---------------------+---------------------+-------------+-----------------+
1 row in set (0.01 sec)

mysql> select * from ovn_revision_numbers where resource_type="floatingips";
+------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
| standard_attr_id | resource_uuid                        | resource_type | revision_number | created_at          | updated_at          |
+------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
|               49 | 5a1e1ffa-0312-4e78-b7a0-551c396bcf6b | floatingips   |               0 | 2020-05-18 20:56:51 | 2020-05-18 20:57:08 |
+------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
1 row in set (0.00 sec)

Maintenance task fixes it up later

May 18 21:50:29 stack neutron-server[909]: DEBUG futurist.periodics [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBIn\
consistenciesPeriodics.check_for_inconsistencies' {{(pid=3186) _process_scheduled /usr/local/lib/python3.6/dist-packages/futurist/periodics.py:642}}
May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Synchronizing Neutron and OVN datab\
ases {{(pid=3186) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:347}}
May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Number of inconsistencies found at \
create/update: floatingips=1 {{(pid=3186) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:325}}
May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Fixing resource 6b876a35-d286-4407-\
b538-9ce07ab1a281 (type: floatingips) at create/update {{(pid=3186) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:359}}
May 18 21:50:29 stack neutron-server[909]: INFO neutron.db.ovn_revision_numbers_db [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Successfully bumped revision number for resource 6b876a35-d286-4407-b538-9ce07ab1\
a281 (type: floatingips) to 1
May 18 21:50:29 stack neutron-server[909]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Synchronization finished (took 0.08 \
seconds)

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  NOTE: This is a low priority issue, mostly because it eventually gets
  fixed by maintenance task
  
  CheckRevisionNumberCommand relies in finding a corresponding entry in OVN's NAT table
  in order to update the OVN_REV_NUM_EXT_ID_KEY to keep ovn and neutron databases in sync.
  
  Ref: http://lucasgom.es/posts/neutron_ovn_database_consistency.html
  
- Trouble is that unless the floating ip is associated, there will be no entries in OVN's NAT
- table, causing the call to 
+ Trouble is that unless the floating ip is associated, there will be no
+ entries in OVN's NAT table, causing the call to
  
-        db_rev.bump_revision(context, floatingip,
- ovn_const.TYPE_FLOATINGIPS)
+  db_rev.bump_revision(context, floatingip, ovn_const.TYPE_FLOATINGIPS)
  
  to not take place.
- 
  
  Steps to reproduce it:
  
  # create a floating ip but do not associate it with anything so router_id is None
  FIP=172.24.4.8
  openstack floating ip create --floating-ip-address ${FIP} public
  FIP_UUID=$(openstack floating ip show ${FIP} -f value -c id) ; echo $FIP_UUID
  
  # Mess with its name, which will bump revision on fip object
  openstack floating ip set --description foo ${FIP_UUID}
  
- 
  Code when there is no NAT for a given FIP makes line 1044 skip line 1045
  
  https://github.com/openstack/neutron/blob/15088b39bab715e40d8161a85c95ca400708c83f/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1044
  
  check_rev_cmd.result is None
- 
  
  The dbs are now the inconsistent state
  
  mysql> use neutron;
  Reading table information for completion of table and column names
  You can turn off this feature to get a quicker startup with -A
  
  Database changed
  mysql> select * from standardattributes where resource_type="floatingips";
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  | id | resource_type | created_at          | updated_at          | description | revision_number |
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  | 49 | floatingips   | 2020-05-18 20:56:51 | 2020-05-18 20:58:58 | foo2        |               2 |
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  1 row in set (0.01 sec)
  
  mysql> select * from ovn_revision_numbers where resource_type="floatingips";
  +------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
  | standard_attr_id | resource_uuid                        | resource_type | revision_number | created_at          | updated_at          |
  +------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
  |               49 | 5a1e1ffa-0312-4e78-b7a0-551c396bcf6b | floatingips   |               0 | 2020-05-18 20:56:51 | 2020-05-18 20:57:08 |
  +------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
  1 row in set (0.00 sec)
  
- 
- 
  Maintenance task fixes it up later
  
  May 18 21:50:29 stack neutron-server[909]: DEBUG futurist.periodics [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBIn\
  consistenciesPeriodics.check_for_inconsistencies' {{(pid=3186) _process_scheduled /usr/local/lib/python3.6/dist-packages/futurist/periodics.py:642}}
  May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Synchronizing Neutron and OVN datab\
  ases {{(pid=3186) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:347}}
  May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Number of inconsistencies found at \
  create/update: floatingips=1 {{(pid=3186) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:325}}
  May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Fixing resource 6b876a35-d286-4407-\
  b538-9ce07ab1a281 (type: floatingips) at create/update {{(pid=3186) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:359}}
  May 18 21:50:29 stack neutron-server[909]: INFO neutron.db.ovn_revision_numbers_db [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Successfully bumped revision number for resource 6b876a35-d286-4407-b538-9ce07ab1\
  a281 (type: floatingips) to 1
  May 18 21:50:29 stack neutron-server[909]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Synchronization finished (took 0.08 \
  seconds)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1879407

Title:
  [OVN] Modifying FIP that is no associated causes ovn_revision_numbers
  to go stale

Status in neutron:
  New

Bug description:
  NOTE: This is a low priority issue, mostly because it eventually gets
  fixed by maintenance task

  CheckRevisionNumberCommand relies in finding a corresponding entry in OVN's NAT table
  in order to update the OVN_REV_NUM_EXT_ID_KEY to keep ovn and neutron databases in sync.

  Ref: http://lucasgom.es/posts/neutron_ovn_database_consistency.html

  Trouble is that unless the floating ip is associated, there will be no
  entries in OVN's NAT table, causing the call to

   db_rev.bump_revision(context, floatingip, ovn_const.TYPE_FLOATINGIPS)

  to not take place.

  Steps to reproduce it:

  # create a floating ip but do not associate it with anything so router_id is None
  FIP=172.24.4.8
  openstack floating ip create --floating-ip-address ${FIP} public
  FIP_UUID=$(openstack floating ip show ${FIP} -f value -c id) ; echo $FIP_UUID

  # Mess with its name, which will bump revision on fip object
  openstack floating ip set --description foo ${FIP_UUID}

  Code when there is no NAT for a given FIP makes line 1044 skip line
  1045

  https://github.com/openstack/neutron/blob/15088b39bab715e40d8161a85c95ca400708c83f/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1044

  check_rev_cmd.result is None

  The dbs are now the inconsistent state

  mysql> use neutron;
  Reading table information for completion of table and column names
  You can turn off this feature to get a quicker startup with -A

  Database changed
  mysql> select * from standardattributes where resource_type="floatingips";
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  | id | resource_type | created_at          | updated_at          | description | revision_number |
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  | 49 | floatingips   | 2020-05-18 20:56:51 | 2020-05-18 20:58:58 | foo2        |               2 |
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  1 row in set (0.01 sec)

  mysql> select * from ovn_revision_numbers where resource_type="floatingips";
  +------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
  | standard_attr_id | resource_uuid                        | resource_type | revision_number | created_at          | updated_at          |
  +------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
  |               49 | 5a1e1ffa-0312-4e78-b7a0-551c396bcf6b | floatingips   |               0 | 2020-05-18 20:56:51 | 2020-05-18 20:57:08 |
  +------------------+--------------------------------------+---------------+-----------------+---------------------+---------------------+
  1 row in set (0.00 sec)

  Maintenance task fixes it up later

  May 18 21:50:29 stack neutron-server[909]: DEBUG futurist.periodics [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBIn\
  consistenciesPeriodics.check_for_inconsistencies' {{(pid=3186) _process_scheduled /usr/local/lib/python3.6/dist-packages/futurist/periodics.py:642}}
  May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Synchronizing Neutron and OVN datab\
  ases {{(pid=3186) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:347}}
  May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Number of inconsistencies found at \
  create/update: floatingips=1 {{(pid=3186) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:325}}
  May 18 21:50:29 stack neutron-server[909]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Fixing resource 6b876a35-d286-4407-\
  b538-9ce07ab1a281 (type: floatingips) at create/update {{(pid=3186) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:359}}
  May 18 21:50:29 stack neutron-server[909]: INFO neutron.db.ovn_revision_numbers_db [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Successfully bumped revision number for resource 6b876a35-d286-4407-b538-9ce07ab1\
  a281 (type: floatingips) to 1
  May 18 21:50:29 stack neutron-server[909]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-35091ee8-f2fe-47cc-b757-8bb70f750b47 None None] Maintenance task: Synchronization finished (took 0.08 \
  seconds)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1879407/+subscriptions
Follow ups

[Bug 1879407] Re: [OVN] Modifying FIP that is no associated causes ovn_revision_numbers to go stale
From: Brian Haley, 2024-03-07