← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1828375] [NEW] Bulk creation of subports fails with StaleDataError

 

Public bug reported:

ENV:
Neutron Ocata (10.0.4)
Kola Ansible
Rhel 7.4
One compute node (openvswitch) with 3 virtual machine kubernetes minions with Nested Kuryr. 

REPRODUCTION:
Creation of multiple containers in very short time (in fact creation of many subports in very short time)
Some OpenStack ports(trunk subports) will never transition to Active status.

EXCEPTION:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
  res = self.dispatcher.dispatch(message)
  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
  return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
  result = func(ctxt, **new_args)
  File "/usr/lib/python2.7/site-packages/neutron/services/trunk/rpc/server.py", line 110, in update_trunk_status
  trunk.update(status=status)
  File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 203, in decorator
  res = func(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/objects/trunk.py", line 127, in update
  super(Trunk, self).update()
  File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 618, in update
  self._get_composite_keys()))
  File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 80, in update_object
  db_obj.save(session=context.session)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
  session.flush()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2027, in flush
  self._flush(objects)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2145, in _flush
  transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2109, in _flush
  flush_context.execute()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
  rec.execute(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
  uow
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
  mapper, table, update)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 728, in _emit_update_statements
  (table.description, len(records), rows))
StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.


ADDITIONAL INVESTIGATION:
Actually there are multiple concurrent update calls to database for trunk ports revision_number. Trunks in database:

MariaDB [neutron]> select * from trunks;
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
| admin_state_up | project_id                       | id                                   | name         | port_id                              | status   | standard_attr_id |
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
|              1 | 9e8f068a23914209a839be451fe7533a | ce3c9271-70e9-4e4a-bdaf-bcb1a8d3901f | owc4-trunk-1 | e4392b5f-ecc7-4b94-bfe3-a2a8bc001057 | ACTIVE   |               56 |
|              1 | 9e8f068a23914209a839be451fe7533a | d9242ed7-4274-4537-962e-5e40b6540cc5 | owc4-trunk-2 | fe0c1041-4ccd-4fd7-b2b9-0c78c6b8c5fa | ACTIVE   |               58 |
|              1 | 9e8f068a23914209a839be451fe7533a | f50f1f1e-539b-4915-b410-d4b349c70d4b | owc4-trunk-0 | e4ff627d-37ca-447e-85cc-23347a9ba871 | ACTIVE   |               49 |
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
3 rows in set (0.00 sec)
 

standardattributes for trunks:

MariaDB [neutron]> select * from standardattributes where resource_type='trunks';
+----+---------------+---------------------+---------------------+-------------+-----------------+
| id | resource_type | created_at          | updated_at          | description | revision_number |
+----+---------------+---------------------+---------------------+-------------+-----------------+
| 49 | trunks        | 2019-04-16 08:14:10 | 2019-04-25 08:37:32 |             |             710 |
| 56 | trunks        | 2019-04-16 08:14:45 | 2019-04-25 08:37:32 |             |             739 |
| 58 | trunks        | 2019-04-16 08:15:09 | 2019-04-25 08:37:56 |             |             654 |
+----+---------------+---------------------+---------------------+-------------+-----------------+
3 rows in set (0.00 sec)
 
MariaDB [neutron]>

database queries dump:
        726462 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:36' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        728730 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:37' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        726445 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:37' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        727770 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:38' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        727468 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:38' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        727550 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:39' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        727048 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        728694 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        728697 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
        727765 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:41' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        727766 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        727470 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        728726 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        727550 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        726430 Query    UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
        728693 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        728688 Query    UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
        728692 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        727770 Query    UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:45' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
        727051 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:46' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        727469 Query    UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:47' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
        728732 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:47' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
        727767 Query    UPDATE standardattributes SET revision_number=833, updated_at='2019-04-25 11:00:49' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 832

I tried also to queue requests of port updates in agent/server code. After creating locks on methods:
1. def update_subport_bindings(self, context, subports): in file  neutron/services/trunk/rpc/server.py
2. def update_trunk_status(self, context, trunk_id, status): in file neutron/services/trunk/rpc/server.py
3. def add_subports(self, context, trunk_id, subports): in file neutron/services/trunk/plugin.py
4. def handle_subports(self, context, resource_type, subports, event_type): in file neutron/services/trunk/drivers/openvswitch/agent/driver.py

Those modification made subport creation work in more stable way.

Interesting fact is that restarting of agent is bringing all ports up.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1828375

Title:
  Bulk creation of subports fails with StaleDataError

Status in neutron:
  New

Bug description:
  ENV:
  Neutron Ocata (10.0.4)
  Kola Ansible
  Rhel 7.4
  One compute node (openvswitch) with 3 virtual machine kubernetes minions with Nested Kuryr. 

  REPRODUCTION:
  Creation of multiple containers in very short time (in fact creation of many subports in very short time)
  Some OpenStack ports(trunk subports) will never transition to Active status.

  EXCEPTION:
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
    res = self.dispatcher.dispatch(message)
    File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
    File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
    result = func(ctxt, **new_args)
    File "/usr/lib/python2.7/site-packages/neutron/services/trunk/rpc/server.py", line 110, in update_trunk_status
    trunk.update(status=status)
    File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 203, in decorator
    res = func(self, *args, **kwargs)
    File "/usr/lib/python2.7/site-packages/neutron/objects/trunk.py", line 127, in update
    super(Trunk, self).update()
    File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 618, in update
    self._get_composite_keys()))
    File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 80, in update_object
    db_obj.save(session=context.session)
    File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
    session.flush()
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2027, in flush
    self._flush(objects)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2145, in _flush
    transaction.rollback(_capture_exception=True)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2109, in _flush
    flush_context.execute()
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 728, in _emit_update_statements
    (table.description, len(records), rows))
  StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.

  
  ADDITIONAL INVESTIGATION:
  Actually there are multiple concurrent update calls to database for trunk ports revision_number. Trunks in database:

  MariaDB [neutron]> select * from trunks;
  +----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
  | admin_state_up | project_id                       | id                                   | name         | port_id                              | status   | standard_attr_id |
  +----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
  |              1 | 9e8f068a23914209a839be451fe7533a | ce3c9271-70e9-4e4a-bdaf-bcb1a8d3901f | owc4-trunk-1 | e4392b5f-ecc7-4b94-bfe3-a2a8bc001057 | ACTIVE   |               56 |
  |              1 | 9e8f068a23914209a839be451fe7533a | d9242ed7-4274-4537-962e-5e40b6540cc5 | owc4-trunk-2 | fe0c1041-4ccd-4fd7-b2b9-0c78c6b8c5fa | ACTIVE   |               58 |
  |              1 | 9e8f068a23914209a839be451fe7533a | f50f1f1e-539b-4915-b410-d4b349c70d4b | owc4-trunk-0 | e4ff627d-37ca-447e-85cc-23347a9ba871 | ACTIVE   |               49 |
  +----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
  3 rows in set (0.00 sec)
   

  standardattributes for trunks:

  MariaDB [neutron]> select * from standardattributes where resource_type='trunks';
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  | id | resource_type | created_at          | updated_at          | description | revision_number |
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  | 49 | trunks        | 2019-04-16 08:14:10 | 2019-04-25 08:37:32 |             |             710 |
  | 56 | trunks        | 2019-04-16 08:14:45 | 2019-04-25 08:37:32 |             |             739 |
  | 58 | trunks        | 2019-04-16 08:15:09 | 2019-04-25 08:37:56 |             |             654 |
  +----+---------------+---------------------+---------------------+-------------+-----------------+
  3 rows in set (0.00 sec)
   
  MariaDB [neutron]>

  database queries dump:
          726462 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:36' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          728730 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:37' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          726445 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:37' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          727770 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:38' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          727468 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:38' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          727550 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:39' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          727048 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          728694 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          728697 Query    UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
          727765 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:41' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          727766 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          727470 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          728726 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          727550 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          726430 Query    UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
          728693 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          728688 Query    UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
          728692 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          727770 Query    UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:45' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
          727051 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:46' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          727469 Query    UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:47' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
          728732 Query    UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:47' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
          727767 Query    UPDATE standardattributes SET revision_number=833, updated_at='2019-04-25 11:00:49' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 832

  I tried also to queue requests of port updates in agent/server code. After creating locks on methods:
  1. def update_subport_bindings(self, context, subports): in file  neutron/services/trunk/rpc/server.py
  2. def update_trunk_status(self, context, trunk_id, status): in file neutron/services/trunk/rpc/server.py
  3. def add_subports(self, context, trunk_id, subports): in file neutron/services/trunk/plugin.py
  4. def handle_subports(self, context, resource_type, subports, event_type): in file neutron/services/trunk/drivers/openvswitch/agent/driver.py

  Those modification made subport creation work in more stable way.

  Interesting fact is that restarting of agent is bringing all ports up.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1828375/+subscriptions


Follow ups